Two Useful Concepts from Statistics for Web Marketers
We're going to discuss two important and useful concepts from statistics here: confounders and interactions. These come up in web marketing, usability engineering and other contexts when you're running A/B split tests to determine whether one design alternative is better than another (for example, whether one ad creative is better than another, or one button label is better than another, etc.). Here we'll stick with web marketing as a domain but it's easy enough to apply the ideas elsewhere.
Confounders
The first of the two concepts we want to discuss is that of confounders. We can illustrate this with an example. Suppose that we're an online university, and we have two different "call to action" (CTA) buttons that we want to compare, such as "Apply Now". The buttons are as follows:
- Button A: Blue, displayed on programs page
- Button B: Red, displayed on home page
We run an A/B split test and find that the clickthrough rate (i.e., # clicks divided by # impressions) for the buttons is as follows:
- Button A: 5.6%
- Button B: 3.2%
From this, can we conclude that blue buttons are generally more effective? It turns out that we can't, because we can't tell whether it's the blue color that accounts for the improved clickthrough rate, the fact that the button is displayed on the programs page, or the combination of the two. (We'll assume for the sake of discussion that our sample sizes are large enough to exclude sampling error—the inherent "noise" you get when you sample from a population—as a possible explanation.) In trying to analyze the effect of the color variable on clickthrough rates, the page variable is acting as a confounder: it makes it impossible to tell whether changes in the color really explain changes in the clickthrough rate.
Good experimental design attempts to eliminate or else control the influence of confounders. Sometimes that's easier said than done; it's not always obvious when confounders are at work.
In this particular case, though, it's pretty easy to address the issue. If we wanted to study the impact of color, we'd pick an individual page and then vary only the color. That would give us the apples-to-apples comparison that we want, even if we had to qualify our result by saying that the impact of color choice might be different if we were to have used a different page. That brings us to our next topic, interactions.
