Interactions
Let's modify the example a bit. We still have the blue and red buttons, but realizing the error of our ways, we decide to show the blue button on both the home page and the programs page, and similarly for the red button. That way when we compare the two buttons, we have more confidence that we're really comparing the impact of the color difference rather than the page. We run the A/B split test again and come up with the following numbers:
- Blue button: 2,500 clicks out of 50,000 impressions (5.0% clickthrough rate)
- Red button: 3,500 clicks out of 50,000 impressions (7.0% clickthrough rate)
At first glance this would seem to contradict the results of the first test. Could it be that red buttons are better after all? Maybe, but we can't draw that conclusion from the information we've presented so far. Suppose that a more complete picture of the data is as follows:
| Home page | Programs page | Total | |
|---|---|---|---|
| Blue button | 1,100 clicks of 25,000 impressions (4.4% CTR) | 1,400 clicks of 25,000 impressions (5.6% CTR) | 2,500 clicks of 50,000 impressions (5.0% CTR) |
| Red button | 800 clicks of 25,000 impressions (3.2% CTR) | 2,700 clicks of 25,000 impressions (10.8% CTR) | 3,500 clicks of 50,000 impressions (7.0% CTR) |
| Total | 1,900 clicks of 50,000 impressions (3.8% CTR) | 4,100 clicks of 50,000 impressions (8.2% CTR) | 6,000 clicks of 100,000 impressions (6.0% CTR) |
The table above (known in statistics as a contingency table) gives a more refined picture of what's going on. Let's look at the salient points:
- The blue button outperforms the red button on the home page, while the red button vastly outperforms the blue button on the programs page. This means that it's too simplistic in this example to say that one color is better than the other color. There's an interaction between the color and the page: it's the combination of the two that determines the CTR, not either of them taken alone.
- Studies of this sort can give rise to interesting questions and insights. One might ask, for example, why the blue button wins on the home page but the red one wins on the programs page. Then you might notice that the home page is mostly red while the programs page is mostly blue. That might lead you to the useful insight that maybe the color contrast between the page and the button is what's really doing the work here.
- For both buttons considered individually, the programs page outperforms the home page. That suggests that the page is the more important factor here—perhaps (unsurprisingly) people are more apt to click on the CTA after they've had a chance to examine the programs. If we continue with the scenario described in (2) above (i.e., home page mostly red, programs page mostly blue), it would be reasonable to hypothesize that the color of the button should be chosen to contrast with the color of the surrounding page, and that it's especially important to get this right on the programs page, which is where users are more likely to click the CTA.
That does it for the discussion of confounders and interactions. The concepts are easy enough, but it makes sense to ask whether it's worth it to go through the hassle. In the next section I'll show that it very often is.
