Anatomy of an A/B Test

Intuitions on Null Hypothesis, Statistical Significance, p-Values

Photo by Icons8 Team

👋 This post appeared in my weekly newsletter on Product Management. 
Sign up here to get the next post straight to your inbox.

When building products everyone has an opinion.

“Change this button to green and people will buy more”

“Let’s change this banner image to something else and we will see more signups”.

And while opinions galore, A/B testing is a quick and easy way to test them and see if they really hold water. Product Managers love it.

Here’s what typically goes on:

1. You have a hypothesis — say about your landing page conversions: “Changing this landing page button from Grey to Red will make people will click it more often”

2. You create 2 versions of your landing page: Version A is the original page, Version B is the variation.

3. You use a tool to Run the A/B test ….and wait for results

The tool then starts serving the control and treated landing pages randomly to your selected target audience and after a while spits out results.

And the results say — the Red button generated 5% more clicks than the other button……and the results are statistically significant.

Statistical Significance!

Most Product Managers know that when looking at the results of an A/B test, Statistical Significance is king. This thing — whatever it means — decides whether or not the experiment’s results are conclusive. If the results are statistically significant, it’s usually inferred that they are valid. Business leaders love to ask if the results are statistically significant.

In our example above, our A/B test tool does the heavy lifting and simply declares that the results are significant — so we declare victory and change the color of the button to Red. Now we better see a lift in click-throughs!