How to conduct robust Product experiments

Understanding the effect of chance in test results

Photo by Edge2Edge Media
👋 This post appeared in my weekly newsletter on Product Management. 
Sign up here to get the next post straight to your inbox.

How many samples do we need to reach Statistical Significance? What’s the intuition?

Tl;dr

Significance is Relative

Statistical Significance is not just about Sample Size

1. Sample Size

2. Size of the Effect you want to detect

3. Variability in Samples

4. Power of a Statistical Test

The power of a statistical test is the probability of the test detecting an effect IF there is truly an effect. In other words, its the probability that the test correctly rejects the null hypothesis.

The above Concepts in Action

👋 Thanks for reading! Follow me on twitter where I share learnings on managing and building products.