ProductEngineer.pro
06.4.5 · Skill in Experiment design

Sample size

How many users an experiment needs to detect an effect.

Sample size depends on the baseline rate, the smallest effect worth detecting and the confidence you want. Calculating it before the test tells you how long to run it, or whether to run it at all. Underpowered tests produce noise that looks like results.

Your status · Sample size

Saved in this browser only. It colours this skill's tile on your map.

Resources

2
  • videoStart here
    Power analysis, clearly explained ↗
    StatQuest with Josh Starmer

    How to work out the sample size a test needs before running it.

  • articleStart here
    How not to run an A/B test ↗
    Evan Miller · 2010 · 8 min

    The classic explanation of why peeking at results and stopping early produces false wins, and what to do instead.

Tools

1