06.4 · Capability in Measurement

Experiment design

Designing tests that show whether a change caused an effect, not just whether something moved.

Map6 skills

Not startedLearningComfortable

6 skillsClick a hexagon to set your status

06.4.1HypothesesA testable prediction of what a change will do and why.
06.4.2Control groupsA comparison group that does not get the change.
06.4.3RandomizationAssigning users to groups by chance so the groups are comparable.
06.4.4A/B testsComparing two versions with real users to see which performs better.
06.4.5Sample sizeHow many users an experiment needs to detect an effect.
06.4.6Experiment guardrailsChecks that protect users and the validity of a running experiment.

Mark all 6

Where to start

3 resources

articleStart here
How not to run an A/B test ↗
Evan Miller · 2010 · 8 min
The classic explanation of why peeking at results and stopping early produces false wins, and what to do instead.
articleGo deeper
A/B testing intuition busters: common misunderstandings in online controlled experiments ↗
Ron Kohavi, Alex Deng, Lukas Vermeer · 2022 · 10 pages
Why surprising wins are usually wrong, why low-powered tests exaggerate effects, and how to read p-values correctly.
bookGo deeper · Paid
Trustworthy online controlled experiments: a practical guide to A/B testing ↗
Ron Kohavi, Diane Tang, Ya Xu · 2020 · 288 pages
The standard reference on running experiments at scale, written by the people who built experimentation at Microsoft, Google and LinkedIn.

4 tools

ToolOpen source
GrowthBook
growthbook.io ↗
Open-source feature flags and A/B testing that analyzes results from your own data warehouse.
ToolFreemium
Statsig
statsig.com ↗
Feature flags, experiments and product analytics with built-in statistics such as variance reduction.
ToolOpen source
PostHog
posthog.com ↗
Open-source product analytics with session replay, feature flags, experiments and surveys in one tool.
ToolFree
Evan's Awesome A/B Tools
evanmiller.org ↗
Free calculators for sample size, significance of results and sequential tests.