06.5 · Capability in Measurement

Statistical reasoning

Enough statistics to tell signal from noise and to avoid the classic traps when reading data.

Map7 skills

Not startedLearningComfortable

7 skillsClick a hexagon to set your status

06.5.1Uncertainty and varianceAccepting that every measurement has noise around it.
06.5.2Confidence intervalsA range of plausible values for a measured effect.
06.5.3Effect sizesHow big a difference is, not only whether it exists.
06.5.4Statistical powerThe chance a test detects an effect that really exists.
06.5.5Sampling and selection biasWhen the data comes from a group that does not represent everyone.
06.5.6ConfoundingA hidden factor that affects both the cause and the effect.
06.5.7Simpson's paradoxWhen a trend reverses after the data is split into groups.

Mark all 7

Where to start

3 resources

courseStart here
Seeing Theory: a visual introduction to probability and statistics ↗
Daniel Kunin and others (Brown University) · 2017
Interactive visualizations of distributions, inference and confidence intervals. The fastest way to build intuition for the basics.
videoStart here
Statistical power, clearly explained ↗
StatQuest with Josh Starmer
A short, visual explanation of what power is and why sample size changes it.
articleGo deeper
A/B testing intuition busters: common misunderstandings in online controlled experiments ↗
Ron Kohavi, Alex Deng, Lukas Vermeer · 2022 · 10 pages
Why surprising wins are usually wrong, why low-powered tests exaggerate effects, and how to read p-values correctly.

1 tools

ToolFree
Evan's Awesome A/B Tools
evanmiller.org ↗
Free calculators for sample size, significance of results and sequential tests.