06.7.1 · Skill in Evidence-based decisions
Interpreting results
Reading experiment and analysis results without fooling yourself.
Interpreting results means checking data quality, segments, guardrails and the size of the effect before deciding. Surprising results deserve suspicion first. Writing the interpretation down makes the reasoning reviewable.
Resources
2- articleGo deeperA/B testing intuition busters: common misunderstandings in online controlled experiments ↗Ron Kohavi, Alex Deng, Lukas Vermeer · 2022 · 10 pages
Why surprising wins are usually wrong, why low-powered tests exaggerate effects, and how to read p-values correctly.
- articleGo deeperDiagnosing sample ratio mismatch in A/B testing ↗Microsoft Experimentation Platform · 2020 · 12 min
The check that catches broken experiments before anyone reads the results, with a taxonomy of the causes.
Tools
none yetTools for this topic are being researched.