06.5.7 · Skill in Statistical reasoning
Simpson's paradox
When a trend reverses after the data is split into groups.
A metric can improve overall while getting worse in every segment, because the mix of users changed. It is a common trap after launches and pricing changes. Checking key segments before drawing conclusions avoids it.
Resources
1- docsStart hereSimpson's paradox: discover possibilities with your segments, not shipping decisions ↗Optimizely docs · 4 min
Why a result that reverses across segments, or a win in one segment, should lead to a new test rather than a shipping decision.
Tools
none yetTools for this topic are being researched.