06.4.6 · Skill in Experiment design
Experiment guardrails
Checks that protect users and the validity of a running experiment.
Experiment guardrails include guardrail metrics, sample ratio mismatch checks and rules for when to stop. They catch broken assignment and changes that harm the product while improving the target metric. Peeking at results and stopping early is a common way to fool yourself.
Resources
3- articleStart hereGuardrail metrics for A/B tests, explained ↗PostHog · 5 min
Written for product engineers. What guardrail metrics are, which ones to pick, and how companies such as Airbnb use them.
- articleGo deeperDiagnosing sample ratio mismatch in A/B testing ↗Microsoft Experimentation Platform · 2020 · 12 min
The check that catches broken experiments before anyone reads the results, with a taxonomy of the causes.
- articleGo deeperBetter product decisions with guardrail metrics ↗Spotify (Confidence) · 2024 · 8 min
How Spotify uses inferiority tests on guardrail metrics to decide when a change should be rolled back.
Tools
none yetTools for this topic are being researched.