Data Scientist how-to
How to Analyze an A/B Test
Careful analysis prevents false conclusions.
Quick answer
Careful analysis prevents false conclusions. In short: confirm; check; calculate, then complete the remaining steps below.
Step by step
Confirm the test was randomized.
Check sample sizes for each group.
Calculate the metric for each group.
Run a statistical test.
Check for practical significance, not just statistical.
Report the result and recommendation.
Safety reminders for data scientists
- Protect personal and sensitive data
- Follow privacy laws and company policies
- Check models for bias and fairness
- Take screen breaks and use ergonomic setups
Key terms
Dataset · Feature · Model · Training data · Overfitting · Regression: see the full data scientist glossary.