In 30 seconds
- Split users randomly between version A and version B.
- Measure one main metric decided in advance.
- Run until you have enough data, not until you like the result.
How it works
- Pick one change and one main metric, such as sign-up rate.
- Randomly send some users to version A (current) and some to version B (changed).
- Run until you reach the sample size you planned.
- Compare results and decide: ship B, keep A, or test again.
Designing a fair test
- Write the hypothesis first; see hypothesis-driven development
- Decide sample size and duration before starting
- Change one thing at a time
- Run across full weekly cycles to avoid day-of-week bias
Common mistakes
- Stopping as soon as results look good (peeking)
- Testing on too little traffic to detect a real difference
- Tracking many metrics and cherry-picking the winner
Myth vs reality
Myth · A/B tests prove why users behave a certain way.
Reality · They show what happened; research explains why.
Myth · Every change should be A/B tested.
Reality · Low-traffic products and obvious fixes often don't need it.
Myth · A winning test wins forever.
Reality · Effects can fade; keep watching key metrics.
FAQ
How much traffic do I need?
Enough to detect the size of change you care about; online calculators help.
How long should a test run?
At least one to two full weeks, and until the planned sample size is reached.
What's the difference from multivariate testing?
Multivariate tests several changes at once and needs much more traffic.