In 30 seconds
- Treat ideas as assumptions to test, not features to build.
- Each experiment has a hypothesis, a metric and a pass line.
- Cheap, fast experiments beat big, slow bets.
An experiment in four parts
- Hypothesis: we believe X will cause Y.
- Metric: what we'll measure.
- Pass line: what result means success, decided in advance.
- Timebox: how long we'll run it.
Common experiment types
- A/B tests on live traffic
- Fake-door tests: a button for a feature that doesn't exist yet
- Concierge or manual tests of a service
- Prototype tests with real users
Acting on results
Pass: invest further. Fail: drop or rethink the idea, which is a cheap win. Unclear: tighten the test or try a different one. Record what was learned so it isn't re-tested later.
Myth vs reality
Myth · A failed experiment is a failure.
Reality · It saved you from building the wrong thing.
Myth · Experiments need big traffic.
Reality · Many useful experiments are qualitative and small.
Myth · Experimentation slows delivery.
Reality · It stops delivery of things nobody wants.
FAQ
How many experiments should we run?
Enough to test your riskiest assumptions; quality beats volume.
Who should design experiments?
The product trio: product, design and engineering together.
Is a fake-door test ethical?
Yes, if you explain promptly and don't charge users.