In 30 seconds
- AI features are probabilistic; outcomes vary.
- Treat each feature as an experiment with evals and success metrics.
- Monitor in production; quality can drift.
What's different
| Traditional feature | AI feature |
|---|---|
| Same input, same output | Output varies |
| Pass or fail tests | Quality scored on a scale |
| Stable after release | Can drift with data or model changes |
Adapted practices
- Evals as acceptance criteria
- Spikes to test feasibility before committing
- Hypotheses with measurable outcomes
- Gradual rollouts behind feature flags
Quality over time
Monitor real outputs, collect user feedback, add failures to the eval set, and re-run evals when models or prompts change. Treat model updates like dependency upgrades: tested before rollout.
Myth vs reality
Myth · AI features are done once they launch.
Reality · They need ongoing monitoring.
Myth · A demo proves an AI feature works.
Reality · Demos show best cases; evals show the range.
Myth · Agile doesn't fit AI work.
Reality · Its experiment-driven loop fits AI work well.
FAQ
How do we estimate AI features?
With wider ranges and early spikes.
What's model drift?
Quality changing over time as data or the model changes.
Who decides if quality is good enough?
The Product Owner, using eval results and user feedback.