Learning roadmap
LLM Ops & Evals
Ship LLM features that stay good: eval sets with real thresholds, prompts under version control with CI gates, observability that catches drift, and the incident runbook for the day a model deprecation eats your product.
4 stages~8 h
That leaves 4 stages, ~8 hours of focused work. You can clear it in any order, but the stages compound — try them top-to-bottom your first time.
- You'll pick up:
golden sets, rubric judging, judge agreement, edge case sampling, anonymized inputs
Practice quizzesLab taskBuild an "eval lab" page: paste a prompt, run it against a 20-case golden set, show per-case scores, the aggregate pass rate against a written threshold, and a judge-vs-human agreement panel that warns when the judge drifts below 90% agreement.
Build it in the LabCheckpointEvery prompt in production has a versioned eval set, and the last two changes were judged by it — not by feel.
What you'll be able to do
- Build an eval set whose pass rate predicts what users will experience.
- Make a prompt change a PR: diffable, evaluated, and blocking on regression.
- See drift, latency, and cost as dashboards — not as user complaints.
- Survive a provider deprecation with a tested fallback instead of a fire drill.
Or try a related path