What you will be able to do
- Phrase an improvement target relative to a measured baseline
- Choose between an offline eval comparison and a live A/B test for a given change
- Set up a fair comparison: same test set, pinned versions, multiple trials
Key concept
Baseline comparison — A prompt or model change only counts as an improvement once it has been measured against the current version on the same criteria. A/B testing is the controlled version of that comparison: the baseline and the candidate are evaluated side by side, and the difference between them is the result.
1.Start from a baseline and a target
Anthropic's evaluation cookbook calls getting the most accuracy out of Claude "an empirical science, and a process of continuous improvement." The point is that you can't tell whether a rewritten prompt, an added retrieval step, or a different model is better just by reading a few outputs. You have to run the old version and the new version against the same questions and compare the scores. So every A/B test starts with a baseline: the current production prompt or model, already measured.
The baseline also shapes the success criteria. Anthropic's guide to success criteria contrasts a vague goal ("The model should classify sentiments well") with one that is specific, measurable, relevant and achievable. Its good example asks for an F1 score of at least 0.85 on a held-out set of 10,000 diverse Twitter posts, and adds that this is a 5% improvement over the current baseline. That last part is what makes the target achievable: it is set relative to measured performance, not picked out of the air. The guide says to base targets on "industry benchmarks, prior experiments, AI research, or expert knowledge", and earlier versions of your own system are the most direct kind of prior experiment.
Once you have a target written as "better than the baseline by X on metric Y", an A/B test can answer it with a yes or no. Without one, two variants just produce two numbers and nobody has agreed which difference matters.
Checkpoint 1 of 5· Check yourself
A team wants to test a revised classification prompt. Which success criterion is best suited to judging the A/B result?
A good criterion names a metric, a test set and a target relative to the measured baseline. 'Noticeably better' can't be measured, 100% ignores what is achievable, and a handful of examples is not a controlled comparison.
“which is a 5% improvement over the current baseline (Achievable).”Source: platform.claude.com
2.Offline comparisons versus live A/B tests
"A/B testing" can mean two different things, and they are used at different stages. The first is offline: you run two or more variants over a fixed test set and compare the scores. The classification cookbook lists "Multiple prompt variants: A/B testing different phrasings, structures, and approaches" and "Model comparisons" among the things production evaluation infrastructure has to support. The second is live: you split real user traffic between variants and measure what happens to those users.
Anthropic's engineering post on agent evals compares the two directly. Automated evals give fast, reproducible results with no user impact, but they can create false confidence if the test set doesn't match real usage. Live A/B testing measures actual user outcomes and controls for confounds, but it is slow and only tests changes you have already deployed. So the order is: offline evals catch problems on every change, and "A/B testing validates significant changes once you have sufficient traffic." The post's summary is that the most effective teams use automated evals for fast iteration, production monitoring for ground truth, and periodic human review for calibration.
| Aspect | Automated evals (offline) | A/B testing (live traffic) |
|---|---|---|
| What runs | Tests run programmatically without real users | Variants compared with real user traffic |
| Main strength | Faster iteration, fully reproducible, can run on every commit | Measures actual user outcomes (retention, task completion), controls for confounds |
| Main cost | Up-front build effort; false confidence if it doesn't match real usage | Days or weeks to reach significance; needs sufficient traffic |
| Scope | Tests scenarios at scale without a production deployment | Only tests changes you deploy |
| When to use | Pre-launch and in CI/CD, on each change | Validating significant changes once traffic allows |
Checkpoint 2 of 5· Match them up
Match each evaluation method to the limitation Anthropic lists for it
Tap a term, then the definition that fits it.
Each method covers a gap left by the others. A/B tests are slow and need traffic, offline evals can drift from real usage, and monitoring only sees problems after users hit them.
“Slow; days or weeks to reach significance and requires sufficient traffic”Source: www.anthropic.com
Checkpoint 3 of 5· Exam question
A support-ticket triage team maintains a categorical labeling eval (urgent/normal/low, with human-labeled ground truth) used to compare successive versions of their Claude-based classification prompt. Before promoting a revised prompt candidate, they want the fastest, most objective way to confirm it outperforms the current production prompt on this task. Which approach should they use?
Correct answer: A — Run both prompt versions against the same held-out labeled set and compare exact-match accuracy against the ground-truth category labels
- A. Correct. Since ground-truth labels already exist for this categorical task, exact-match accuracy on a held-out set gives an objective, scalable, and directly comparable score between prompt versions before rollout.
- B. Incorrect. Manual agent voting is slow, inconsistent across raters, and does not scale the way automated grading against existing ground truth does.
- C. Incorrect. Self-reported confidence is not a validated accuracy measure and can be miscalibrated, so it does not substitute for grading against known correct labels.
- D. Incorrect. Shipping to all traffic without a controlled held-out comparison risks degrading production quality and conflates classification accuracy with an indirect proxy metric.
3.Make the comparison fair
A comparison only tells you something if the variant is the only thing that changed. The sources point to four controls.
Same test set. In the classification cookbook, every prompt variant and temperature setting was scored on the same 68 examples, specifically for "Ensuring fair comparison across all configurations." If the two arms run on different inputs, a score gap may come from the inputs, not the prompt.
Pinned versions. In the Managed Agents versioning cookbook, a session created with a plain agent ID gets whatever the latest version is. Passing an explicit version "pins to an exact version, which is what we want for a controlled comparison." If someone updates the agent while your test is running, an unpinned arm quietly switches prompts partway through.
Multiple trials. Model outputs vary from run to run, so a single run per case can mistake noise for a real difference. Anthropic's agent-evals post says: "Because model outputs vary between runs, we run multiple trials to produce more consistent results."
Representative and large enough. The success-criteria guide says to design evals that mirror your real-world task distribution, including edge cases. It also says to automate grading and prefer volume: more automatically graded questions beat fewer hand-graded ones. Volume matters even more in A/B work, because you re-grade every time you test a new variant.
For quick manual comparisons, the Developer Console's output comparison mode shows outputs from different prompts side by side. It is a good way to read differences before you score them at scale.
Checkpoint 4 of 5· Check yourself
An engineer A/B tests two Managed Agents prompt versions, but the harness creates every session with the bare agent ID. A colleague calls agents.update halfway through. What is the problem?
A bare agent ID resolves to the latest version, so the arm under test changes mid-run. Pinning an explicit version keeps each arm fixed.
“Passing a plain string (agent=AGENT_ID) gets you whatever the latest version is.”Source: platform.claude.com
Checkpoint 5 of 5· Exam question
A documentation team is iterating on a prompt that generates article summaries. They have 250 articles paired with human-written reference summaries and want an automated score that rewards a candidate summary for preserving the key information from the reference in a similar order, so they can compare prompt variants at scale during iteration. Which evaluation method best fits this need?
Correct answer: A — Score each candidate summary against its reference using ROUGE-L and track the F1 score across prompt variants
- A. Correct. ROUGE-L measures the longest common subsequence between candidate and reference text, capturing whether key information appears in a similar order, and it is well suited to scoring summarization quality across many articles automatically.
- B. Incorrect. Exact string match is appropriate for short categorical answers, not free-text summaries, since two good summaries will rarely match a reference word-for-word.
- C. Incorrect. A binary PHI check evaluates a safety property, not whether the summary preserves the reference's key information and ordering.
- D. Incorrect. Latency measures response speed, not whether the generated content faithfully reflects the reference summary's content.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Every prompt change should be validated first with a live A/B test, since only real users show true performance.Why is that wrong?
Automated offline evals are the first line of defense on each change. Live A/B tests are slow, need sufficient traffic, and are reserved for validating significant changes.
Covered in Offline comparisons versus live A/B tests
2.Referencing an agent by its ID is enough to keep an A/B comparison stable.Why is that wrong?
A bare agent ID resolves to the latest version. Only an explicit version pin keeps each arm of the comparison fixed.
Covered in Make the comparison fair
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Base your targets on industry benchmarks, prior experiments, AI research, or expert knowledge.”
↩︎ Start from a baseline and a target“A/B testing: Compare performance against a baseline model or earlier version.”
↩︎ Offline comparisons versus live A/B tests“Design evals that mirror your real-world task distribution.”
↩︎ Make the comparison fair“More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals.”
↩︎ Make the comparison fair“A/B testing: Compare performance against a baseline model or earlier version.”
↩︎ Key concept“which is a 5% improvement over the current baseline (Achievable).”
↩︎ Checkpoint - 2.https://platform.claude.com/cookbook/misc-building-evalsSecondary source
“Optimizing Claude to give you the highest possible accuracy on a task is an empirical science, and a process of continuous improvement.”
↩︎ Start from a baseline and a target - 3.
“Multiple prompt variants: A/B testing different phrasings, structures, and approaches”
↩︎ Offline comparisons versus live A/B tests“The same test set (68 examples): Ensuring fair comparison across all configurations”
↩︎ Make the comparison fair - 4.
“The most effective teams combine these methods: automated evals for fast iteration, production monitoring for ground truth, and periodic human review for calibration.”
↩︎ Offline comparisons versus live A/B tests“Because model outputs vary between runs, we run multiple trials to produce more consistent results.”
↩︎ Make the comparison fair“A/B testing validates significant changes once you have sufficient traffic.”
↩︎ Exam trap 1“running on each agent change and model upgrade as the first line of defense against quality problems”
↩︎ Prediction“Slow; days or weeks to reach significance and requires sufficient traffic”
↩︎ Checkpoint - 5.
“Compare the outputs from different prompts side by side in the new output comparison mode in the Developer Console.”
↩︎ Make the comparison fair
Also cited
“pins to an exact version, which is what we want for a controlled comparison.”
↩︎ Exam trap 2“Passing a plain string (agent=AGENT_ID) gets you whatever the latest version is.”
↩︎ Checkpoint