A/B Testing Prompts and Models Against a Baseline

1.An A/B test needs a baseline and a target

Anthropic's guidance on success criteria lists A/B testing as a quantitative way to measure performance. It defines A/B testing as comparing performance against a baseline model or an earlier version. So an A/B test always has two arms. The baseline is what you run today: the current prompt, model or pipeline. The variant is the change you think will help. Both arms are measured the same way, and the question is whether the variant is better by enough to matter.

That means you decide on the measure and the target before you run the test. The documentation gives an example of a good criterion. A sentiment classifier should reach an F1 score of at least 0.85 on a held-out test set of 10,000 diverse Twitter posts, 'which is a 5% improvement over the current baseline'. The target is stated relative to the baseline. The guidance calls that part of the criterion 'Achievable' and says targets should rest on industry benchmarks, earlier experiments, AI research or expert knowledge. A variant that clears an absolute number but loses to the baseline has not won.

Sources1

2.Offline A/B: prompt variants on a shared test set

The cheapest A/B test runs offline. Each variant goes through the same evaluation set, and no real users are involved. The success-criteria guidance says metrics should be measurable: quantitative metrics or well-defined qualitative scales. Offline comparison is where that pays off, because every variant gets a number on the same scale. The Building Evals cookbook describes this as knowing whether a change to your prompt improved a key metric.

Anthropic's classification cookbook works through an example. It compares three prompt approaches for a ticket classifier: Simple, RAG, and RAG with chain-of-thought (CoT). The cookbook reports the results as a progression: 'Random baseline: ~10% → Simple: ~70% → RAG: 94% → RAG + CoT: 97%'. It says this data-driven approach showed which techniques added value and by how much. For production-scale testing it lists 'Multiple prompt variants: A/B testing different phrasings, structures, and approaches' and model comparisons across Claude models or temperature settings. It runs these in Promptfoo, an open-source evaluation toolkit. Each configuration ran on the same 68-example test set, to ensure fair comparison across all configurations, with each prediction checked automatically against ground-truth labels.

Haiku: T-0.0 Prompt: RAG w/ CoT    95.588235
Haiku: T-0.2 Prompt: RAG w/ CoT    95.588235
Haiku: T-0.8 Prompt: RAG w/ CoT    95.588235
Haiku: T-0.0 Prompt: RAG           94.117647
Haiku: T-0.4 Prompt: RAG           94.117647
Haiku: T-0.6 Prompt: RAG           94.117647
Haiku: T-0.4 Prompt: RAG w/ CoT    94.117647
Haiku: T-0.6 Prompt: RAG w/ CoT    94.117647
Haiku: T-0.2 Prompt: RAG           92.647059
Haiku: T-0.8 Prompt: RAG           89.705882
Haiku: T-0.6 Prompt: Simple        72.058824
Haiku: T-0.0 Prompt: Simple        70.588235
Haiku: T-0.2 Prompt: Simple        70.588235
Haiku: T-0.8 Prompt: Simple        70.588235
Haiku: T-0.4 Prompt: Simple        69.117647

Sources123

3.Online A/B: what live traffic tells you, and what it doesn't

Offline results show how a variant scores on your test set. They don't show how users respond to it. For that, the success-criteria guidance points to user feedback, including 'Implicit measures like task completion rates'. Anthropic's engineering post on agent evals describes A/B testing as comparing variants with real user traffic. It lists this against the other ways to understand system performance, and each comes with a clear cost.

The post places each method at a stage. Automated evals run on each agent change and model upgrade as 'the first line of defense against quality problems'. 'A/B testing validates significant changes once you have sufficient traffic.' In practice, you screen variants offline first. Only the few that beat the baseline go to a live split, because each live test takes days or weeks. And when a metric moves in a live test, the test alone gives little signal on why. You find the reason by reading transcripts.

Sources14

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Temperature is the main setting to tune when you A/B test a classification prompt.Why is that wrong?

   In Anthropic's Promptfoo comparison, the prompt approach (Simple vs RAG vs RAG w/ CoT) drove accuracy. The simple prompt stayed around 70% at every temperature, and CoT accuracy barely moved as temperature changed.

   Covered in Offline A/B: prompt variants on a shared test set

2. 2.A live A/B test gives a quick answer, so run one for every small prompt tweak.Why is that wrong?

   Live A/B tests are slow and need enough traffic to reach significance. Use automated evals on every change and save live A/B tests for significant changes.

   Covered in Online A/B: what live traffic tells you, and what it doesn't

3. 3.When a live A/B test shows a metric moved, the test also tells you why.Why is that wrong?

   A/B tests measure outcomes, not causes. Finding out why a metric moved means reading transcripts.

   Covered in Online A/B: what live traffic tells you, and what it doesn't