What you will be able to do
- Design a paired (counterfactual) test that shows whether a disparity across groups comes from the model rather than from real differences in the inputs
- Tell data bias, prompt-framing bias and evaluation-design bias apart by changing one part of the pipeline at a time
- Pick a mitigation that fits the bias source you identified, and reject fixes that only add noise or hide the gap
- Spot when an automated judge, not the system under test, is causing an apparent group disparity
Key concept
Counterfactual paired testing — Send inputs that are identical in every legitimate factor and differ only in the group attribute, such as name, region or dialect, then compare the outcomes. A gap that survives the pairing is caused by the system. A gap that disappears was caused by differences in the inputs.
1.A disparity across groups is evidence to test, not a verdict
A gap in outcomes between groups tells you something differs. It does not tell you what. Take a lender that uses Claude to turn loan applications into approve or refer recommendations. An audit finds a 31% refer rate in one postal region against 18% elsewhere. That gap has two possible explanations. The applications from that region may differ in income, debt or credit history, so the model is responding to real risk. Or the model may be reacting to the region itself, or to something that stands in for it. The audit numbers alone cannot separate the two, because the groups differ in many ways at once.
The direct test is a counterfactual pair. Build applications with the same income, debt and credit history, change only the postal region, and compare refer rates across the two sets. Since nothing else differs, any gap that remains is caused by the region or by text correlated with it. If the gap disappears, the original disparity came from the applicant pool, and the question becomes whether those underlying factors are fair to use. That is a policy question, not a model defect. Run enough pairs and repeat each one, because a single comparison can come out either way by chance.
The tempting alternatives each skip the measurement. Adding "ignore applicant location" to the system prompt is a mitigation, not a test. If the refer rates then move, you still don't know whether the model was biased or the instruction simply changed behaviour. Proxies such as a street name or a local employer also survive the instruction, because instructions are followed by pattern-matching rather than understanding. Moving to a larger model tier changes two things at once and also measures nothing. Asking Claude to list the factors behind each decision gives you generated text about the decision, not a measurement of what caused it. An explanation that never mentions the region does not show the region had no effect.
Checkpoint 1 of 8· Check yourself
The lender adds "Ignore applicant location" to the system prompt and re-runs the audit sample. The refer rate in the flagged postal region drops from 31% to 22%. What can the team conclude?
Adding the instruction is a mitigation, not a test. A shift in rates doesn't show whether the model was biased or the instruction just changed its behaviour. Proxies for location can also survive the instruction, because Claude follows instructions by pattern-matching. Only paired inputs that differ in nothing but the region can isolate the cause.
“instructions are followed via pattern-matching, not understanding”Source: academy.claude.com
Checkpoint 2 of 8· Exam question
A lender uses the Claude Messages API to turn loan applications into an approve/refer recommendation. An audit finds a 31% refer rate for applicants in one postal region versus 18% elsewhere. You are the architect who must establish whether the model itself is biased before anything is changed. Which next step gives the most direct evidence?
Correct answer: A — Submit paired applications with identical income, debt and credit history but different postal regions, then compare the refer rates between the two groups.
- A. Correct. Paired applications that share the same financial profile and differ only in region isolate location as the single changed variable. A refer-rate difference between the pairs is direct evidence about the model's behaviour.
- B. Incorrect. This changes the system before the cause is established, which is a mitigation rather than a diagnosis. A narrower gap on the audit sample would not show whether the original gap came from model bias or from real differences between the applicant groups.
- C. Incorrect. Moving to a different model tier is itself a change to the system, and it alters many things besides location sensitivity. A gap that persists or shrinks in the audit sample says little about whether the current model is biased.
- D. Incorrect. Factors a model lists after the fact are not a dependable record of what drove its decision. Counting region mentions therefore cannot confirm or rule out region-driven behaviour, and it does not test the model's response to location directly.
2.Three places bias enters: data, prompt framing and evaluation design
Once a paired test shows a real gap, the next question is where it comes from. The mitigation depends on the answer, and a fix aimed at the wrong source changes nothing or makes things worse. Bias in a Claude application usually comes from one of three sources.
Data bias comes from what the model or pipeline learned from or reads. The model writes text based on patterns in what it saw during training, and fine-tuning adds the preferences of human raters. Skewed associations in that material surface as skewed outputs. In your own system, data bias also lives in retrieved documents, historical labels used as reference answers, and input fields that act as proxies for a protected attribute.
Prompt-framing bias comes from how you ask. Few-shot examples, persona descriptions and leading wording all steer the output. A prompt that opens with an assumption tends to get answers that go along with that assumption.
Evaluation-design bias comes from how you measure. The system's outputs may be equally good for every group, but a test set that under-samples a group, reference answers written for one group, or a grader that rewards surface features can make one group look worse.
You separate the three by changing one thing at a time. First, rerun the paired inputs through a neutralized prompt with balanced examples and no leading wording. If the gap closes, the framing was the cause. If it stays, the cause sits in the model's learned associations or in the data your pipeline feeds it. When an automated metric shows a gap but human raters scoring the same outputs do not, suspect the measurement before the system.
Checkpoint 3 of 8· Check yourself
In a support assistant's eval, a Claude-based grader scores answers to customers who write in one regional dialect lower. Human raters scoring those same outputs find no quality gap between dialect groups. Where should the team look first?
The outputs are the same and humans rate them equally, so the gap comes from how the grader measures, not from what the system produces. The grader needs checking against human judgment, and changing the data, prompt or model would leave it untouched.
“LLM graders with criteria defined by the product team and periodic human calibration”Source: www.anthropic.com
3.Fixing prompt-framing bias at the examples and wording
Few-shot examples are the strongest framing signal in most prompts. Claude imitates them more reliably than it follows abstract instructions. Picture a coaching assistant whose three examples always show a male user asking about promotion and a female user asking about work-life balance. Name-swap tests confirm the result: identical questions get promotion advice for male names and balance advice for female names. The model is copying the gender-to-topic pattern the examples taught it.
A mitigation that fits this source has two parts:
- Rewrite the examples so each persona appears with promotion, pay and work-life questions. That leaves no gender-to-topic link for Claude to copy.
- Re-run matched pairs that differ only in the user's name through the revised prompt, and compare the advice topics before release. This confirms the gap actually closed and gives you a regression test for later prompt edits.
Fixes that leave the examples alone don't address the source. A higher sampling temperature makes the advice less predictable, but the skew stays in the average. Adding a retrieval step over career articles gives Claude more material to draw on while the biased examples keep steering which material it uses. A larger context window makes the examples a smaller share of the prompt, but they remain the clearest pattern to imitate. Leading wording works the same way: start a prompt with an assumption and the model tends to agree with it. Prompts that describe a user's group should say nothing about what that group wants.
Checkpoint 4 of 8· Check yourself
Name-swap tests on a career-coaching assistant show that the same question gets promotion advice for male names and work-life balance advice for female names. All three few-shot examples pair a male user with promotion and a female user with balance. Which change fixes the source of the bias?
The skew comes from the examples, and Claude copies examples more reliably than it follows abstract instructions. Rebalancing the examples removes the gender-to-topic pattern, and the matched pairs confirm the gap closed. Temperature only adds noise around the same skew. Retrieval adds material, but the examples still steer how Claude uses it.
“Provide examples of the output you want. This is more effective than abstract instructions.”Source: platform.claude.com
Checkpoint 5 of 8· Exam question
A coaching assistant built on the Claude Messages API uses a system prompt with three few-shot examples. In every example a male user asks about promotion and a female user asks about work-life balance. Name-swap tests show that otherwise identical questions get promotion advice for male names and balance advice for female names. Choose two mitigations that match the source of this bias.(Select 2)
Correct answers: A, B — Rewrite the prompt's examples so each persona appears with promotion, pay and work-life questions, giving Claude no gender-to-topic pattern to imitate.; Send matched question pairs that differ only in the user's name through the revised prompt, comparing the advice topics before release to measure the gap.
- A. Correct. The skew comes from the examples, which always show male users asking about promotion and female users asking about work-life balance. Pairing each persona with several topics removes the pattern Claude was imitating, which addresses the bias at its source.
- B. Correct. Pairs of otherwise identical questions that differ only in the user's name isolate the effect of apparent gender. Running them against the revised prompt shows whether the topic gap has narrowed or whether sensitivity to gender remains.
- C. Incorrect. Temperature changes how random the sampling is, not what the examples teach. The gender-to-topic association stays in the prompt and still pulls responses toward it, so the added variation only makes the advice less consistent and does not fix the bias.
- D. Incorrect. Retrieved passages add material but leave the skewed examples in the system prompt. Nothing ensures the passages override the pattern those examples set, and the name-swap disparity can persist, so the source is untouched.
- E. Incorrect. A larger context window increases capacity but does not alter what the three examples show. They still demonstrate the same gender-to-topic pairing, and the model still follows that pattern, so the bias persists.
4.When the evaluation, not the system, is biased
Sometimes the evaluation creates the disparity. A team compares two prompt variants using Claude as a pairwise judge. Variant B gives shorter answers to users who write in a regional dialect, and the judge ranks those answers lower. Human raters score the same answers as equally good. The system under test is fine for that group. The judge is reacting to length or phrasing, not quality. Making product decisions on that signal would "fix" something that was never broken.
The fixes target the judge itself. Validate it against human ratings on a labelled sample from each dialect group, and keep tracking judge-human agreement by group. A judge that agrees with people overall but not for one group is biased against that group. Score factual correctness against reference answers with a rubric that ignores length. Shuffle which variant appears first in each pairwise comparison so position preference evens out. The design of the judge itself is covered in LLM-as-judge design. The point here is that a judge's errors can look like group bias in the system.
Two tempting moves make the problem worse. Using the model that wrote the answers as the judge adds a preference for its own style, and step-by-step reasoning does not remove that. Dropping the dialect cases from the judged set and reporting the overall score hides the gap instead of measuring it, even with a footnote. The eval should reflect the real mix of users, including the groups where it is hardest to score.
Checkpoint 6 of 8· Check yourself
A release deadline is close, and the Claude judge still ranks Variant B's answers to dialect users lower, although human raters score them as equally good. A team member proposes dropping the dialect cases from the judged set and reporting the overall score with a footnote. What should the team do instead?
Dropping the group hides the gap instead of measuring it, even with a footnote. The eval should reflect the real mix of users, and the judge needs checking against human ratings for each group. A judge that is also the answering model adds a preference for its own style. Adding more samples doesn't fix a bias that points the same way every time.
“Design evals that mirror your real-world task distribution.”Source: platform.claude.com
Checkpoint 7 of 8· Exam question
A team compares two prompt variants with Claude acting as a pairwise judge. Variant B's answers to users who write in a regional dialect are consistently shorter, and the judge ranks them lower, implying B is worse for that group. Human raters score the same answers as equally good. Choose two actions that address the likely source.(Select 2)
Correct answers: A, B — Validate the judge against human ratings on a labeled sample from each dialect group, and monitor judge-human agreement by group over time.; Score factual correctness against reference answers with a rubric that ignores length, and shuffle which variant appears first in each pairwise comparison.
- A. Correct. The judge and the human raters disagree for one dialect group, so checking the judge against human labels per group exposes where its scores are unreliable and keeps them from being trusted without that check.
- B. Correct. The judge's ranking tracks answer length rather than quality. A reference-based correctness rubric that ignores length, plus shuffling which variant comes first, removes the verbosity and position biases from the judge's design.
- C. Incorrect. A consistent lean toward longer answers is a systematic bias, not random noise, so averaging more verdicts at higher temperature reproduces the same lean. Adding more prompts only gives a tighter estimate of the same biased ranking.
- D. Incorrect. A judge that is also the generator brings in self-preference bias, and reasoning before the verdict does not remove the preference for longer answers. Nothing here checks the judge against the human ratings that disagree with it.
- E. Incorrect. Dropping the dialect cases hides the disparity instead of fixing the judge, so the product goes unmeasured for those users. The length bias stays in the judge and can still distort rankings on the remaining prompts.
5.Mitigating data bias you cannot retrain away
When a paired test still shows a gap after the prompt is neutralized, the bias comes from learned associations or from data your pipeline supplies. You cannot retrain Claude, so mitigation works on what reaches the model and on how its outputs are checked.
Data-level mitigations, in order of how directly they hit the source:
- Remove or coarsen input fields that act as proxies for the protected attribute, such as a full postal code where the decision needs none. Then re-run the paired test to confirm the gap closed.
- Rebalance retrieval corpora and correct skewed historical labels, so neither the context nor the reference answers carry the old disparity.
- Route cases near the decision boundary, and any decision where paired tests showed sensitivity, to human review checkpoints instead of letting Claude decide alone.
- Monitor outcomes by group in production with a stated tolerance, so a new prompt, model version or data shift that reopens the gap shows up in monitoring rather than in a complaint.
Monitoring needs a number to hold to, not a goal to "be fair". A routing system can set a target, such as keeping accuracy within a few points across customer groups, and audit against it on a schedule. The paired test that diagnosed the problem becomes the regression check that keeps it fixed.
Checkpoint 8 of 8· Check yourself
The lender's paired test still showed a gap after the prompt was neutralized. The team coarsened the full postal code to district level and re-ran the paired test, and the gap closed. Which plan keeps it closed after the next prompt edit, model version or data shift?
Monitoring needs a number to hold to, measured for each group and checked on a schedule. The paired test that found the problem then serves as the regression check. A goal to 'be fair' only catches problems when someone complains. An overall rate can stay flat while the gap between regions reopens. Claude's own account of its decision is generated text, not a measurement of what caused it.
“Even “hazy” topics such as ethics and safety can be quantified”Source: platform.claude.com
Sources
(7)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Regularly audit routing decisions for potential biases”
↩︎ A disparity across groups is evidence to test, not a verdict“Regularly audit routing decisions for potential biases”
↩︎ Mitigating data bias you cannot retrain away“Regularly audit routing decisions for potential biases”
↩︎ Key concept - 2.
“instructions are followed via pattern-matching, not understanding”
↩︎ A disparity across groups is evidence to test, not a verdict“instructions are followed via pattern-matching, not understanding”
↩︎ Exam trap 1 - 3.https://academy.claude.com/courses/ai-capabilities-and-limitations/next-token-predictionOfficial docs
“It writes answers word by word based on what tends to follow what.”
↩︎ Three places bias enters: data, prompt framing and evaluation design - 4.https://academy.claude.com/courses/ai-capabilities-and-limitations/how-ai-gets-its-characterOfficial docs
“Fine-tuning uses human judgments about good responses, and those judgments leave fingerprints”
↩︎ Three places bias enters: data, prompt framing and evaluation design“See whether the AI validates your framing or pushes back.”
↩︎ Fixing prompt-framing bias at the examples and wording - 5.https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/increase-consistencyOfficial docs
“Provide examples of the output you want. This is more effective than abstract instructions.”
↩︎ Fixing prompt-framing bias at the examples and wording“Provide examples of the output you want. This is more effective than abstract instructions.”
↩︎ Exam trap 2 - 6.
“Design evals that mirror your real-world task distribution.”
↩︎ When the evaluation, not the system, is biased“Most use cases need multidimensional evaluation along several success criteria.”
↩︎ Mitigating data bias you cannot retrain away“Design evals that mirror your real-world task distribution.”
↩︎ Exam trap 4“Even “hazy” topics such as ethics and safety can be quantified”
↩︎ Checkpoint - 7.
“LLM graders with criteria defined by the product team and periodic human calibration”
↩︎ When the evaluation, not the system, is biased“LLM graders with criteria defined by the product team and periodic human calibration”
↩︎ Exam trap 3“LLM graders with criteria defined by the product team and periodic human calibration”
↩︎ Checkpoint“Because model outputs vary between runs, we run multiple trials to produce more consistent results.”
↩︎ Prediction