CertSafari
    Anthropic Claude Certified Developer – Professional (CCDV-P)· Lessons

    Domain 5 · Lesson 15/25

    LLM-as-judge calibration, bias and drift

    LLM-as-Judge Engineering

    19 min read
    4% of exam
    7 sources
    Published 19 Nov 2026
    Docs as of 18 Nov 2026

    What you will be able to do

    • Validate an LLM judge against human-labeled data before trusting its scores at scale
    • Design rubrics with observable anchors and one isolated judge per dimension
    • Measure inter-rater reliability among human labelers first, so the judge is compared against a stable ceiling
    • Detect and correct position, length and self-preference effects by testing whether verdicts depend on things humans do not grade on
    • Explain why judge-human agreement should be chance-corrected rather than reported as raw percent agreement
    • Pin and re-validate a judge model so a model or infrastructure change is not mistaken for a quality change

    Key concept

    Judge calibration against human labels — An LLM judge is a measuring instrument. Its scores mean something only after you have measured how closely they match expert human labels on the same items, and you keep re-measuring that whenever the rubric, the judge model or the traffic changes.

    1.The judge is an instrument you have to validate

    An LLM judge turns free-form output into a number, and every decision downstream inherits that number's errors. Code checks such as exact match are fast and reliable, but they can't grade tone, completeness or whether an answer is faithful. Human graders can, but they are slow and expensive. Model-based grading sits in between: fast, scalable and able to make complex judgments. It comes with one condition, which is that you prove it agrees with humans before you rely on it. For how a judge fits alongside code and human graders in an overall eval, see evaluation system design.

    In practice, you validate the judge before you scale it. Take a set of items that expert humans have already labeled, run the judge on the same items, and measure how far the two diverge. Once agreement is high and stable, human review drops to occasional spot checks. It never disappears. People testing outputs still find failures that a rubric never anticipated. In one Anthropic research system, human testers noticed the agents consistently preferred SEO content farms over authoritative sources, and no automated grader had flagged it.

    A low score doesn't prove the system under test is bad, because the grader can be wrong too. Opus 4.5 initially scored 42% on CORE-Bench. Researchers then found rigid grading that rejected a correctly rounded number, ambiguous task specs, and tasks that couldn't be reproduced. After those were fixed, the score rose to 95%. Before you trust an aggregate, read transcripts alongside their grades and check that each failure is fair: you can see what the output got wrong, not a grader rejecting a valid answer.

    Inter-rater reliability is how consistently independent raters give the same item the same label, and it sets the ceiling for any judge. Have two expert humans label an overlapping sample from the rubric alone, and compare their labels before the judge enters the picture. Where they split, the rubric is ambiguous: tighten its anchors and relabel. A judge that matches the humans at 0.6 when the humans match each other at 0.6 is performing as well as the rubric allows. Once the humans are consistent, resolve their disagreements into one label set and treat the judge as one more rater measured against it. Teams that run human grading in production call this inter-annotator agreement, and it is how human labels earn the right to calibrate a model grader.

    Checkpoint 1 of 9· Check yourself

    A team grades its support bot's free-form answers for tone and completeness with an LLM judge. A release candidate scores 62%, and the team wants to block releases that score below 70%. They have not yet compared the judge against human graders. What should they do before gating on this score?

    Sources123

    2.Rubrics with observable anchors, one dimension per judge

    Most low judge-human agreement starts in the rubric. Score levels described with words like "good", "adequate" or "excellent" let the judge and the humans each read the scale their own way. An anchor should be something a reader can check in the text, for example "every numeric claim is sourced" or "the answer names the company in the first sentence, otherwise it is incorrect". Asking whether something is beautiful gets inconsistent answers. Asking whether it follows stated principles gives the grader a concrete standard.

    Output format matters as much as wording. Restrict the judge to a fixed set of outputs, either a binary correct/incorrect or a defined 1–5 scale, because purely qualitative verdicts can't be aggregated at scale. Ask it to reason before it scores; a grader with thinking enabled performs better on complex judgments. Also give it a way out. An explicit "Unknown" option when the evidence is insufficient stops the judge from inventing a verdict.

    Combining several dimensions into one number hides which dimension failed. If accuracy, tone, completeness and formatting go into a single score, a 3 could mean a factual error or a formatting slip. Engineers can't tell which, and you can't calibrate a dimension you can't see. The fix is structural: write a separate, clearly anchored rubric for each dimension and grade each one in its own isolated judge call. Each judge then gets its own agreement measurement against human labels, and a failure points to one dimension. The tempting patches don't fix this. Stating weights in one prompt still produces one combined number you can't diagnose. Scoring all four dimensions in a single call and keeping only the lowest still mixes the dimensions within one call and throws three of the four scores away. Adding more adjectives to each level makes the vague-wording problem worse.

    A binary LLM grader. Anything other than an exact result tag containing correct counts as incorrect, so an 'Unknown' option would need its own parse branch or it silently becomes a fail.python
    def grade_completion(output, golden_answer):
        grader_message = client.messages.create(
            model="claude-opus-5-5",
            max_tokens=2048,
            messages=[
                {"role": "user", "content": build_grader_prompt(output, golden_answer)}
            ],
        )
        grader_response = next(
            block.text for block in grader_message.content if block.type == "text"
        )
    
        return (
            "correct"
            if "<result>correct</result>" in grader_response.lower()
            else "incorrect"
        )

    Checkpoint 2 of 9· Check yourself

    Your judge prompt scores accuracy, tone, completeness and formatting as one 1–5 number. Agreement with human labels is low, and engineers can't tell why individual items fail. Which change fixes the underlying problem?

    Checkpoint 3 of 9· Exam question

    Saffron Travel runs an assistant that drafts replies to booking-change requests, and its quality team grades samples with a single judge prompt that scores accuracy, tone, completeness and formatting into one 1-5 number. Agreement with human reviewers is low. The prompt already ranks the dimensions by importance, and a previous revision that expanded the scale wording changed nothing. When reviewers dispute a score, they say the judge treats 'good' as meaning different things on different items, and a reply scored 2 might have failed on fare figures, a curt tone, or a missing refund step. Engineers cannot tell why items fail, and they have human labels for each dimension separately that they can compare against. Which two changes should the team make? Select TWO.(Select 2)

    0 of 2 selected

    Sources124

    3.Measuring judge-human agreement

    Measuring agreement treats the judge as one more rater and asks how reliably it matches the humans. That requires a held-out set of items, each labeled by expert humans who worked from the same rubric the judge uses. Run the judge on those items, compare item by item, and look closely at the disagreements: each one shows either a rubric gap, a judge error or a labeling error. Labels from the people closest to the product requirements count most, because they are the ones defining success.

    Raw percent agreement can mislead. If 90% of items are passes, a judge that passes everything agrees with humans 90% of the time while telling you nothing. Chance-corrected statistics such as Cohen's kappa subtract the agreement you would expect from the label distribution alone. That is why teams report them for judge-human comparisons. Run the same check on the humans themselves: if two expert graders often disagree on a dimension, the rubric is ambiguous, and no judge can do better than the humans it is measured against.

    Agreement also depends on the items. A judge validated on easy, clear-cut cases can still be wrong on borderline ones. Keep the labeled set's distribution close to the real mix of questions and difficulty, and add new kinds of cases as the product grows. If the agreement set never changes while the product does, the number you check stops describing the work the judge actually grades.

    Checkpoint 4 of 9· Check yourself

    Six months after launch, a judge still agrees closely with expert human labels on its original calibration set of clear-cut support questions. The product has since added multi-step troubleshooting questions, and engineers keep finding the judge misgrading them. What should the team do about the calibration set?

    Sources125

    4.Position, length and self-preference effects

    A bias in a judge is a verdict that depends on something the human graders don't reward. Three show up often. Position: in pairwise comparison, the answer shown first wins more often than its quality warrants. Length: longer answers score higher regardless of content. Self-preference: an LLM grader is inclined to be generous toward LLM-generated output, and especially toward its own. When agents grade their own work they reliably skew positive, even when the work is mediocre. Separating the generator from the evaluator helps, and tuning a standalone evaluator to be skeptical is much easier than making a generator criticize itself.

    Every one of these effects has the same test: vary the irrelevant attribute, hold quality fixed, and see whether the verdict moves. For position, present each pair in both orders. A win counts only when both orders agree, and a disagreement is a tie, so slot order can't decide the outcome. Setting temperature to 0 makes a biased verdict repeatable without making it correct. Telling the judge to ignore position is an instruction, not a measurement. Switching to a larger model while keeping the same order leaves the bias untested.

    For length, compare the judge's behavior with the humans'. If judge scores track token count much more strongly than human labels do, the rubric is rewarding volume. Fix the cause: rewrite the rubric to credit the required facts and penalize padding. Then re-check agreement with human labels on held-out pairs that are matched for length, where length can't drive the verdict. Truncating responses, or subtracting points per hundred tokens, patches the score with a formula and can penalize long answers that are correct. Raising the bar for long answers and then checking pass rates measures how many items pass, not whether the judge now agrees with humans.

    Testing a judge for effects humans do not grade on
    EffectHow it shows upTest that isolates itFix that addresses the cause
    PositionFirst-slot answer wins too often; verdicts flip when the pair is swappedRun every pair in both ordersCount a win only when both orders agree; disagreements score as ties
    LengthJudge score tracks token count far more than human labels doCompare judge and human scores on length-matched pairsRubric credits required facts and penalizes padding; re-check against human labels
    Self-preferenceGrader is lenient toward LLM output, especially its ownCompare a self-grading setup against human labelsSeparate generator from evaluator; tune the evaluator to be skeptical

    Checkpoint 5 of 9· Check yourself

    An agent builds landing-page designs and then scores each one against the team's design rubric. Its scores stay high, but human reviewers rate many of the same designs as mediocre. Which change addresses the cause of the gap?

    Checkpoint 6 of 9· Exam question

    Heron Robotics runs an evaluation pipeline that ranks responses from its warehouse-assistant models, and the rankings decide which model ships. A pairwise judge compares two candidate answers per prompt. Audit logs show it picks the answer shown in the first slot 68% of the time, and 31% of verdicts flip when the same pair is presented in swapped order. Rerunning the audit with sampling made deterministic produced identical verdicts, flips included. The judge prompt already tells it to ignore answer position, and a trial with a larger judge model showed a similar first-slot preference. The team has no budget for human labeling but needs rankings it can trust and a measurable bias signal. What is the best fix?

    Checkpoint 7 of 9· Exam question

    Indigo Software runs an AI support assistant for its developer-tools customers and uses an LLM judge to score each answer for helpfulness on a 1-5 scale before weekly release decisions. The judge's scores correlate 0.7 with response token count, while human labels on the same items correlate only 0.2 with length. Reviewers who read the highest-scored long answers found repeated restatements and boilerplate. The judge's rubric simply asks for "thorough, comprehensive answers" and never says what a correct answer must contain. Meanwhile, some migration-troubleshooting answers are legitimately long because every step is required, and human raters score those highly. The team wants to reduce the length bias without discarding genuine content. What is the best change?

    Sources14

    5.Judge drift across model versions

    Your agreement numbers are tied to the exact judge model that produced them. From the Claude 4.6 generation on, a model ID such as claude-opus-4-6 is the snapshot itself: Anthropic does not change the weights or configuration of an existing ID, and updates ship under new IDs. Earlier models work differently. On the Claude API, a short name such as claude-sonnet-4-5 is an alias that resolves to the most recent dated snapshot. Pin those as dated IDs, for example claude-haiku-4-5-20251001, or the judge can change underneath a calibration you already finished.

    A CI eval run that sets the judge model separately from the model under test. Note that claude-haiku-4-5 is a pre-4.6 alias, not a pinned snapshot.bash
    claude plugin eval . \
      --trust-plugin \
      --json results.json \
      --threshold 0.8 \
      --model claude-sonnet-5 \
      --judge-model claude-haiku-4-5 \
      --no-publish \
      --max-cost-usd 20

    A pinned ID still doesn't guarantee identical behavior. The serving infrastructure around a model, including the router, safety classifiers and sampling logic, can change and cause minor differences in behavior on an ID that hasn't changed. So a judge that scored a high kappa at launch needs re-validation, not trust: re-run the human-labeled set on a schedule, re-run it every time the judge model changes, and grow it as new intents appear. Treat a judge upgrade like any other model migration. Score the same labeled set with the old and new judge, compare both against the human labels, and switch only once agreement holds. Few-shot examples with detailed score breakdowns in the judge prompt help as well; in one Anthropic harness they aligned the evaluator with the author's preferences and reduced score drift across iterations. Watching judge scores on live traffic falls under online quality measurement.

    Checkpoint 8 of 9· Fill the gap

    In this CI eval run, the agent under test and the grader use different models. Which flag sets the model used by the llm and baseline graders?

    claude plugin eval . \
      --trust-plugin \
      --json results.json \
      --threshold 0.8 \
      --model claude-sonnet-5 \
       ?  claude-haiku-4-5 \
      --no-publish \
      --max-cost-usd 20

    Checkpoint 9 of 9· Check yourself

    Nobody has touched a support bot, but its CI judge scores dropped sharply overnight. The judge is configured as claude-haiku-4-5 and matched human labels closely at launch. What is the right first move?

    Sources674

    Sources

    (7)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “LLM-based grading: Fast and flexible, scalable and suitable for complex judgment. Test to ensure reliability first, then scale.”
      ↩︎ The judge is an instrument you have to validate
      “Purely qualitative evaluations are hard to assess quickly and at scale.”
      ↩︎ Rubrics with observable anchors, one dimension per judge
      “Use a grader model with thinking on, so that it reasons before it produces an evaluation score.”
      ↩︎ Rubrics with observable anchors, one dimension per judge
      “Test to ensure reliability first, then scale.”
      ↩︎ Measuring judge-human agreement
      “Test to ensure reliability first, then scale.”
      ↩︎ Position, length and self-preference effects
      “LLM-based grading: Fast and flexible, scalable and suitable for complex judgment. Test to ensure reliability first, then scale.”
      ↩︎ Exam trap 3
      “LLM-based grading: Fast and flexible, scalable and suitable for complex judgment. Test to ensure reliability first, then scale.”
      ↩︎ Prediction
    2. 2.
      “Opus 4.5 initially scored 42% on CORE-Bench”
      ↩︎ The judge is an instrument you have to validate
      “we do not take eval scores at face value until someone digs into the details of the eval”
      ↩︎ The judge is an instrument you have to validate
      “Used to calibrate model-based graders”
      ↩︎ The judge is an instrument you have to validate
      “To avoid hallucinations, give the LLM a way out”
      ↩︎ Rubrics with observable anchors, one dimension per judge
      “and then grade each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions.”
      ↩︎ Rubrics with observable anchors, one dimension per judge
      “LLM-as-judge graders should be closely calibrated with human experts to gain confidence that there is little divergence between the human grading and model grading.”
      ↩︎ Measuring judge-human agreement
      “The people closest to product requirements and users are best positioned to define success.”
      ↩︎ Measuring judge-human agreement
      “LLM-as-judge graders should be closely calibrated with human experts to gain confidence that there is little divergence between the human grading and model grading.”
      ↩︎ Key concept
      “Inter-annotator agreement”
      ↩︎ Exam trap 1
      “and then grade each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions.”
      ↩︎ Exam trap 2
      “LLM-as-judge graders should be closely calibrated with human experts to gain confidence that there is little divergence between the human grading and model grading.”
      ↩︎ Checkpoint
      “grade each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions”
      ↩︎ Checkpoint
    3. 4.
      “gives Claude something concrete to grade against”
      ↩︎ Rubrics with observable anchors, one dimension per judge
      “the evaluator is still an LLM that is inclined to be generous towards LLM-generated outputs”
      ↩︎ Position, length and self-preference effects
      “agents reliably skew positive when grading their own work”
      ↩︎ Position, length and self-preference effects
      “tuning a standalone evaluator to be skeptical turns out to be far more tractable than making a generator critical of its own work”
      ↩︎ Position, length and self-preference effects
      “I calibrated the evaluator using few-shot examples with detailed score breakdowns.”
      ↩︎ Judge drift across model versions
    4. 5.
      “try to have the distribution in your eval represent ~ the real life distribution of questions and question difficulties”
      ↩︎ Measuring judge-human agreement
    5. 6.
      “Anthropic does not update the weights or configuration of an existing model ID.”
      ↩︎ Judge drift across model versions
      “An alias such as claude-sonnet-4-5 is a convenience pointer that resolves to the most recent dated snapshot for that minor version.”
      ↩︎ Judge drift across model versions
      “Occasionally, infrastructure updates produce minor differences in observable behavior even when the model ID and weights have not changed.”
      ↩︎ Judge drift across model versions
      “An alias such as claude-sonnet-4-5 is a convenience pointer that resolves to the most recent dated snapshot for that minor version.”
      ↩︎ Exam trap 4
      “Occasionally, infrastructure updates produce minor differences in observable behavior even when the model ID and weights have not changed.”
      ↩︎ Exam trap 5

    Ready to test yourself?

    Practise Anthropic Claude Certified Developer – Professional (CCDV-P) in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.