CertSafari
    Anthropic Claude Certified Developer – Professional (CCDV-P)· Lessons

    Domain 8 · Lesson 23/25

    AI code review operations: shadow mode, calibration and merge gates

    Verification and AI Code Review Operations

    19 min read
    2.5% of exam
    8 sources
    Published 19 Nov 2026
    Docs as of 18 Nov 2026

    What you will be able to do

    • Explain why review and verification become the bottleneck once agents generate most code
    • Roll out an AI code reviewer in shadow mode and say what evidence it must produce before it may block merges
    • Turn adjudicated findings into a merge gate that blocks only on precise categories, and read a Code Review check run correctly in CI
    • Design an independent reviewer that has not seen the authoring session, and choose the points where humans stay in control

    Key concept

    Earned blocking authority — An AI reviewer starts as an advisory signal. It may block merges only for the finding categories where its output has been checked against human review and proved precise. Human sign-off stays in place for everything the reviewer has not proved it can catch.

    1.When generation outpaces review

    Once agents write most of the code, the queue of changes waiting for review grows faster than people can clear it. Anthropic engineers now ship about eight times as much code per quarter as they did in 2021–2025, and Claude writes 80% of it. Claude also takes on much of the reviewing and approving. The constraint moves downstream: once code is cheap to write, the scarce resources are review attention and CI capacity. Anthropic's CI job volume rose 25x in six months.

    The tempting response is to let the agent's own judgement stand in for review. Teams that ship fast with agents do not do this. They pair agents with deterministic checks and human governance, especially in regulated domains where a wrong output is a compliance event rather than a typo. The way out of a review backlog is more verification capacity that you can actually trust, not less verification.

    Claude Code Review is a managed example of that capacity. Several agents read the diff and the surrounding code in parallel. A verification step then checks each candidate finding against the code and drops false positives. The remaining findings are deduplicated, ranked by severity and posted to the pull request as inline comments plus a check run named Claude Code Review. The review reads code but does not build or run it, so it adds to tests, lint and typecheck and does not replace them.

    Checkpoint 1 of 8· Check yourself

    A platform team's coding agents now open pull requests much faster than its engineers can review them, and the review queue keeps growing. Which response matches how teams that ship fast with agents handle this?

    Sources123

    2.Introducing an AI reviewer in shadow mode

    Shadow mode means the AI reviewer runs on the real pull request stream but has no say in what merges. Its findings are logged and later compared with what human reviewers concluded. Humans keep reviewing exactly as before. The point is evidence: for each finding, did a human also raise it, was it a real issue humans missed, or was it noise? Without that comparison, nobody knows how often a block from the reviewer would be wrong.

    Run shadow mode on the full stream, not a hand-picked slice. A reviewer measured only on easy PRs tells you nothing about the agent-authored changes that make up most of the queue. Code Review fits this posture by default, because its check run never fails a PR. The admin dashboard shows a daily count of findings by severity and a count of comments that were auto-resolved because a developer fixed the flagged issue. Both are raw material for the comparison. A reviewer you run yourself, in your own CI or as a custom subagent, puts the output wherever you choose, which makes it easier to keep findings away from authors while you measure.

    Three shortcuts are tempting during rollout, and each fails for its own reason:

    • Making the reviewer a required check on day one gives it blocking power before its precision is known. Every false positive becomes a blocked merge and an author round trip.
    • Letting its verdict replace human sign-off on agent-authored PRs removes human review from the very PRs that most need it.
    • Tuning the prompt until it matches last quarter's approvals teaches it to agree with past decisions, including the defects that slipped through. One team's first self-correcting loop overfitted by encoding specific cases, and they had to force general principles instead: fix the principle, not the example.

    A shadow pilot is not the same as a cost pilot. The documented way to estimate spend is to enable Code Review on a few representative repositories, set a monthly spend limit, and multiply the average cost per review by the number of PRs. That sizes the bill. It says nothing about whether the findings are right. The trigger mode controls how many reviews you pay for, and the review level controls what each one costs: Quick is the lightest pass, fastest and cheapest; Standard, the default, balances depth, speed and cost; Deep is the most thorough pass, and takes the longest and costs the most. A level changes how deep each review looks and what it costs. It does not tell you whether the findings agree with human judgement.

    Code Review trigger modes and what each costs in review volume
    Review runs settingWhat starts a reviewCost effect
    Once after PR creationPR opened, reopened, or marked ready for reviewOne review per PR in the common case
    After every pushEach push to the PR branchMultiplies cost by the number of pushes
    Manual (@claude review)Only a comment commandNo reviews on open or push
    @claude review always (in Once or Manual)Subscribes the PR to push-triggered reviewsAdds cost per push after the comment

    Checkpoint 2 of 8· Check yourself

    An engineering manager enabled Code Review on three representative repositories with a monthly spend limit. They read the average cost per review and multiplied it by monthly PR volume. They now propose making the Claude Code Review result a merge requirement across the org, saying "the pilot went fine." What is the gap in their reasoning?

    Checkpoint 3 of 8· Exam question

    Pebble Retail runs its online storefront and order-fulfillment platform, where coding agents now open about 120 pull requests a day. Human reviewers can clear roughly 60, so the median review wait has reached three days and agent-authored changes pile up in the queue. Engineering leadership wants to add an AI code reviewer to ease the backlog. The team has never run it on its own repositories, so nobody knows how often it flags clean changes or misses real defects. Authors are already frustrated with delays, and the payments code owners require that any control affecting merges be backed by measured evidence. What is the best first rollout step?

    Sources3452

    3.Calibrating against human review before gating merges

    The decision to gate rests on precision per finding category, not on overall recall. Take a shadow period in which humans adjudicated 310 AI findings: 190 matched issues humans also raised, 40 were valid issues humans missed, and 80 were noise. Humans also found 90 valid issues the AI never flagged. Precision is 230/310, about 74%, so roughly one block in four would be wrong if every finding blocked. Recall across all 320 valid issues is 230/320, about 72%. Precision for a single category uses the same arithmetic on that category's findings alone: valid findings divided by all adjudicated findings of that kind.

    The same 310 adjudicated findings, split by category
    CategoryFindingsValidNoisePrecision
    Unscoped database queries6057395%
    Migrations that aren't backward compatible3028293%
    Incorrect logic90702078%
    PII in logs40301075%
    Style, naming and refactoring90454550%

    Each of those numbers points to a different decision. Precision of 74% across all findings is too low for a blocking gate, and the table shows why the average misleads: it hides a spread from 95% down to 50%. Code Review defines no precision threshold, so the team sets a bar before it reads the results, based on what a wrong block costs: an author round trip and a delayed merge. With a bar of 90%, unscoped queries and backward-incompatible migrations may block. Incorrect logic and PII in logs stay advisory and keep collecting adjudications. Style and naming findings, at 50%, should be demoted to Nit. A percentage also needs enough findings behind it: 28 valid out of 30 is evidence, while 3 out of 3 is not.

    Recall of 72% means the AI misses issues that humans catch, so human review stays on every merge. Low recall is not a reason to keep the AI advisory everywhere, because recall gaps are covered by the human reviewer, not by the gate. Blocking only on findings a human also raised adds nothing: that agreement is unknown when the gate runs, and the human would block anyway. The method is the same as calibrating an LLM judge against human labels, covered in judge calibration against human labels.

    REVIEW.md is where calibration becomes configuration. It holds review-only instructions: what to flag, at what severity, and what to skip. Narrow 🔴 Important to the classes of finding that cleared the bar, demote the rest to Nit, skip paths CI already enforces, and require evidence for categories that produced noise. For example: behavior claims need a file:line citation in the source, not an inference from naming. The severity line a gate reads carries counts by severity, not by category, so REVIEW.md is what limits Important, and therefore blocking, to the proven categories. Code Review reads REVIEW.md from the base or default branch, so a pull request cannot loosen its own review.

    Code Review severities and how each fits a merge gate
    MarkerSeverityMeaningGate role
    🔴ImportantA bug that should be fixed before mergingCandidate for blocking once the category is calibrated
    🟡NitA minor issue, worth fixing but not blockingAdvisory
    🟣Pre-existingA bug that exists in the codebase but wasn’t introduced by this PRAdvisory

    The gate itself is a job you write, because the check run never fails on its own. The job reads the check run's output, whose last line is a severity line like this one:

    A severity line, the last line of the check run's output texttext
    <!-- bughunter-severity: {"nit":1,"normal":2,"pre_existing":0} -->

    The normal key holds the count of Important findings, whatever its name suggests, so this line reports two open Important findings, one Nit and no pre-existing bugs. Fail the job when normal is above zero. Read the check run's title before the counts, because two titles mean there is no fresh count to read:

    • A title starting with Code review not repeated: Code Review kept an earlier result whose Important findings are still open, and this check run has no severity line. The title already says what you need, so no lookup is required: fail the job if you want to gate the merge.
    • A title ending with no new changes: no Important finding from the earlier review is still open, so let the job pass without reading the counts.
    • Any other title: read the severity line, and treat a missing one as no verdict.
    Reading the severity counts from a Code Review check runbash
    gh api repos/OWNER/REPO/check-runs/CHECK_RUN_ID --jq '
      (.output.text | split("\n") | last
        | scan("^<!-- bughunter-severity: (.*) -->$")[0] | fromjson)?
      // error("no severity line")'

    A missing severity line is the case that catches teams out. If the review failed, timed out, was cancelled or was superseded, the check run has no severity line, and some completed reviews have none either. A gate that treats a missing line as zero findings waves through exactly the PRs that were never reviewed.

    Checkpoint 4 of 8· Fill the gap

    A merge-gate job reads the Claude Code Review check run with this command. Which jq function belongs in the blank so that a check run with no severity line stops the gate instead of letting the PR through?

    gh api repos/OWNER/REPO/check-runs/CHECK_RUN_ID --jq '
      (.output.text | split("\n") | last
        | scan("^<!-- bughunter-severity: (.*) -->$")[0] | fromjson)?
      //  ? ("no severity line")'

    Checkpoint 5 of 8· Exam question

    Quarry Robotics runs fleet-control software for warehouse robots, and every merge to its production repository currently needs a human review. After four weeks of running an AI code reviewer in shadow mode, the team adjudicated 310 AI findings: 190 were also raised by human reviewers, 40 were valid issues humans missed, and 80 were noise. Humans also raised 90 valid issues the AI never flagged, mostly design and concurrency problems. The adjudication logs show the noise clustered in a few categories, such as naming and style, while the input-validation and resource-leak categories had almost no false positives. Engineering leadership says a wrongly blocked merge delays robot firmware releases and erodes trust in the tool, but a defect reaching the fleet is costly. Which decision best fits this evidence?

    Sources67

    4.Independent reviewers, not self-review

    A reviewer is only useful if it is independent of the author. A reviewer in the author's session inherits the author's reasoning: it saw why each choice was made, and it tends to agree with it. A bigger model with the same history agrees more fluently. A rule that it must find at least one defect produces invented defects. Three votes over the same context are correlated, not independent. What fixes it is a reviewer that has not seen how the code was written, given only the diff, the spec it was built from, and the test results. How to set up separate sessions and subagents is covered under subagents vs separate sessions.

    Independence is enforced more reliably by structure than by a prompt. A custom subagent limited to Read, Grep and Glob cannot edit anything, and Claude Code enforces that limit rather than relying on the prompt to ask for it. Code Review applies the same principle at the service level. Its repository access is read-only, it pushes no commits, and it takes its instructions from the base branch rather than from the change it is judging.

    Checkpoint 6 of 8· Check yourself

    Over the last month, a reviewer subagent that shares the authoring agent's session has approved 97% of agent-written PRs, and defects keep reaching production. Which change is most likely to make its reviews catch what the author missed?

    Checkpoint 7 of 8· Exam question

    Russet Software runs a payments-reconciliation platform in production and recently added an AI reviewer agent to its code pipeline. The reviewer runs inside the same session as the authoring agent, reusing its context, so it can see every prior turn of the author's reasoning. Over the last quarter it approved 97% of the author's changes, and when it did flag something, the comment usually echoed a concern the author had already raised. Post-release audits show escaped defect rates are no better than before the reviewer was added. The team wants to keep one automated review step per change, without adding human reviewers or significantly increasing per-change cost. What change best fixes this?

    Sources83

    5.Designing human oversight around the AI reviewer

    Human oversight works when it is built into the reviewer's controls, not added as a final approval at the end. Code Review gives humans the following controls:

    • Who can trigger a review. A comment command starts a review only for someone with write, maintain or admin permission. Comments from bots never start one, so an agent cannot ask for reviews over and over until it gets the result it wants.
    • Who can dismiss a finding. A human dismisses a finding by resolving its thread on GitHub. Replying alone does not dismiss it. The next review reads the replies and may resolve the thread or post one follow-up.
    • Who sets the rules. When the same kind of finding keeps coming back, a maintainer adds a rule to REVIEW.md. The fix becomes policy rather than an argument repeated on every PR.

    Humans should spend their time on judgement, not on mechanical re-checking. In one team's audit loop, experts comment on the agent's reasoning, and those corrections feed back into the instructions rather than being patched one case at a time. A back-test against a golden set then catches regressions before anything ships. What reaches engineers is a short list of the cases the system could not resolve. A review pipeline should end the same way: the AI handles the precise, high-volume categories, and a human reviewer stays on every merge for the rest.

    Checkpoint 8 of 8· Match them up

    Match each action on a pull request reviewed by Claude Code Review to what it actually does.

    Tap a term, then the definition that fits it.

    Sources72

    Sources

    (8)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Claude authors 80% of that code and it also plays a large role in reviewing and approving PRs as well.”
      ↩︎ When generation outpaces review
      “Writing code is no longer the constraint, and once PR review gets accelerated, CI starts feeling the pressure.”
      ↩︎ When generation outpaces review
    2. 2.
      “none of these startups are having agents merge to main and hoping for the best.”
      ↩︎ When generation outpaces review
      “fix the principle, not the example”
      ↩︎ Introducing an AI reviewer in shadow mode
      “Engineers spend their time on genuinely hard cases rather than the mechanical 80%”
      ↩︎ Designing human oversight around the AI reviewer
      “You can't automate a process unless you have a reliable means of monitoring and verifying the outcome.”
      ↩︎ Key concept
      “You can't automate a process unless you have a reliable means of monitoring and verifying the outcome.”
      ↩︎ Checkpoint
      “conduct an adversarial review of another agent's work”
      ↩︎ Checkpoint
    3. 3.
      “a verification step checks the candidate findings against the code and filters out false positives.”
      ↩︎ When generation outpaces review
      “a review works from reading the diff and the surrounding code, without building or running your code.”
      ↩︎ When generation outpaces review
      “The check run doesn’t conclude as a failure.”
      ↩︎ Introducing an AI reviewer in shadow mode
      “REVIEW.md from the base or default branch and never from the pull request’s own changes.”
      ↩︎ Independent reviewers, not self-review
      “The check run doesn’t conclude as a failure.”
      ↩︎ Exam trap 1
    4. 4.
      “Count of review comments that were auto-resolved because a developer addressed the issue”
      ↩︎ Introducing an AI reviewer in shadow mode
      “Turn on Code Review for a few representative repositories with Review runs set to Once after PR creation or Manual (@claude review).”
      ↩︎ Introducing an AI reviewer in shadow mode
      “Multiply that average by the number of pull requests the repository opens in a month.”
      ↩︎ Checkpoint
    5. 5.
      “Balanced depth, speed, and cost. The default level”
      ↩︎ Introducing an AI reviewer in shadow mode
      “The most thorough pass. Takes the longest and costs the most”
      ↩︎ Introducing an AI reviewer in shadow mode
    6. 6.
      “State which classes of finding are Important and which are Nit at most.”
      ↩︎ Calibrating against human review before gating merges
      “behavior claims need a file:line citation in the source, not an inference from naming”
      ↩︎ Calibrating against human review before gating merges
    7. 7.
      “The normal key holds the count of Important findings”
      ↩︎ Calibrating against human review before gating merges
      “Fail the job if you want to gate the merge.”
      ↩︎ Calibrating against human review before gating merges
      “let the job pass without reading the counts”
      ↩︎ Calibrating against human review before gating merges
      “Treat that as no verdict rather than as zero findings.”
      ↩︎ Calibrating against human review before gating merges
      “Claude checks this with GitHub for each person who comments, and comments from bots don’t start reviews.”
      ↩︎ Designing human oversight around the AI reviewer
      “If the same kind of finding keeps coming back, ask a maintainer to add a rule to REVIEW.md.”
      ↩︎ Designing human oversight around the AI reviewer
      “Treat that as no verdict rather than as zero findings.”
      ↩︎ Exam trap 2
      “Replying alone doesn’t dismiss a finding.”
      ↩︎ Exam trap 4
      “Dismiss it without a code change: resolve the thread yourself on GitHub. Replying alone doesn’t dismiss a finding.”
      ↩︎ Checkpoint
    8. 8.
      “Getting a reviewer that hasn’t seen how the code was written”
      ↩︎ Independent reviewers, not self-review
      “that limit is enforced by Claude Code rather than requested in a prompt”
      ↩︎ Independent reviewers, not self-review
      “Getting a reviewer that hasn’t seen how the code was written”
      ↩︎ Exam trap 3

    Ready to test yourself?

    Practise Anthropic Claude Certified Developer – Professional (CCDV-P) in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.