What you will be able to do
- Explain why review and verification become the bottleneck once agents generate most code
- Roll out an AI code reviewer in shadow mode and say what evidence it must produce before it may block merges
- Turn adjudicated findings into a merge gate that blocks only on precise categories, and read a Code Review check run correctly in CI
- Design an independent reviewer that has not seen the authoring session, and choose the points where humans stay in control
Key concept
Earned blocking authority — An AI reviewer starts as an advisory signal. It may block merges only for the finding categories where its output has been checked against human review and proved precise. Human sign-off stays in place for everything the reviewer has not proved it can catch.
1.When generation outpaces review
Once agents write most of the code, the queue of changes waiting for review grows faster than people can clear it. Anthropic engineers now ship about eight times as much code per quarter as they did in 2021–2025, and Claude writes 80% of it. Claude also takes on much of the reviewing and approving. The constraint moves downstream: once code is cheap to write, the scarce resources are review attention and CI capacity. Anthropic's CI job volume rose 25x in six months.
The tempting response is to let the agent's own judgement stand in for review. Teams that ship fast with agents do not do this. They pair agents with deterministic checks and human governance, especially in regulated domains where a wrong output is a compliance event rather than a typo. The way out of a review backlog is more verification capacity that you can actually trust, not less verification.
Claude Code Review is a managed example of that capacity. Several agents read the diff and the surrounding code in parallel. A verification step then checks each candidate finding against the code and drops false positives. The remaining findings are deduplicated, ranked by severity and posted to the pull request as inline comments plus a check run named Claude Code Review. The review reads code but does not build or run it, so it adds to tests, lint and typecheck and does not replace them.
Checkpoint 1 of 8· Check yourself
A platform team's coding agents now open pull requests much faster than its engineers can review them, and the review queue keeps growing. Which response matches how teams that ship fast with agents handle this?
A review backlog is a capacity problem. The fix is more verification you can trust: deterministic checks, human governance, and an AI reviewer that is measured before it gets any authority. Letting the agent judge its own work, or handing merges to an unmeasured reviewer, clears the queue by giving up verification. Code Review also doesn't build or run code, so it adds to tests and lint and doesn't replace them.
“You can't automate a process unless you have a reliable means of monitoring and verifying the outcome.”Source: claude.com
2.Introducing an AI reviewer in shadow mode
Shadow mode means the AI reviewer runs on the real pull request stream but has no say in what merges. Its findings are logged and later compared with what human reviewers concluded. Humans keep reviewing exactly as before. The point is evidence: for each finding, did a human also raise it, was it a real issue humans missed, or was it noise? Without that comparison, nobody knows how often a block from the reviewer would be wrong.
Run shadow mode on the full stream, not a hand-picked slice. A reviewer measured only on easy PRs tells you nothing about the agent-authored changes that make up most of the queue. Code Review fits this posture by default, because its check run never fails a PR. The admin dashboard shows a daily count of findings by severity and a count of comments that were auto-resolved because a developer fixed the flagged issue. Both are raw material for the comparison. A reviewer you run yourself, in your own CI or as a custom subagent, puts the output wherever you choose, which makes it easier to keep findings away from authors while you measure.
Three shortcuts are tempting during rollout, and each fails for its own reason:
- Making the reviewer a required check on day one gives it blocking power before its precision is known. Every false positive becomes a blocked merge and an author round trip.
- Letting its verdict replace human sign-off on agent-authored PRs removes human review from the very PRs that most need it.
- Tuning the prompt until it matches last quarter's approvals teaches it to agree with past decisions, including the defects that slipped through. One team's first self-correcting loop overfitted by encoding specific cases, and they had to force general principles instead: fix the principle, not the example.
A shadow pilot is not the same as a cost pilot. The documented way to estimate spend is to enable Code Review on a few representative repositories, set a monthly spend limit, and multiply the average cost per review by the number of PRs. That sizes the bill. It says nothing about whether the findings are right. The trigger mode controls how many reviews you pay for, and the review level controls what each one costs: Quick is the lightest pass, fastest and cheapest; Standard, the default, balances depth, speed and cost; Deep is the most thorough pass, and takes the longest and costs the most. A level changes how deep each review looks and what it costs. It does not tell you whether the findings agree with human judgement.
| Review runs setting | What starts a review | Cost effect |
|---|---|---|
| Once after PR creation | PR opened, reopened, or marked ready for review | One review per PR in the common case |
| After every push | Each push to the PR branch | Multiplies cost by the number of pushes |
| Manual (@claude review) | Only a comment command | No reviews on open or push |
| @claude review always (in Once or Manual) | Subscribes the PR to push-triggered reviews | Adds cost per push after the comment |
Checkpoint 2 of 8· Check yourself
An engineering manager enabled Code Review on three representative repositories with a monthly spend limit. They read the average cost per review and multiplied it by monthly PR volume. They now propose making the Claude Code Review result a merge requirement across the org, saying "the pilot went fine." What is the gap in their reasoning?
The documented pilot estimates spend: average cost per review times PR count. It says nothing about whether the findings are right. Earning blocking authority needs a shadow run on the real PR stream, with findings compared against what human reviewers concluded. The review level (Quick, Standard or Deep) trades depth against cost, so rerunning at Deep would still only size a bill.
“Multiply that average by the number of pull requests the repository opens in a month.”Source: code.claude.com
Checkpoint 3 of 8· Exam question
Pebble Retail runs its online storefront and order-fulfillment platform, where coding agents now open about 120 pull requests a day. Human reviewers can clear roughly 60, so the median review wait has reached three days and agent-authored changes pile up in the queue. Engineering leadership wants to add an AI code reviewer to ease the backlog. The team has never run it on its own repositories, so nobody knows how often it flags clean changes or misses real defects. Authors are already frustrated with delays, and the payments code owners require that any control affecting merges be backed by measured evidence. What is the best first rollout step?
Correct answer: B — Run the AI reviewer on the full PR stream in shadow mode, with findings hidden from authors and checked against human review outcomes.
- A. A required check on day one gives an unmeasured reviewer control over merges. Its false positives would block authors, and its misses would pass unseen because humans only see what it flags.
- B. Shadow mode runs the reviewer on real PR traffic without affecting merges or authors. The team can measure it against actual human decisions before it gets any authority.
- C. Replacing human sign-off on agent PRs removes oversight where the volume is highest. Nothing yet shows the reviewer catches what humans catch, so a blocking verdict there is premature.
- D. Matching past approvals teaches the reviewer to repeat human misses and overfits to historical PRs. Letting it auto-approve afterwards skips any check against live traffic.
3.Calibrating against human review before gating merges
The decision to gate rests on precision per finding category, not on overall recall. Take a shadow period in which humans adjudicated 310 AI findings: 190 matched issues humans also raised, 40 were valid issues humans missed, and 80 were noise. Humans also found 90 valid issues the AI never flagged. Precision is 230/310, about 74%, so roughly one block in four would be wrong if every finding blocked. Recall across all 320 valid issues is 230/320, about 72%. Precision for a single category uses the same arithmetic on that category's findings alone: valid findings divided by all adjudicated findings of that kind.
| Category | Findings | Valid | Noise | Precision |
|---|---|---|---|---|
| Unscoped database queries | 60 | 57 | 3 | 95% |
| Migrations that aren't backward compatible | 30 | 28 | 2 | 93% |
| Incorrect logic | 90 | 70 | 20 | 78% |
| PII in logs | 40 | 30 | 10 | 75% |
| Style, naming and refactoring | 90 | 45 | 45 | 50% |
Each of those numbers points to a different decision. Precision of 74% across all findings is too low for a blocking gate, and the table shows why the average misleads: it hides a spread from 95% down to 50%. Code Review defines no precision threshold, so the team sets a bar before it reads the results, based on what a wrong block costs: an author round trip and a delayed merge. With a bar of 90%, unscoped queries and backward-incompatible migrations may block. Incorrect logic and PII in logs stay advisory and keep collecting adjudications. Style and naming findings, at 50%, should be demoted to Nit. A percentage also needs enough findings behind it: 28 valid out of 30 is evidence, while 3 out of 3 is not.
Recall of 72% means the AI misses issues that humans catch, so human review stays on every merge. Low recall is not a reason to keep the AI advisory everywhere, because recall gaps are covered by the human reviewer, not by the gate. Blocking only on findings a human also raised adds nothing: that agreement is unknown when the gate runs, and the human would block anyway. The method is the same as calibrating an LLM judge against human labels, covered in judge calibration against human labels.
REVIEW.md is where calibration becomes configuration. It holds review-only instructions: what to flag, at what severity, and what to skip. Narrow 🔴 Important to the classes of finding that cleared the bar, demote the rest to Nit, skip paths CI already enforces, and require evidence for categories that produced noise. For example: behavior claims need a file:line citation in the source, not an inference from naming. The severity line a gate reads carries counts by severity, not by category, so REVIEW.md is what limits Important, and therefore blocking, to the proven categories. Code Review reads REVIEW.md from the base or default branch, so a pull request cannot loosen its own review.
| Marker | Severity | Meaning | Gate role |
|---|---|---|---|
| 🔴 | Important | A bug that should be fixed before merging | Candidate for blocking once the category is calibrated |
| 🟡 | Nit | A minor issue, worth fixing but not blocking | Advisory |
| 🟣 | Pre-existing | A bug that exists in the codebase but wasn’t introduced by this PR | Advisory |
The gate itself is a job you write, because the check run never fails on its own. The job reads the check run's output, whose last line is a severity line like this one:
<!-- bughunter-severity: {"nit":1,"normal":2,"pre_existing":0} -->The normal key holds the count of Important findings, whatever its name suggests, so this line reports two open Important findings, one Nit and no pre-existing bugs. Fail the job when normal is above zero. Read the check run's title before the counts, because two titles mean there is no fresh count to read:
- A title starting with
Code review not repeated: Code Review kept an earlier result whose Important findings are still open, and this check run has no severity line. The title already says what you need, so no lookup is required: fail the job if you want to gate the merge. - A title ending with
no new changes: no Important finding from the earlier review is still open, so let the job pass without reading the counts. - Any other title: read the severity line, and treat a missing one as no verdict.
gh api repos/OWNER/REPO/check-runs/CHECK_RUN_ID --jq '
(.output.text | split("\n") | last
| scan("^<!-- bughunter-severity: (.*) -->$")[0] | fromjson)?
// error("no severity line")'A missing severity line is the case that catches teams out. If the review failed, timed out, was cancelled or was superseded, the check run has no severity line, and some completed reviews have none either. A gate that treats a missing line as zero findings waves through exactly the PRs that were never reviewed.
Checkpoint 4 of 8· Fill the gap
A merge-gate job reads the Claude Code Review check run with this command. Which jq function belongs in the blank so that a check run with no severity line stops the gate instead of letting the PR through?
gh api repos/OWNER/REPO/check-runs/CHECK_RUN_ID --jq '
(.output.text | split("\n") | last
| scan("^<!-- bughunter-severity: (.*) -->$")[0] | fromjson)?
// ? ("no severity line")'error makes the command exit with a non-zero status when the severity line is missing, so the gate sees no verdict rather than zero findings. Failed, timed-out, cancelled and superseded reviews have no severity line. With empty, the command prints nothing, and with debug, the message goes to stderr. Either way, a gate could read the result as clean and wave through a PR that was never reviewed.
Checkpoint 5 of 8· Exam question
Quarry Robotics runs fleet-control software for warehouse robots, and every merge to its production repository currently needs a human review. After four weeks of running an AI code reviewer in shadow mode, the team adjudicated 310 AI findings: 190 were also raised by human reviewers, 40 were valid issues humans missed, and 80 were noise. Humans also raised 90 valid issues the AI never flagged, mostly design and concurrency problems. The adjudication logs show the noise clustered in a few categories, such as naming and style, while the input-validation and resource-leak categories had almost no false positives. Engineering leadership says a wrongly blocked merge delays robot firmware releases and erodes trust in the tool, but a defect reaching the fleet is costly. Which decision best fits this evidence?
Correct answer: C — Block on finding categories with high adjudicated precision, keep the remaining categories advisory, and keep a human reviewer on each merge.
- A. Blocking on the full set means the 80 noisy findings, about a quarter of the total, would stop merges. The 90 valid issues the AI never flagged show that human review cannot be reduced to spot checks.
- B. Waiting for recall parity delays blocking even where adjudicated precision is already good. Switching on the full set afterwards still lets noisy findings block merges, because recall says nothing about the 80 noise findings.
- C. Adjudicated precision by category shows where a block is safe, and advisory mode keeps noisy categories from stopping merges. The 90 valid issues the AI never flagged mean a human reviewer is still needed.
- D. Gating only on findings humans also raised adds no protection beyond human review. The 40 valid issues humans missed, which are the AI's real contribution, would never block a merge.
4.Independent reviewers, not self-review
A reviewer is only useful if it is independent of the author. A reviewer in the author's session inherits the author's reasoning: it saw why each choice was made, and it tends to agree with it. A bigger model with the same history agrees more fluently. A rule that it must find at least one defect produces invented defects. Three votes over the same context are correlated, not independent. What fixes it is a reviewer that has not seen how the code was written, given only the diff, the spec it was built from, and the test results. How to set up separate sessions and subagents is covered under subagents vs separate sessions.
Independence is enforced more reliably by structure than by a prompt. A custom subagent limited to Read, Grep and Glob cannot edit anything, and Claude Code enforces that limit rather than relying on the prompt to ask for it. Code Review applies the same principle at the service level. Its repository access is read-only, it pushes no commits, and it takes its instructions from the base branch rather than from the change it is judging.
The author chose which checks to run and how to read their output, so its report and the code share the same blind spots. Verification means a separate check, such as CI on the PR or a reviewer with fresh context, reaching its own result from the diff and the evidence.
Checkpoint 6 of 8· Check yourself
Over the last month, a reviewer subagent that shares the authoring agent's session has approved 97% of agent-written PRs, and defects keep reaching production. Which change is most likely to make its reviews catch what the author missed?
The problem is the shared context, not the model's ability. A reviewer that saw how the code was written inherits the author's reasoning. A bigger model in that session just agrees more fluently. A required-defect rule produces invented defects, and votes taken over the same context are correlated. A reviewer that has never seen the authoring session is the one that judges the work on its own.
“conduct an adversarial review of another agent's work”Source: claude.com
Checkpoint 7 of 8· Exam question
Russet Software runs a payments-reconciliation platform in production and recently added an AI reviewer agent to its code pipeline. The reviewer runs inside the same session as the authoring agent, reusing its context, so it can see every prior turn of the author's reasoning. Over the last quarter it approved 97% of the author's changes, and when it did flag something, the comment usually echoed a concern the author had already raised. Post-release audits show escaped defect rates are no better than before the reviewer was added. The team wants to keep one automated review step per change, without adding human reviewers or significantly increasing per-change cost. What change best fixes this?
Correct answer: B — Run the reviewer as a separate session that receives only the diff, the spec and the test results, with none of the authoring conversation.
- A. A larger model with extended thinking still reads the author's reasoning in the shared session. It anchors on that rationale and approves nearly as readily, so the correlated blind spots remain.
- B. A separate session that sees only the diff, spec and test results does not inherit the author's reasoning. The reviewer judges the change on its own evidence, which removes the shared-context bias behind the rubber-stamp approvals.
- C. A rule demanding a defect pushes the reviewer toward invented or trivial findings, while it still reads the author's rationale and anchors on it. It changes the instruction, not the reviewer's independence.
- D. Three passes over the same inherited conversation produce highly correlated verdicts. A majority vote mostly repeats the same anchored approval three times, so false approvals are not meaningfully reduced.
5.Designing human oversight around the AI reviewer
Human oversight works when it is built into the reviewer's controls, not added as a final approval at the end. Code Review gives humans the following controls:
- Who can trigger a review. A comment command starts a review only for someone with write, maintain or admin permission. Comments from bots never start one, so an agent cannot ask for reviews over and over until it gets the result it wants.
- Who can dismiss a finding. A human dismisses a finding by resolving its thread on GitHub. Replying alone does not dismiss it. The next review reads the replies and may resolve the thread or post one follow-up.
- Who sets the rules. When the same kind of finding keeps coming back, a maintainer adds a rule to REVIEW.md. The fix becomes policy rather than an argument repeated on every PR.
Humans should spend their time on judgement, not on mechanical re-checking. In one team's audit loop, experts comment on the agent's reasoning, and those corrections feed back into the instructions rather than being patched one case at a time. A back-test against a golden set then catches regressions before anything ships. What reaches engineers is a short list of the cases the system could not resolve. A review pipeline should end the same way: the AI handles the precise, high-volume categories, and a human reviewer stays on every merge for the rest.
Checkpoint 8 of 8· Match them up
Match each action on a pull request reviewed by Claude Code Review to what it actually does.
Tap a term, then the definition that fits it.
Resolving the thread is what dismisses a finding. A reply is only input for the next review. Bot comments never start a review, so an agent cannot keep asking for reviews until it gets the result it wants. A finding that keeps coming back is handled by a REVIEW.md rule, which is read from the base branch, not by arguing on each PR.
“Dismiss it without a code change: resolve the thread yourself on GitHub. Replying alone doesn’t dismiss a finding.”Source: code.claude.com
Sources
(8)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://claude.com/blog/agentic-coding-is-straining-ci-heres-how-we-scaled-test-impact-analysis-at-anthropicSecondary source
“Claude authors 80% of that code and it also plays a large role in reviewing and approving PRs as well.”
↩︎ When generation outpaces review“Writing code is no longer the constraint, and once PR review gets accelerated, CI starts feeling the pressure.”
↩︎ When generation outpaces review - 2.https://claude.com/blog/claude-code-guide-for-startupsSecondary source
“none of these startups are having agents merge to main and hoping for the best.”
↩︎ When generation outpaces review“fix the principle, not the example”
↩︎ Introducing an AI reviewer in shadow mode“Engineers spend their time on genuinely hard cases rather than the mechanical 80%”
↩︎ Designing human oversight around the AI reviewer“You can't automate a process unless you have a reliable means of monitoring and verifying the outcome.”
↩︎ Key concept“You can't automate a process unless you have a reliable means of monitoring and verifying the outcome.”
↩︎ Checkpoint“conduct an adversarial review of another agent's work”
↩︎ Checkpoint - 3.https://code.claude.com/docs/en/code-reviewOfficial docs
“a verification step checks the candidate findings against the code and filters out false positives.”
↩︎ When generation outpaces review“a review works from reading the diff and the surrounding code, without building or running your code.”
↩︎ When generation outpaces review“The check run doesn’t conclude as a failure.”
↩︎ Introducing an AI reviewer in shadow mode“REVIEW.md from the base or default branch and never from the pull request’s own changes.”
↩︎ Independent reviewers, not self-review“The check run doesn’t conclude as a failure.”
↩︎ Exam trap 1 - 4.https://code.claude.com/docs/en/code-review-costsOfficial docs
“Count of review comments that were auto-resolved because a developer addressed the issue”
↩︎ Introducing an AI reviewer in shadow mode“Turn on Code Review for a few representative repositories with Review runs set to Once after PR creation or Manual (@claude review).”
↩︎ Introducing an AI reviewer in shadow mode“Multiply that average by the number of pull requests the repository opens in a month.”
↩︎ Checkpoint - 5.https://code.claude.com/docs/en/code-review-setupOfficial docs
“Balanced depth, speed, and cost. The default level”
↩︎ Introducing an AI reviewer in shadow mode“The most thorough pass. Takes the longest and costs the most”
↩︎ Introducing an AI reviewer in shadow mode - 6.
“State which classes of finding are Important and which are Nit at most.”
↩︎ Calibrating against human review before gating merges“behavior claims need a file:line citation in the source, not an inference from naming”
↩︎ Calibrating against human review before gating merges - 7.
“The normal key holds the count of Important findings”
↩︎ Calibrating against human review before gating merges“Fail the job if you want to gate the merge.”
↩︎ Calibrating against human review before gating merges“let the job pass without reading the counts”
↩︎ Calibrating against human review before gating merges“Treat that as no verdict rather than as zero findings.”
↩︎ Calibrating against human review before gating merges“Claude checks this with GitHub for each person who comments, and comments from bots don’t start reviews.”
↩︎ Designing human oversight around the AI reviewer“If the same kind of finding keeps coming back, ask a maintainer to add a rule to REVIEW.md.”
↩︎ Designing human oversight around the AI reviewer“Treat that as no verdict rather than as zero findings.”
↩︎ Exam trap 2“Replying alone doesn’t dismiss a finding.”
↩︎ Exam trap 4“Dismiss it without a code change: resolve the thread yourself on GitHub. Replying alone doesn’t dismiss a finding.”
↩︎ Checkpoint - 8.https://code.claude.com/docs/en/best-practicesOfficial docs
“Getting a reviewer that hasn’t seen how the code was written”
↩︎ Independent reviewers, not self-review“that limit is enforced by Claude Code rather than requested in a prompt”
↩︎ Independent reviewers, not self-review“Getting a reviewer that hasn’t seen how the code was written”
↩︎ Exam trap 3