What you will be able to do
- Explain why instructions such as "be conservative" or "only report high-confidence findings" fail to make a review prompt more precise
- Write report and skip criteria that name concrete categories, such as bugs and security issues to report and style or local patterns to skip
- Use XML tags to keep each category's criteria separate inside a long review prompt
- Define severity levels with a concrete code example for each, so findings are classified the same way every time
- Decide when to switch off a category with many false positives while its prompt is rewritten
Key concept
Explicit categorical criteria — A review prompt that states which kinds of finding to report and which to skip, and where the line between them falls. Words about confidence or caution don't do this, because they leave the model's idea of what counts as an issue unchanged.
1.Why "be conservative" does not fix a noisy reviewer
A review prompt produces false positives when the model's idea of what counts as an issue differs from yours. Say a security reviewer flags every string concatenation near a database call, including the ones that only join constants. The model is wrong about which category of code is a problem. Being unsure is not the cause. Fixing that means telling it where the category starts and ends.
Phrases like "be conservative", "don't nitpick" or "only report high-confidence findings" give the model a mood, not a rule. The model is the one that labelled those constant-only concatenations as injection risks, and it was fairly confident when it did. Asking for more confidence sends the same findings through the same judgment, so the false-positive rate barely changes. Recent models also follow such wording faithfully. They investigate as thoroughly as before and then stay quiet about real findings they rank below the bar, so recall drops and the precision problem stays.
The fix is to move the bar from adjectives into categories. The vendor's suggested wording for a single-pass reviewer names the outcomes that make a finding reportable: incorrect behavior, a test failure, a misleading result. It also names exactly what may be left out: pure style and naming preferences. A colleague given that sentence knows what to do. A colleague told to "be conservative" has to guess. The prompting guide's golden rule applies directly: if a person with minimal context would be confused by the instruction, Claude will be too.
| Vague instruction | Explicit criterion | What the explicit version settles |
|---|---|---|
| "Check that comments are accurate" | Flag a comment only when the behavior it claims contradicts what the code actually does | Outdated wording, typos and terse comments are out of scope. A behavioral mismatch is in scope. |
| "Only report important issues" | Report any bugs that could cause incorrect behavior, a test failure, or a misleading result; only omit nits like pure style or naming preferences | Which outcomes make a finding reportable, and which kinds may be dropped |
| "Be conservative about security" | Flag unscoped database queries and PII in logs or error messages | The specific patterns that count as a security finding |
Here is the comment criterion applied to a real case. A docstring says a function returns the cached value if one is present and otherwise fetches from the API, but the body calls the API every time. That is a finding. The comment claims a behavior, a cache lookup, that the code does not perform. It doesn't matter that the function still returns correct data, or that no caller is known to rely on the cache. The criterion asks one question: does the claimed behavior contradict the actual behavior? Here it does. Precise criteria work well with recent models because they read prompts literally.
Checkpoint 1 of 9· Check yourself
A team's single-pass code reviewer flags too many non-issues, and they want it to filter its own findings. Which line should they add to the review prompt?
Only the third line says where the bar is: it names the outcomes that make a finding worth reporting and the kinds of finding to leave out. The other lines use mood or confidence words, so the model keeps making the same calls that caused the noise.
“be concrete about where the bar is rather than using qualitative terms like "important": for example, "report any bugs that could cause incorrect behavior”Source: platform.claude.com
Checkpoint 2 of 9· Exam question
A prompt engineer tries to fix a noisy security-findings category by adding the line "only report high-confidence findings" to the system prompt. After a week of testing, the false positive rate is essentially unchanged. What is the most likely explanation for why this change failed to improve precision?
Correct answer: C — General confidence language gives the model no concrete rule for what to report, so it still applies the same underlying judgment that produced the false positives before the change.
- A. Incorrect. The unchanged false positive rate is not caused by increased output length or a larger token set. The added qualifier fails to improve precision because it lacks concrete criteria, not because it makes the output longer.
- B. Incorrect. The model does not have a safety-training conflict that forces over-reporting; it does not default to over-reporting risks regardless of instructions. The instruction's vagueness, not a safety override, explains why false positives persisted.
- C. Correct. Vague confidence terms give the model no clear, checkable rule for distinguishing true findings from false positives, so it continues applying the same underlying judgment. Precision only improves when explicit, categorical criteria replace subjective confidence thresholds.
- D. Incorrect. The model's vocabulary does not cause it to ignore instructions containing 'confidence'; the phrase is processed normally, but it fails to change behavior because it does not provide a concrete, actionable constraint.
Checkpoint 3 of 9· Exam question
A reviewer flags a docstring that says "returns the cached value if present, otherwise fetches from the API," but the function under review always calls the API regardless of a cache. Under an explicit-criteria rule that only flags comments contradicted by actual code behavior, should this finding be reported?
Correct answer: A — Report it, since the docstring promises a cache lookup before the API call and the function body goes straight to the API on each invocation.
- A. Correct. The docstring states a concrete, checkable behavior (look in the cache, then call the API), and the function never consults a cache. A comment contradicted by what the code actually does is exactly what the explicit criterion flags, so the finding is reportable.
- B. Incorrect. Whether the returned data is still correct, or the uncached path is merely slower, does not matter to this criterion. It asks only whether the comment's claim is contradicted by the code's behavior, and here the docstring promises a cache that does not exist. A separate performance review does not remove the contradiction.
- C. Incorrect. The criterion does not exclude comments about internal mechanisms. It covers any comment that makes a specific claim the code contradicts. This docstring explicitly describes caching as part of what the function does, so its claim can be checked against the code and is wrong.
- D. Incorrect. The criterion applies when a comment's claim is contradicted by the code's actual behavior. It does not require evidence that a caller relies on the claim. The docstring is wrong as written, so the finding is reportable without tracing callers first.
2.Writing what to report and what to skip
A complete criteria set has two lists. One says what must always be reported. The other says what must never be reported. Anthropic's Code Review product is built this way. By default it reports correctness bugs that would break production and leaves out formatting preferences and missing test coverage. Teams then customize it with a REVIEW.md file that says what to flag, at what severity, and how to report it.
## Do not report
- Anything CI already enforces: lint, formatting, type errors
- Generated files under `src/gen/` and any `*.lock` file
- Test-only code that intentionally violates production rules
## Always check
- New API routes have an integration test
- Log lines don't include email addresses, user IDs, or request bodies
- Database queries are scoped to the caller's tenantEach skip item names a category the model can recognize. It doesn't ask for a confidence level. "Anything CI already enforces" removes a whole class of duplicate noise. "Test-only code that intentionally violates production rules" covers code that looks wrong and is correct on purpose. The always-check items work the same way in the other direction: each one is a pattern the model can test the diff against. Without a list like this, a reviewer tends to wander into style nits, pre-existing issues and scope creep.
Local patterns are a common source of false positives, and they need a skip rule of their own. Suppose a file is full of legacy functions written in several old naming styles, and a new helper follows the project's documented convention. A reviewer that measures names against the surrounding code will flag the one function that is right. The criterion should judge names against the written project standard and not against the most common style in nearby code. Code Review works this way with CLAUDE.md: it treats the file as project context and flags new violations of it as nits. Real naming defects still get caught, because a name that breaks the documented standard is still a finding.
Checkpoint 4 of 9· Fill the gap
Which heading belongs above this list in a team's REVIEW.md?
## ?
- Anything CI already enforces: lint, formatting, type errors
- Generated files under `src/gen/` and any `*.lock` file
- Test-only code that intentionally violates production rulesEach item names a category the reviewer must skip. The list covers duplicate CI noise, generated files and code that is wrong on purpose. It lowers false positives through categories, not through a confidence bar. The always-check list does the opposite job: it names patterns that must always be flagged.
Source: code.claude.comCheckpoint 5 of 9· Check yourself
In a legacy module, most functions use mixed old naming styles. A PR adds one helper named according to the project's documented convention in CLAUDE.md. How should well-written review criteria treat that helper's name?
Local patterns can themselves be legacy or inconsistent, so the criterion uses the written standard as the baseline. Code Review treats CLAUDE.md as project context and flags new violations of it, so the helper that follows the standard is correct. A name that breaks the standard would still be caught.
“Code Review reads it as project context and flags newly introduced violations as nits.”Source: code.claude.com
3.Keeping each category's criteria separate
A prompt with security, correctness and style criteria, each with its own include rules and severity examples, can blur categories together. The model may apply a style threshold to a security finding, or treat a style example as a rule for correctness. XML tags prevent this. They help Claude parse prompts that mix instructions, context, examples and variable inputs. Wrap each category in its own descriptively named tag, such as <security_criteria>, <correctness_criteria> and <style_criteria>, and put that category's rules and examples inside it.
The other layouts all lose these boundaries in some way. Prose paragraphs that each start by naming the category rely on the model to notice where one ends. A single list sorted by severity scatters each category across the prompt. A shared examples block at the end separates every example from the rules it illustrates. Tags keep each category's rules and examples together in one block, and nesting lets you add a tag for each severity level inside a category when the structure has that natural hierarchy. Use the same tag names in every prompt, so the reviewer and anyone maintaining the prompt can find a category's rules quickly.
Checkpoint 6 of 9· Check yourself
One review prompt holds security, correctness and style criteria, and each category has its own rules and severity examples. Security findings are now being judged against the style threshold. Which way of laying out the prompt best stops one category's rules from leaking into another?
A named tag around each category keeps its rules and examples together and marks clearly where it ends, so one category's thresholds don't spill into another. Prose paragraphs rely on the model to notice where a category ends. A severity-sorted list spreads each category across the prompt. A shared examples block separates each example from the rules it illustrates.
“Wrapping each type of content in its own tag (for example, <instructions>, <context>, <input>) reduces misinterpretation.”Source: platform.claude.com
Checkpoint 7 of 9· Exam question
An architect is structuring a long review prompt that defines separate criteria for security, correctness, and style categories, each with its own inclusion rules and severity examples. Which structuring approach best helps Claude apply the right criteria to the right category without cross-contamination?
Correct answer: D — Wrap each category's inclusion rules and severity examples in its own named XML tag, such as <security_criteria> and <correctness_criteria>, inside the review prompt.
- A. Incorrect. Opening sentences are weak markers in a long prompt. Prose paragraphs have no explicit start and end, so rules and examples from adjacent categories can still blur together. Delimited sections keep categories apart much more reliably.
- B. Incorrect. Sorting by severity interleaves rules from all three categories, so the model must rely on a parenthetical label at the end of each bullet to know which category a criterion belongs to. That is the mixing the architect wants to prevent.
- C. Incorrect. Each category's severity examples are tied to its own inclusion rules. Pooling them in one shared block separates examples from the rules they illustrate and invites the model to apply a security example to a style finding, or the reverse.
- D. Correct. Giving each category's rules and examples its own descriptively named XML tag creates explicit boundaries. Rules and examples stay together, which is the recommended way to structure a prompt that combines several sets of instructions.
Sources2
4.Anchoring each severity level with a code example
Severity labels drift unless each level has a definition and an example. Without them, the same race condition might be called critical on one run and medium on the next, and two teams reading the reports will not agree on what "high" means. The fix has two parts. Define each level by its consequences, then attach a concrete code example of each one.
## What Important means here
Reserve Important for findings that would break behavior, leak data,
or block a rollback: incorrect logic, unscoped database queries, PII
in logs or error messages, and migrations that aren't backward
compatible. Style, naming, and refactoring suggestions are Nit at
most.The definition sets the boundary. Examples show where real code falls on either side of it. Few-shot examples steer output by showing instead of telling. For severity, the aim is one example per level so each label is tied to a recognizable kind of code. Code Review's own output shows what such anchors look like: each level is paired with a specific location and defect.
| Severity | Meaning | Anchoring example |
|---|---|---|
| 🔴 Important | A bug that should be fixed before merging | src/auth/session.ts:142 — Token refresh races with logout, leaving stale sessions active |
| 🟡 Nit | A minor issue, worth fixing but not blocking | src/auth/session.ts:88 — parseExpiry silently returns 0 on malformed input |
| 🟣 Pre-existing | A bug that exists in the codebase but was not introduced by this PR | Any defect of either kind that predates the diff under review |
Choose the examples carefully, because recent models pay very close attention to details in examples. If every critical example is about authentication, the model may decide that criticality is about authentication. Make the examples relevant to your codebase and varied enough to cover borderline cases, such as a bug that is only reachable from test code. Wrap them in <example> tags so they are not mistaken for instructions.
"Seriously" is a qualitative word, so each reviewer supplies their own threshold. A definition that names consequences, such as broken behavior, data leaks or a blocked rollback, plus a code example of each, gives everyone the same test to apply.
Checkpoint 8 of 9· Match them up
A team anchors each severity level in its review prompt with one concrete finding. Match each finding to the severity level it anchors.
Tap a term, then the definition that fits it.
Each label is tied to a recognizable kind of code, so every run applies the same threshold and doesn't make up its own. The race that leaves stale sessions active breaks behavior, so it is Important. The silent bad return is worth fixing but doesn't block the merge, so it is a Nit. Anything that predates the diff is Pre-existing, whatever its size.
“🔴 Important | src/auth/session.ts:142 | Token refresh races with logout, leaving stale sessions active”Source: code.claude.com
5.False positives, developer trust, and switching a category off
Developers judge a reviewer as a whole, not one category at a time. If one category keeps raising flags that turn out to be wrong, developers learn to skim or dismiss the bot's comments, and the accurate security and correctness findings get dismissed along with the noise. That is why Code Review spends effort on verification before it posts anything. After its agents find candidate issues, a separate step checks each one against the actual code to filter out false positives.
When one category is clearly the noisy one, switch it off for now and fix its prompt. You can do this by adding it to the skip list. That restores trust right away, because every comment that still appears comes from a category that has earned it. Then rewrite the disabled category's criteria in explicit terms, run the new version against a set of real cases with known answers, and turn it back on only when it performs well. The vendor recommends this kind of testing: iterate on prompts against a subset of your evals, meaning test cases with known answers, and check recall or F1 instead of just whether the output looks quieter. Recall is the share of the real issues in those cases that the reviewer reports, so it catches a prompt that got quiet by dropping real findings. F1 is a single score that balances recall against precision, the share of reported findings that are real.
Once the category is back on, the Code Review analytics dashboard gives a rough live signal. Its Feedback count is the number of review comments that were auto-resolved because someone addressed the issue: Claude resolves a comment's thread when a developer fixes what it flagged. A steady count of auto-resolved comments from the re-enabled category shows developers are acting on them. It does not count findings that were ignored or missed, so it supplements the test against known answers and does not replace it.
Checkpoint 9 of 9· Check yourself
After developers began dismissing all of a reviewer's comments, a team added its noisy performance-hints category to the skip list. Security and correctness comments are now trusted again, and the team has rewritten the performance criteria in explicit terms. What should decide when the category goes back on?
Switching a category off is a temporary way to protect trust while its criteria are fixed. It goes back on once the new criteria pass a test against cases with known answers, judged on recall or F1. A quieter output alone doesn't show the fix works. Confidence wording doesn't fix the criteria. Leaving the category off for good treats a temporary measure as the solution.
“Iterate on prompts against a subset of your evals or test cases to validate recall or F1 score gains.”Source: platform.claude.com
Sources
(7)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-4-8Official docs
“it may investigate the code just as thoroughly, identify the bugs, and then not report findings it judges to be below your stated bar.”
↩︎ Why "be conservative" does not fix a noisy reviewer“report any bugs that could cause incorrect behavior, a test failure, or a misleading result; only omit nits like pure style or naming preferences.”
↩︎ Why "be conservative" does not fix a noisy reviewer“Iterate on prompts against a subset of your evals or test cases to validate recall or F1 score gains.”
↩︎ False positives, developer trust, and switching a category off“be concrete about where the bar is rather than using qualitative terms like”
↩︎ Key concept“be concrete about where the bar is rather than using qualitative terms like "important": for example, "report any bugs that could cause incorrect behavior”
↩︎ Checkpoint - 2.https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practicesOfficial docs
“If they'd be confused, Claude will be too.”
↩︎ Why "be conservative" does not fix a noisy reviewer“XML tags help Claude parse complex prompts unambiguously, especially when your prompt mixes instructions, context, examples, and variable inputs.”
↩︎ Keeping each category's criteria separate“Use consistent, descriptive tag names across your prompts.”
↩︎ Keeping each category's criteria separate“Diverse: Cover edge cases and vary enough that Claude doesn't pick up unintended patterns.”
↩︎ Anchoring each severity level with a code example“Wrapping each type of content in its own tag (for example, <instructions>, <context>, <input>) reduces misinterpretation.”
↩︎ Exam trap 2“Wrapping each type of content in its own tag (for example, <instructions>, <context>, <input>) reduces misinterpretation.”
↩︎ Checkpoint - 3.https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-sonnet-5Official docs
“It does not silently generalize an instruction from one item to another, and it does not infer requests you didn't make.”
↩︎ Why "be conservative" does not fix a noisy reviewer“report any bugs that could cause incorrect behavior, a test failure, or a misleading result; only omit nits like pure style or naming preferences.”
↩︎ Exam trap 1 - 4.
“By default, Code Review focuses on correctness: bugs that would break production, not formatting preferences or missing test coverage.”
↩︎ Writing what to report and what to skip“a verification step checks results against actual code behavior to filter out false positives.”
↩︎ False positives, developer trust, and switching a category off“Feedback: Count of review comments that were auto-resolved because someone addressed the issue.”
↩︎ False positives, developer trust, and switching a category off“Claude auto-resolves threads when you fix previously flagged issues.”
↩︎ False positives, developer trust, and switching a category off - 5.https://code.claude.com/docs/en/code-reviewOfficial docs
“Use it to say what your team wants flagged, at what severity, and how findings are reported.”
↩︎ Writing what to report and what to skip“Code Review reads it as project context and flags newly introduced violations as nits.”
↩︎ Writing what to report and what to skip“Reserve Important for findings that would break behavior, leak data,”
↩︎ Anchoring each severity level with a code example“Code Review reads it as project context and flags newly introduced violations as nits.”
↩︎ Exam trap 3“Reserve Important for findings that would break behavior, leak data,”
↩︎ Exam trap 4“🔴 Important | src/auth/session.ts:142 | Token refresh races with logout, leaving stale sessions active”
↩︎ Checkpoint - 6.
“Without a no-fire list, the grader thrashes on style nits, pre-existing issues, and scope creep.”
↩︎ Writing what to report and what to skip - 7.https://claude.com/blog/best-practices-for-prompt-engineeringSecondary source
“Claude 4.x and similar advanced models pay very close attention to details in examples.”
↩︎ Anchoring each severity level with a code example