CertSafari
    Anthropic Claude Certified Architect – Foundations (CCAR-F)· Lessons

    Domain 4 · Lesson 19/30

    Explicit review criteria that cut false positives

    Design prompts with explicit criteria to improve precision and reduce false positives

    17 min read
    3.33% of exam
    7 sources
    Published 20 Oct 2026
    Docs as of 14 Oct 2026

    What you will be able to do

    • Explain why instructions such as "be conservative" or "only report high-confidence findings" fail to make a review prompt more precise
    • Write report and skip criteria that name concrete categories, such as bugs and security issues to report and style or local patterns to skip
    • Use XML tags to keep each category's criteria separate inside a long review prompt
    • Define severity levels with a concrete code example for each, so findings are classified the same way every time
    • Decide when to switch off a category with many false positives while its prompt is rewritten

    Key concept

    Explicit categorical criteria — A review prompt that states which kinds of finding to report and which to skip, and where the line between them falls. Words about confidence or caution don't do this, because they leave the model's idea of what counts as an issue unchanged.

    1.Why "be conservative" does not fix a noisy reviewer

    A review prompt produces false positives when the model's idea of what counts as an issue differs from yours. Say a security reviewer flags every string concatenation near a database call, including the ones that only join constants. The model is wrong about which category of code is a problem. Being unsure is not the cause. Fixing that means telling it where the category starts and ends.

    Phrases like "be conservative", "don't nitpick" or "only report high-confidence findings" give the model a mood, not a rule. The model is the one that labelled those constant-only concatenations as injection risks, and it was fairly confident when it did. Asking for more confidence sends the same findings through the same judgment, so the false-positive rate barely changes. Recent models also follow such wording faithfully. They investigate as thoroughly as before and then stay quiet about real findings they rank below the bar, so recall drops and the precision problem stays.

    The fix is to move the bar from adjectives into categories. The vendor's suggested wording for a single-pass reviewer names the outcomes that make a finding reportable: incorrect behavior, a test failure, a misleading result. It also names exactly what may be left out: pure style and naming preferences. A colleague given that sentence knows what to do. A colleague told to "be conservative" has to guess. The prompting guide's golden rule applies directly: if a person with minimal context would be confused by the instruction, Claude will be too.

    Vague instructions versus explicit criteria for the same intent
    Vague instructionExplicit criterionWhat the explicit version settles
    "Check that comments are accurate"Flag a comment only when the behavior it claims contradicts what the code actually doesOutdated wording, typos and terse comments are out of scope. A behavioral mismatch is in scope.
    "Only report important issues"Report any bugs that could cause incorrect behavior, a test failure, or a misleading result; only omit nits like pure style or naming preferencesWhich outcomes make a finding reportable, and which kinds may be dropped
    "Be conservative about security"Flag unscoped database queries and PII in logs or error messagesThe specific patterns that count as a security finding

    Here is the comment criterion applied to a real case. A docstring says a function returns the cached value if one is present and otherwise fetches from the API, but the body calls the API every time. That is a finding. The comment claims a behavior, a cache lookup, that the code does not perform. It doesn't matter that the function still returns correct data, or that no caller is known to rely on the cache. The criterion asks one question: does the claimed behavior contradict the actual behavior? Here it does. Precise criteria work well with recent models because they read prompts literally.

    Checkpoint 1 of 9· Check yourself

    A team's single-pass code reviewer flags too many non-issues, and they want it to filter its own findings. Which line should they add to the review prompt?

    Checkpoint 2 of 9· Exam question

    A prompt engineer tries to fix a noisy security-findings category by adding the line "only report high-confidence findings" to the system prompt. After a week of testing, the false positive rate is essentially unchanged. What is the most likely explanation for why this change failed to improve precision?

    Checkpoint 3 of 9· Exam question

    A reviewer flags a docstring that says "returns the cached value if present, otherwise fetches from the API," but the function under review always calls the API regardless of a cache. Under an explicit-criteria rule that only flags comments contradicted by actual code behavior, should this finding be reported?

    Sources123

    2.Writing what to report and what to skip

    A complete criteria set has two lists. One says what must always be reported. The other says what must never be reported. Anthropic's Code Review product is built this way. By default it reports correctness bugs that would break production and leaves out formatting preferences and missing test coverage. Teams then customize it with a REVIEW.md file that says what to flag, at what severity, and how to report it.

    A REVIEW.md excerpt with an explicit skip list and an always-check listmarkdown
    ## Do not report
    
    - Anything CI already enforces: lint, formatting, type errors
    - Generated files under `src/gen/` and any `*.lock` file
    - Test-only code that intentionally violates production rules
    
    ## Always check
    
    - New API routes have an integration test
    - Log lines don't include email addresses, user IDs, or request bodies
    - Database queries are scoped to the caller's tenant

    Each skip item names a category the model can recognize. It doesn't ask for a confidence level. "Anything CI already enforces" removes a whole class of duplicate noise. "Test-only code that intentionally violates production rules" covers code that looks wrong and is correct on purpose. The always-check items work the same way in the other direction: each one is a pattern the model can test the diff against. Without a list like this, a reviewer tends to wander into style nits, pre-existing issues and scope creep.

    Local patterns are a common source of false positives, and they need a skip rule of their own. Suppose a file is full of legacy functions written in several old naming styles, and a new helper follows the project's documented convention. A reviewer that measures names against the surrounding code will flag the one function that is right. The criterion should judge names against the written project standard and not against the most common style in nearby code. Code Review works this way with CLAUDE.md: it treats the file as project context and flags new violations of it as nits. Real naming defects still get caught, because a name that breaks the documented standard is still a finding.

    Checkpoint 4 of 9· Fill the gap

    Which heading belongs above this list in a team's REVIEW.md?

    ##  ? 
    
    - Anything CI already enforces: lint, formatting, type errors
    - Generated files under `src/gen/` and any `*.lock` file
    - Test-only code that intentionally violates production rules

    Checkpoint 5 of 9· Check yourself

    In a legacy module, most functions use mixed old naming styles. A PR adds one helper named according to the project's documented convention in CLAUDE.md. How should well-written review criteria treat that helper's name?

    Sources456

    3.Keeping each category's criteria separate

    A prompt with security, correctness and style criteria, each with its own include rules and severity examples, can blur categories together. The model may apply a style threshold to a security finding, or treat a style example as a rule for correctness. XML tags prevent this. They help Claude parse prompts that mix instructions, context, examples and variable inputs. Wrap each category in its own descriptively named tag, such as <security_criteria>, <correctness_criteria> and <style_criteria>, and put that category's rules and examples inside it.

    The other layouts all lose these boundaries in some way. Prose paragraphs that each start by naming the category rely on the model to notice where one ends. A single list sorted by severity scatters each category across the prompt. A shared examples block at the end separates every example from the rules it illustrates. Tags keep each category's rules and examples together in one block, and nesting lets you add a tag for each severity level inside a category when the structure has that natural hierarchy. Use the same tag names in every prompt, so the reviewer and anyone maintaining the prompt can find a category's rules quickly.

    Checkpoint 6 of 9· Check yourself

    One review prompt holds security, correctness and style criteria, and each category has its own rules and severity examples. Security findings are now being judged against the style threshold. Which way of laying out the prompt best stops one category's rules from leaking into another?

    Checkpoint 7 of 9· Exam question

    An architect is structuring a long review prompt that defines separate criteria for security, correctness, and style categories, each with its own inclusion rules and severity examples. Which structuring approach best helps Claude apply the right criteria to the right category without cross-contamination?

    Sources2

    4.Anchoring each severity level with a code example

    Severity labels drift unless each level has a definition and an example. Without them, the same race condition might be called critical on one run and medium on the next, and two teams reading the reports will not agree on what "high" means. The fix has two parts. Define each level by its consequences, then attach a concrete code example of each one.

    A severity definition written in terms of consequences and named patternsmarkdown
    ## What Important means here
    
    Reserve Important for findings that would break behavior, leak data,
    or block a rollback: incorrect logic, unscoped database queries, PII
    in logs or error messages, and migrations that aren't backward
    compatible. Style, naming, and refactoring suggestions are Nit at
    most.

    The definition sets the boundary. Examples show where real code falls on either side of it. Few-shot examples steer output by showing instead of telling. For severity, the aim is one example per level so each label is tied to a recognizable kind of code. Code Review's own output shows what such anchors look like: each level is paired with a specific location and defect.

    Severity levels paired with concrete findings, as Code Review reports them
    SeverityMeaningAnchoring example
    🔴 ImportantA bug that should be fixed before mergingsrc/auth/session.ts:142 — Token refresh races with logout, leaving stale sessions active
    🟡 NitA minor issue, worth fixing but not blockingsrc/auth/session.ts:88 — parseExpiry silently returns 0 on malformed input
    🟣 Pre-existingA bug that exists in the codebase but was not introduced by this PRAny defect of either kind that predates the diff under review

    Choose the examples carefully, because recent models pay very close attention to details in examples. If every critical example is about authentication, the model may decide that criticality is about authentication. Make the examples relevant to your codebase and varied enough to cover borderline cases, such as a bug that is only reachable from test code. Wrap them in <example> tags so they are not mistaken for instructions.

    Checkpoint 8 of 9· Match them up

    A team anchors each severity level in its review prompt with one concrete finding. Match each finding to the severity level it anchors.

    Tap a term, then the definition that fits it.

    Sources572

    5.False positives, developer trust, and switching a category off

    Developers judge a reviewer as a whole, not one category at a time. If one category keeps raising flags that turn out to be wrong, developers learn to skim or dismiss the bot's comments, and the accurate security and correctness findings get dismissed along with the noise. That is why Code Review spends effort on verification before it posts anything. After its agents find candidate issues, a separate step checks each one against the actual code to filter out false positives.

    When one category is clearly the noisy one, switch it off for now and fix its prompt. You can do this by adding it to the skip list. That restores trust right away, because every comment that still appears comes from a category that has earned it. Then rewrite the disabled category's criteria in explicit terms, run the new version against a set of real cases with known answers, and turn it back on only when it performs well. The vendor recommends this kind of testing: iterate on prompts against a subset of your evals, meaning test cases with known answers, and check recall or F1 instead of just whether the output looks quieter. Recall is the share of the real issues in those cases that the reviewer reports, so it catches a prompt that got quiet by dropping real findings. F1 is a single score that balances recall against precision, the share of reported findings that are real.

    Once the category is back on, the Code Review analytics dashboard gives a rough live signal. Its Feedback count is the number of review comments that were auto-resolved because someone addressed the issue: Claude resolves a comment's thread when a developer fixes what it flagged. A steady count of auto-resolved comments from the re-enabled category shows developers are acting on them. It does not count findings that were ignored or missed, so it supplements the test against known answers and does not replace it.

    Checkpoint 9 of 9· Check yourself

    After developers began dismissing all of a reviewer's comments, a team added its noisy performance-hints category to the skip list. Security and correctness comments are now trusted again, and the team has rewritten the performance criteria in explicit terms. What should decide when the category goes back on?

    Sources41

    Sources

    (7)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “it may investigate the code just as thoroughly, identify the bugs, and then not report findings it judges to be below your stated bar.”
      ↩︎ Why "be conservative" does not fix a noisy reviewer
      “report any bugs that could cause incorrect behavior, a test failure, or a misleading result; only omit nits like pure style or naming preferences.”
      ↩︎ Why "be conservative" does not fix a noisy reviewer
      “Iterate on prompts against a subset of your evals or test cases to validate recall or F1 score gains.”
      ↩︎ False positives, developer trust, and switching a category off
      “be concrete about where the bar is rather than using qualitative terms like”
      ↩︎ Key concept
      “be concrete about where the bar is rather than using qualitative terms like "important": for example, "report any bugs that could cause incorrect behavior”
      ↩︎ Checkpoint
    2. 2.
      “If they'd be confused, Claude will be too.”
      ↩︎ Why "be conservative" does not fix a noisy reviewer
      “XML tags help Claude parse complex prompts unambiguously, especially when your prompt mixes instructions, context, examples, and variable inputs.”
      ↩︎ Keeping each category's criteria separate
      “Use consistent, descriptive tag names across your prompts.”
      ↩︎ Keeping each category's criteria separate
      “Diverse: Cover edge cases and vary enough that Claude doesn't pick up unintended patterns.”
      ↩︎ Anchoring each severity level with a code example
      “Wrapping each type of content in its own tag (for example, <instructions>, <context>, <input>) reduces misinterpretation.”
      ↩︎ Exam trap 2
      “Wrapping each type of content in its own tag (for example, <instructions>, <context>, <input>) reduces misinterpretation.”
      ↩︎ Checkpoint
    3. 3.
      “It does not silently generalize an instruction from one item to another, and it does not infer requests you didn't make.”
      ↩︎ Why "be conservative" does not fix a noisy reviewer
      “report any bugs that could cause incorrect behavior, a test failure, or a misleading result; only omit nits like pure style or naming preferences.”
      ↩︎ Exam trap 1
    4. 4.
      “By default, Code Review focuses on correctness: bugs that would break production, not formatting preferences or missing test coverage.”
      ↩︎ Writing what to report and what to skip
      “a verification step checks results against actual code behavior to filter out false positives.”
      ↩︎ False positives, developer trust, and switching a category off
      “Feedback: Count of review comments that were auto-resolved because someone addressed the issue.”
      ↩︎ False positives, developer trust, and switching a category off
      “Claude auto-resolves threads when you fix previously flagged issues.”
      ↩︎ False positives, developer trust, and switching a category off
    5. 5.
      “Use it to say what your team wants flagged, at what severity, and how findings are reported.”
      ↩︎ Writing what to report and what to skip
      “Code Review reads it as project context and flags newly introduced violations as nits.”
      ↩︎ Writing what to report and what to skip
      “Reserve Important for findings that would break behavior, leak data,”
      ↩︎ Anchoring each severity level with a code example
      “Code Review reads it as project context and flags newly introduced violations as nits.”
      ↩︎ Exam trap 3
      “Reserve Important for findings that would break behavior, leak data,”
      ↩︎ Exam trap 4
      “🔴 Important | src/auth/session.ts:142 | Token refresh races with logout, leaving stale sessions active”
      ↩︎ Checkpoint
    6. 6.
      “Without a no-fire list, the grader thrashes on style nits, pre-existing issues, and scope creep.”
      ↩︎ Writing what to report and what to skip
    7. 7.
      “Claude 4.x and similar advanced models pay very close attention to details in examples.”
      ↩︎ Anchoring each severity level with a code example

    Spotted a mistake, or was something unclear? Tell us.