What you will be able to do
- Check a cited claim in three steps: the source exists, the quote appears in it, and the quote supports the claim
- Tell an output's statement that work is done apart from evidence that it was done
- Decide where limited review time goes, based on what an error would cost
1.Accurate means supported by the material
When Claude works from documents you supply, accuracy has a precise meaning: every claim is backed by those documents. Anthropic's customer-support guidance measures correctness against the information given to Claude in context, and it sets a 100% target for that introductory company and product information. The practical question for each claim is "where does this come from?", not "does this seem true?"
The easiest outputs to check are the ones that show where each claim came from. Anthropic suggests making a response auditable by asking Claude to cite quotes and sources for every claim. You can also ask Claude to check its own draft, as in this prompt from the documentation:
After drafting, review each claim in your press release. For each claim, find a direct quote from the documents that supports it. If you can't find a supporting quote for a claim, remove that claim from the press release and mark where it was removed with empty [] brackets.A citation shows where to check. It doesn't show that anyone checked. Anthropic's verification rubric for a cited research brief runs three separate checks on every source, and each one catches a failure the others miss.
| Check | Passes when | Fails when |
|---|---|---|
| LIVE | The cited URL itself fetches as a readable page | It returns 404, is parked, login-walled, paywalled or bot-blocked, or a mirror or search snippet is used in its place |
| VERBATIM | QUOTE_MATCH: the exact quoted string appears on the fetched page | NOT_FOUND: the quoted string is not on the page |
| SUPPORTS CLAIM | SUPPORTS_CLAIM: the passage backs the claim it is cited on | UNSUPPORTED: the passage is tangential, contradicts the claim, or is only a general statement of fact |
The third check matters most, and people skip it most often. A real source with a real quote can still fail to support the sentence it's attached to. The rubric also shuts a tempting shortcut: when a link is dead, you can't confirm the claim from a repost or a search snippet. The cited URL itself has to load.
2.What the output says versus what actually happened
Some outputs report on work: "the file is updated", "the booking is made", "all five sections are covered". Anthropic's agent-evaluation guidance separates the transcript, which is everything the agent said and did, from the outcome, which is the real end state. A flight-booking agent can end by saying the flight is booked. Whether it is booked depends on whether a reservation exists in the database. To evaluate the work, check the outcome. The report is not enough.
Consistency gives you a second signal. Anthropic lists consistency as a success criterion of its own: similar inputs should get semantically similar answers. That makes it a practical test. Ask the same question again, or run the same prompt several times. A fact that changes between runs is one Claude is not reliably grounded on, and that is worth knowing before anyone relies on it.
3.Match the depth of checking to the cost of an error
You rarely have time to verify everything, and Anthropic's criteria don't assume you do. Its example success criteria separate errors by consequence. One target is that 90% of errors should cause only inconvenience, not an egregious error. The docs also note that whether something matters depends on context: citation accuracy can be critical in a medical app and matter far less in a casual chatbot.
For a reviewer with fifteen minutes, that sets the order. Check first the claims the decision depends on and the ones that would do the most damage if wrong. Precise figures, named sources, and anything a reader will act on or repeat come before phrasing and background. Say a single specific number would strengthen your case a lot, and it covers a field you don't know. That number goes to the top of the list, not the bottom.
Who does the checking also depends on the task. Anthropic describes three kinds of grader. Code-based checks are fast and objective, but they only work when there is a clear-cut right answer. Model-based grading is flexible and scalable, but it needs calibration against human graders to be accurate. Expert human review is the gold standard, and it is slow and expensive. Having Claude check Claude can widen coverage, but it doesn't replace a person with subject-matter knowledge where an error would be serious.
An operations analyst asked six specific questions about a supplier's performance and received one flowing summary in reply. She is about to forward it to her manager. What should she do first?
Correct answer: B — Go back to her six questions and confirm that each one is actually answered somewhere in the summary.
- A. Reading for oddities catches things she already suspects. A question that was never answered at all leaves no trace in the text, so it is exactly what this kind of reading misses.
- B. Correct. Completeness is judged against what was asked, not against how the answer reads. A fluent summary can cover four of six questions and still feel finished, so checking question by question is what exposes the gap.
- C. Shortening compresses a summary whose coverage has not yet been established, and it can remove the very answers she has not verified are present.
- D. Her manager does not have her list of six questions, so he cannot tell what is missing, and the check she owns is passed to someone less able to make it.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.If a claim has a citation and the linked page is real, the claim is verified.Why is that wrong?
A citation can point to the wrong kind of source, or carry a quote that has drifted from the original. You still have to confirm that the quoted passage appears in the source and supports that specific claim.
Covered in Accurate means supported by the material
2.When the output says a task is complete, the task is complete.Why is that wrong?
The output's account of the work is separate from the resulting state. Evaluation checks what actually exists or changed.
Covered in What the output says versus what actually happened
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Assess the correctness of general company and product information provided to the user, based on the information provided to Claude in context.”
↩︎ Accurate means supported by the material - 2.https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinationsOfficial docs
“Make Claude's response auditable by having it cite quotes and sources for each of its claims.”
↩︎ Accurate means supported by the material“Inconsistencies across outputs could indicate hallucinations.”
↩︎ What the output says versus what actually happened - 3.
“Do NOT corroborate via mirrors, reposts, or search snippets”
↩︎ Accurate means supported by the material“a quote drifts from the source, a citation leans on a press release instead of the original filing”
↩︎ Exam trap 1 - 4.
“The outcome is the final state in the environment at the end of the trial.”
↩︎ What the output says versus what actually happened“Requires calibration with human graders for accuracy”
↩︎ Match the depth of checking to the cost of an error“The outcome is the final state in the environment at the end of the trial.”
↩︎ Exam trap 2 - 5.
“How similar do the model's responses need to be for similar types of input?”
↩︎ What the output says versus what actually happened“90% of errors would cause inconvenience, not egregious error”
↩︎ Match the depth of checking to the cost of an error“Strong citation accuracy might be critical for medical apps but less so for casual chatbots.”
↩︎ Match the depth of checking to the cost of an error