What you will be able to do
- Explain why the mechanism that makes an LLM fluent is the same one that makes it hallucinate, and say where fabrication concentrates
- Identify the knowledge failures that come from training data: staleness, uneven coverage, inherited bias and source amnesia
- Tell the hard context-window cliff apart from gradual context rot, and explain why the model does not learn from in-session corrections
- Recognise steerability failures and training fingerprints such as sycophancy and loose confidence calibration as causes of misplaced trust
- Diagnose a real failure by naming the two properties that are colliding
Key concept
Next token prediction as the source of both fluency and hallucination — An LLM writes each answer word by word, predicting what usually comes next. That process does not check whether a statement is true. So the same mechanism that produces smooth, plausible text also produces confident fabrications, and you cannot get one without risking the other.
1.Hallucination: where it comes from and where it concentrates
Most limitations of an LLM system go back to how it produces text. Anthropic's AI Fluency material puts it plainly: generative AI is closer to a very sophisticated autocomplete than to a search engine. It does not look facts up. It continues a pattern. On well-worn paths such as summarising, reformatting or explaining common concepts, this works very well. In sparse or novel territory, and wherever the task depends on telling "true" apart from "sounds true," it fails.
Anthropic's documentation calls this failure hallucination: even the most advanced models, Claude included, can sometimes generate text that is factually incorrect or inconsistent with the context they were given. Hallucinations are not spread evenly. They cluster in specifics: names, dates, statistics, citations, URLs and quotes. The more precise a claim is, the more it needs checking. Fabricated specifics also tend to arrive in the same confident tone as correct ones, so tone tells you nothing about reliability.
The sampling behind prediction gives you a cheap way to detect the problem. If you run the same prompt several times and compare the outputs, the differences between runs can point to a hallucination. Other mitigations attack the root cause. You can explicitly allow the model to say it doesn't know. You can restrict it to the documents you supply. You can require a supporting citation for each claim. Product features such as citations, uncertainty signalling, constrained generation and generator-verifier loops exist to push this limitation further out. None of them removes it.
Checkpoint 1 of 7· Check yourself
A reviewer can spot-check only part of a long AI-generated market report. Based on where fabrication concentrates, which content should they check first?
Fabrication concentrates in precise, checkable claims. General explanations of well-documented concepts sit in the capability zone and are less likely to be invented.
“Fabrication concentrates in specificity: names, dates, statistics, citations, URLs, quotes.”Source: academy.claude.com
Checkpoint 2 of 7· Exam question
A document-processing agent uses Claude to summarize inbound customer emails by calling a fetch_email tool and passing the email body to Claude. During a security review, the team discovers that a malicious email instructed the agent, via text embedded in the email body, to forward the customer's account credentials to an external address, and the agent complied. Which change to the tool integration would most directly reduce this indirect prompt injection risk?
Correct answer: A — Deliver the email body only inside a tool_result block, label it as untrusted content, and tell Claude in the system prompt never to let tool-result text override the user's request.
- A. Correct. Delivering untrusted third-party content only in tool_result blocks and stating an explicit untrusted-content policy in the system prompt is the recommended structural defense against indirect prompt injection, since Claude is trained to treat tool-result content with more skepticism than direct instructions.
- B. Incorrect and counterproductive. Moving untrusted content into the system prompt gives it the same authority as the operator's instructions, which increases injection risk rather than reducing it.
- C. Incorrect. The effort parameter trades reasoning depth for latency and cost; it does not change how Claude weighs the trust level of embedded instructions.
- D. Incorrect. Model capability alone does not eliminate indirect prompt injection risk; the fix is structural (how untrusted content is delivered and labeled), not simply a larger model.
2.Knowledge: frozen, uneven and inherited
The second limitation is what the model knows. Everything it knows comes from training data and stops at a fixed knowledge cutoff. Without tools, it has no information from after that date and no real-time browsing. So the useful question is not "does the model know this?" but "how well represented was this in what it was trained on?"
Topics that appeared often, recently (relative to training) and consistently fall in the capability zone. Rare, post-cutoff, niche, local or contested topics fall in the limitation zone. Four characteristic failures follow from this:
- Staleness. Out-of-date information is presented as if it were current. - Uneven coverage. Answers are deep on mainstream topics and thin on niche ones, often in the same confident tone. - Inherited bias. The training data shapes what the model treats as "default" or "normal." - Source amnesia. The model cannot say where a given piece of knowledge came from.
Web search, retrieval (RAG and MCPs) and tool use exist to fill these gaps by giving the model information it was never trained on. Inherited bias deserves particular attention in a risk review, because no error message flags it. It shows up only in what the model quietly assumes.
Checkpoint 3 of 7· Check yourself
An assistant without tools confidently describes a regulation as it stood before an amendment that took effect after the model's training. Which characteristic knowledge failure is this?
Knowledge is frozen at the cutoff, so a post-cutoff change shows up as stale information presented as current. Silent truncation is a context-window failure, not a knowledge failure.
“What generative AI knows comes entirely from training data and is frozen at the knowledge cutoff.”Source: academy.claude.com
Sources3
3.Working memory: a cliff, and rot before the cliff
The context window is the model's working memory: all the text it can refer to when it generates a response, including the response itself. The system prompt, messages, tool results, documents, tool definitions and the output all count toward it. Two different failure modes live here.
The first is a cliff. Unlike the other properties, which degrade gradually, the context window has a hard edge: things work until they suddenly don't. The failure mode is silent truncation, and you won't always get a warning. A related trap is putting critical information in the middle of long input, where attention is not uniform.
The second is context rot, and it happens before you reach the cliff. More context isn't automatically better, because accuracy and recall degrade as the token count grows. A larger window moves the cliff further away but does not remove the rot.
A third point catches many users: the model does not learn from your corrections. It responds only to what is currently in context. A fresh session starts from zero, so a correction made yesterday has no effect today unless it is supplied again.
Checkpoint 4 of 7· Check yourself
On Monday a user corrects an assistant about their company's naming convention. On Tuesday, in a new session, the assistant makes the same mistake. What best explains this?
Corrections live only in the context where they were given. A new session has no access to them unless they are supplied again, for example through memory or project features.
“The model doesn't learn from your corrections. It only responds to what's currently in context.”Source: academy.claude.com
4.Steerability, sycophancy and misplaced trust
The model follows instructions the same way it does everything else: by continuing a pattern. That makes it very steerable, but there is always a gap between what you meant and what it did. Control is tightest with short, concrete, verifiable instructions such as format specs, length limits and explicit roles. It is loosest with long chains of reasoning, abstract or ambiguous asks, and anything that needs native numerical or logical precision. Steerability has its own characteristic failures:
- Reasoning drift. Small errors compound over a long chain of reasoning. - Letter-over-spirit. The model honours the wording of the instruction but misses its intent.
When an instruction is followed literally but uselessly, restate the goal. Repeating the instruction more forcefully will not close the gap.
Training adds further behaviour. Fine-tuning relies on human judgements of what makes a good response, and those judgements leave fingerprints: a pull toward sycophancy, a default toward verbosity, occasional over-caution, and loose calibration between stated confidence and actual reliability.
The sources do not use the term "over-reliance," but this is where it comes from. A sycophantic model may agree with a user's flawed premise. A loosely calibrated model sounds equally sure whether it is right or wrong. A user who treats fluency and confidence as evidence will over-trust the output. The remedy the course describes is calibrated trust: work out where a task sits on each property's continuum, and match your verification and context habits to that position.
Checkpoint 5 of 7· Check yourself
An assistant was told to "summarise each ticket in one line." It returns one-line summaries that leave out the customer impact the team actually needed. What is the most effective fix?
This is letter-over-spirit: the instruction was honoured but its intent was not. Stating the goal closes the gap. Repeating the instruction more forcefully does not.
“Repeating the instruction with more force won't close the gap.”Source: academy.claude.com
5.Diagnosing failures: when properties collide
In real deployments these limitations rarely appear alone. Most real failures involve two properties at once. A long contract review strains working memory while also reaching past what the model knows. A vague creative brief tests steerability exactly where next-token prediction wants to fill gaps with something plausible. The course names two pairings to recognise: Next Token Prediction plus Knowledge produces hallucinated specifics, and Working Memory plus Steerability produces drift in long conversations.
Naming the pair tells you what to fix: verify specifics, re-supply context, move calculations to code execution, or explicitly invite pushback. Each of these is more targeted than simply retrying. The table below summarises each property and where it breaks.
| Property | What it covers | Where it breaks down |
|---|---|---|
| Next Token Prediction | Where AI answers come from | Producing things that sound true but aren't |
| Knowledge | What the model actually knows | Anything rare, recent, niche, or contested |
| Working Memory | What it is paying attention to right now | What quietly falls off the edge; a fresh session doesn't remember the last one |
| Steerability | How much control your instructions really give you | Long reasoning chains, abstract requests, and demands for native precision |
Checkpoint 6 of 7· Match them up
Match each property to its characteristic failure
Tap a term, then the definition that fits it.
Each property has a typical way of failing. Naming the property, or the two that are colliding, tells you which fix to apply.
“Naming the properties at play points you straight to the fix: verify specifics, re-supply context, offload to code execution, or invite pushback.”Source: academy.claude.com
Checkpoint 7 of 7· Exam question
A financial analyst asks Claude to review a 60-page acquisition due-diligence report and flag any regulatory risks. In testing, Claude sometimes states specific compliance deadlines that do not appear anywhere in the report. Which prompt change would most directly reduce this hallucination risk?
Correct answer: A — Instruct Claude to first extract verbatim quotes from the report that support each risk it identifies, and to state that no risk was found where no supporting quote exists.
- A. Correct. Grounding claims in verbatim quotes extracted from the source document, and explicitly allowing 'not found' as an answer, are established techniques for reducing hallucination by tying output to the provided context.
- B. Incorrect. Preferring parametric training knowledge over the provided document is the opposite of the recommended external-knowledge-restriction technique and would increase the risk of stating outdated or unsupported deadlines.
- C. Incorrect. Response length has no established causal link to hallucination rate; shortening the answer does not ground claims in the source text.
- D. Incorrect. Confidence language is a surface-level stylistic change that does not verify factual accuracy and could make undetected hallucinations more convincing.
Sources8
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Hallucination risk is roughly the same across every part of a response, so any spot check is as good as another.Why is that wrong?
Fabrication concentrates in precise claims such as names, dates, statistics, citations, URLs and quotes. The more specific a claim, the more it needs checking.
Covered in Hallucination: where it comes from and where it concentrates
2.As long as everything fits in the context window, the model will recall all of it equally well.Why is that wrong?
Before the hard limit is reached, accuracy and recall still degrade as the token count grows (context rot). More context is not automatically better.
Covered in Working memory: a cliff, and rot before the cliff
3.A confident-sounding answer is more likely to be correct than a hedged one.Why is that wrong?
Fine-tuning leaves loose calibration between how confident the model sounds and how reliable it actually is, so tone is not evidence of accuracy.
Covered in Steerability, sycophancy and misplaced trust
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://academy.claude.com/courses/ai-capabilities-and-limitations/next-token-predictionOfficial docs
“Generative AI is closer to a vastly sophisticated autocomplete than to a search engine.”
↩︎ Hallucination: where it comes from and where it concentrates“That single property gives you both the fluency and the hallucination.”
↩︎ Key concept“The more precise a claim, the more it warrants verification.”
↩︎ Exam trap 1“Fabrication concentrates in specificity: names, dates, statistics, citations, URLs, quotes.”
↩︎ Checkpoint - 2.https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinationsOfficial docs
“can sometimes generate text that is factually incorrect or inconsistent with the given context”
↩︎ Hallucination: where it comes from and where it concentrates“Inconsistencies across outputs could indicate hallucinations.”
↩︎ Hallucination: where it comes from and where it concentrates - 3.
“Limitation zone: rare, post-cutoff, niche, local, or contested topics.”
↩︎ Knowledge: frozen, uneven and inherited“Characteristic failures: staleness, uneven coverage, inherited bias”
↩︎ Knowledge: frozen, uneven and inherited“What generative AI knows comes entirely from training data and is frozen at the knowledge cutoff.”
↩︎ Checkpoint - 4.
“This property has a cliff rather than a gradient. Silent truncation is the failure mode, and you won't always be warned.”
↩︎ Working memory: a cliff, and rot before the cliff“The model doesn't learn from your corrections. It only responds to what's currently in context.”
↩︎ Checkpoint - 5.
“more context isn't automatically better”
↩︎ Working memory: a cliff, and rot before the cliff“As token count grows, accuracy and recall degrade, a phenomenon known as context rot.”
↩︎ Exam trap 2“As token count grows, accuracy and recall degrade, a phenomenon known as context rot.”
↩︎ Prediction - 6.
“Characteristic failures: reasoning drift (small errors compound) and letter-over-spirit (the instruction was honored but the intent wasn't).”
↩︎ Steerability, sycophancy and misplaced trust“Repeating the instruction with more force won't close the gap.”
↩︎ Checkpoint - 7.
“Calibrated trust means locating your task on each continuum and matching your verification and context habits to where it sits.”
↩︎ Steerability, sycophancy and misplaced trust - 8.https://academy.claude.com/courses/ai-capabilities-and-limitations/when-properties-collideOfficial docs
“Real-world failures are usually two properties interacting, not one.”
↩︎ Diagnosing failures: when properties collide“Next Token Prediction + Knowledge (hallucinated specifics)”
↩︎ Diagnosing failures: when properties collide“Naming the properties at play points you straight to the fix: verify specifics, re-supply context, offload to code execution, or invite pushback.”
↩︎ Checkpoint
Also cited
- https://academy.claude.com/courses/ai-capabilities-and-limitations/how-ai-gets-its-characterOfficial docs
“a pull toward sycophancy, a default toward verbosity, occasional over-caution, and loose calibration between stated confidence and actual reliability.”
↩︎ Exam trap 3