CertSafari
    Anthropic Claude Certified Developer – Foundations (CCDV-F) - December 7 onwards· Lessons

    Domain 4 · Lesson 11/24

    Failure-mode diagnosis: tracing Claude failures to their cause

    Failure-Mode Diagnosis

    17 min read
    5% of exam
    7 sources
    Published 18 Nov 2026
    Docs as of 18 Nov 2026

    What you will be able to do

    • Read stop_reason and the trace to name a failure mode (truncation, refusal, hallucination, under-acting, context loss, format drift, inconsistent output)
    • Decide whether a failure starts in the integration layer or in the model's output
    • Choose a recovery that removes the cause instead of hiding the symptom

    Key concept

    Locate the layer before the fix — A bad answer from Claude may come from the model, or from the code that built the request or handled the response. Use the trace and stop_reason to find which layer produced the failure, then apply the fix in that layer.

    1.Start with stop_reason, not the reply text

    Before you judge the wording of a broken reply, look at the stop_reason field. It tells you why generation ended, and that reason often explains the failure on its own. Every successful Messages API response includes this field. It is not an error code: an HTTP error means the request failed, while stop_reason describes a request that succeeded and then ended in a particular way. So a cut-off answer, an empty answer and a declined answer can all reach your application as normal 200 responses.

    When you diagnose, read the field first and the text second. A reply that stops mid-sentence and a reply with no content at all look different on screen, but neither is a model failure until stop_reason says so. A value of max_tokens points at your request configuration. A value of refusal points at a classifier decision. A value of end_turn on an empty reply usually points at how your code formatted the message history. The table lists each value together with how it typically shows up downstream.

    stop_reason values and the symptom each one produces downstream
    stop_reasonWhen it occursTypical downstream symptomCause-level action
    end_turnClaude completed its response naturallyNormal reply, or an empty one if the history pattern told Claude it was doneUse the response; fix message structure if empty
    max_tokensThe response reached your max_tokens limitText cut mid-sentence, JSON that fails to parse, incomplete tool_use blockRaise max_tokens or continue the response
    tool_useClaude is calling a toolLoop stalls if the app treats this as a final answerRun the tool and return the result
    pause_turnA server-tool loop reached its iteration limitAgent appears to stop partway through a taskSend the assistant content back to continue
    refusalClaude declined to respondBlank output with no error raisedRead stop_details and retry on a fallback model
    model_context_window_exceededThe response filled the model's context windowOutput cut off even though max_tokens was not reachedTreat the response as truncated

    Checkpoint 1 of 8· Match them up

    Each symptom below arrived in an HTTP 200 response. Match each stop_reason value to the symptom it typically produces downstream.

    Tap a term, then the definition that fits it.

    Sources1

    2.Truncation that looks like a parsing bug

    A truncated reply often shows up far from its cause, for example as a JSON parse error in your own code. Suppose a pipeline passes Claude's text straight to json.loads. If generation stops at the token cap, the string ends in the middle of a value and the parser reports an unterminated string. The traceback names your parser, but the parser is not the problem. The evidence that matters is a group of failing calls that all show stop_reason: max_tokens and an output length equal to the configured limit. That pattern means the reply never finished.

    The recovery acts on the cause. Raise max_tokens above the largest payload you expect. Then add a check on stop_reason before any parser runs, so that a future truncation is reported as truncation and not as malformed JSON. When the last content block of a truncated response is an incomplete tool_use, continuing does not help: resend the request with a higher limit so the full tool call can be generated.

    Several plausible fixes leave the cap in place. Retrying the same request with the same max_tokens runs into the same limit, and keeping whichever reply happens to fit just hides how often the problem occurs. Setting temperature to 0 changes sampling, not length. Forcing the opening brace or validating against a schema fixes format problems, but a reply that was cut off still fails validation. Truncation is a length problem and needs a length fix.

    Checkpoint 2 of 8· Check yourself

    A nightly extraction job passes Claude's reply straight to json.loads, and about 8% of runs crash with 'Unterminated string'. The trace for every failing call shows stop_reason: max_tokens and output_tokens equal to the configured limit. Which change addresses the cause?

    Checkpoint 3 of 8· Exam question

    A pipeline parses Claude's reply with json.loads. About 3% of calls fail with 'Unterminated string'. Logs for the failing calls all show stop_reason: max_tokens and an output length equal to the configured limit of 512. What should the team do first?

    Sources1

    3.Refusals arrive as successful responses

    Current Claude models include safety classifiers that can decline a request. When one fires, you get a normal response, not an error. Its content is empty, stop_reason is refusal, and a stop_details object names the policy area. A blank panel with no logged error is the typical result: the client renders the empty content because it has no code path for that stop reason. The failure has two parts. The model declined the request, and the integration layer had nothing to show when that happened.

    A classifier refusal: HTTP 200, empty content, and the reason in stop_detailsjson
    {
      "id": "msg_01XFUDYJgAACzvnptvVoYEL",
      "type": "message",
      "role": "assistant",
      "model": "claude-fable-5",
      "content": [],
      "stop_reason": "refusal",
      "stop_details": {
        "type": "refusal",
        "category": "cyber",
        "explanation": "This request was declined because it could enable cyber harm."
      },
      "usage": {
        "input_tokens": 412,
        "output_tokens": 0
      }
    }

    Use the response shape to rule out other diagnoses. If the clause had dropped out of context, or the network had lost the reply, you would not see stop_reason: refusal. If the reply had been cut off, stop_reason would read max_tokens and some text would be present. The fix is a branch in the client that checks stop_reason directly. That branch can show the user a message, send the item to a human reviewer, or retry on a fallback model. Do not branch on the text of explanation, because category and explanation can both be null. If a refusal arrives mid-stream, discard any partial output already received.

    Retry on a different model, because sending the same request to the same model usually earns another refusal. The reasoning_extraction category is an exception that needs a prompt change instead. It is triggered when the prompt asks Claude to write out its internal reasoning, for example as a scratchpad section or a reasoning field in JSON output. That wording might be in a system prompt, a skill or a tool description, so check all three. Monitoring needs the same awareness: because a refusal is a 200, dashboards built on error rates never count it. The mechanics of fallback routing belong to integration resilience.

    Checkpoint 4 of 8· Check yourself

    After a release adds a required "reasoning" field to the JSON output schema in the system prompt, many triage requests come back as HTTP 200 with empty content, stop_reason: refusal and stop_details.category: reasoning_extraction. What is the right recovery?

    Checkpoint 5 of 8· Exam question

    A document-review app sends a contract clause to Claude. The response comes back with stop_reason: refusal and the UI renders a blank panel with no error. Other clauses in the same contract work. What is the best diagnosis and response?

    Sources2

    4.Isolating the integration layer from the model

    The same symptom can come from different layers, so the first diagnostic question is where the bad input came from. Claude acts only on the message history, tools and documents your code sends. If that payload is wrong, the model may respond reasonably to a broken request, and the result still looks like a model failure.

    Here is a worked case. A support bot says the refund window for annual plans is 60 days, but the actual policy is 30. The trace shows the retrieval step returned zero documents and the prompt sent an empty <docs> block. The reply is a hallucination, but the cause is in the integration layer: the model was asked a policy question and given no policy. Your application already knows the hit count before it calls Claude. The direct fix is to check that count in code and route zero-hit queries to a fallback reply or a decline-to-answer prompt. A verification step that re-checks claims against an empty <docs> block has nothing to check against. Extra thinking cannot reason its way to facts that were never provided. A bigger model that recalls the policy from training data is the opposite of grounding. On the prompt side, two instructions back up the gate: tell Claude to use only the provided documents, and give it explicit permission to say it doesn't know.

    Tool-calling failures follow the same logic. An empty end_turn reply right after a tool call often means your code added a text block after the tool_result, which teaches Claude that a user message follows every tool result, so it ends its turn. The fix is to send only the tool_result blocks. If that fails, add a continuation prompt in a new user message rather than retrying with the empty response. Compare that with an agent that only suggests edits because the user asked "can you suggest some changes". The symptom is similar, but the cause is in the prompt, not the message structure. The table lists other tool-use symptoms whose cause is in your code or schema, not in the model's judgement.

    Tool-use symptoms that originate in the integration layer
    SymptomLikely causeFix
    Claude refuses to act on a tool result, or asks the user to confirm instructions that came from itYour own instructions are being delivered inside the tool_result contentSend instructions in a user turn after the tool_result block; keep the tool result to just the data
    Claude calls tool A when you wanted tool BDescription ambiguityDifferentiate tools by WHEN to use them, not only WHAT they do
    Claude calls tools sequentially when parallel would be betterMessage history formattingSend multiple tool_result blocks in ONE user message
    String comparison on tool inputs fails with newer modelsUnicode and forward-slash escaping differs between model versionsParse with json.loads() or JSON.parse()

    Checkpoint 6 of 8· Match them up

    In each trace below, the model reacted sensibly to what your code sent, so the fix belongs in the integration layer. Match each observed behaviour to the change that removes its cause.

    Tap a term, then the definition that fits it.

    Checkpoint 7 of 8· Exam question

    A support bot answers 'What is your refund window for annual plans?' with '60 days, no questions asked.' The company's policy is 30 days. The trace shows the retrieval step returned zero documents, and the prompt passed an empty <docs> block to Claude. What is the most direct fix for the cause?

    Sources314

    5.Cause-level recoveries for model-output failures

    Once the payload checks out, the failure is in the model's output, and each failure mode has its own lever. Choosing the wrong one treats the symptom: a stricter parser does not reduce hallucination, and a lower temperature does not stop drift between formats.

    Hallucination with good documents present calls for grounding. Allow Claude to admit uncertainty, restrict it to the supplied material, and for long documents have it extract word-for-word quotes before it answers. To audit an answer, ask for a citation for each claim. Running the same prompt several times also helps, because disagreement between runs is a signal of hallucination. These techniques are covered in depth in grounding and citation. A claim with no supporting record in the trace has the same cause: "all tests pass" when no test tool ran is an unsupported claim. The trace shows the gap, and the fix is to require evidence for each claim.

    Under-acting is usually a wording problem. Claude follows instructions precisely, so "can you suggest some changes" can return suggestions when you wanted the edits made. Phrase the request as an action ("Change this function…"), or tell Claude in the system prompt to implement changes by default rather than only suggesting them.

    Context loss shows up as instructions or facts that Claude ignores, most often in prompts with large inputs. Placement matters: put long documents near the top, above the query, instructions and examples. A 40-page document placed after the question, with rules buried in the middle, is a layout to fix before you rewrite any rule. Managing context across long sessions is covered in context management.

    Format drift means the structure varies from reply to reply even when the content is fine. Define the exact output format with JSON, XML or a template, and include a worked example for Claude to match. The defensive parsing side is covered in structured output. Inconsistent output needs one more point: lowering temperature makes output more conservative, but even at 0 identical inputs can give different outputs across API calls. A difference between two runs at temperature 0 is therefore not proof of an integration bug. Reproduce with fixed inputs over several runs, and judge the spread, not a single pair.

    Checkpoint 8 of 8· Check yourself

    Retrieval returned the full 60-page supplier contract, and the trace shows it reached the prompt intact. Even so, Claude's summary states a 90-day termination notice period that appears nowhere in the contract. Which change targets the cause of this failure?

    Sources3567

    Sources

    (7)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Unlike errors, which indicate failures in processing your request, stop_reason tells you why Claude completed its response generation.”
      ↩︎ Start with stop_reason, not the reply text
      “Treat the response as truncated.”
      ↩︎ Start with stop_reason, not the reply text
      “Raise max_tokens or continue the response.”
      ↩︎ Truncation that looks like a parsing bug
      “the truncated response contains an incomplete tool use block”
      ↩︎ Truncation that looks like a parsing bug
      “so Claude ends its turn to follow that pattern”
      ↩︎ Isolating the integration layer from the model
      “add a continuation prompt in a new user message rather than retrying with the empty response”
      ↩︎ Isolating the integration layer from the model
      “Check this field to decide whether to use the response as-is, continue the conversation, retry, or fall back to another model.”
      ↩︎ Key concept
      “Raise max_tokens or continue the response.”
      ↩︎ Exam trap 1
      “model_context_window_exceeded | The response filled the model’s context window. | Treat the response as truncated.”
      ↩︎ Checkpoint
      “max_tokens | The response reached your max_tokens limit. | Raise max_tokens or continue the response.”
      ↩︎ Checkpoint
    2. 2.
      “A refusal is a successful HTTP 200 response with stop_reason”
      ↩︎ Refusals arrive as successful responses
      “Branch on stop_reason or stop_details.type, not on content or the inner stop_details fields.”
      ↩︎ Refusals arrive as successful responses
      “In either case, treat any partial output as incomplete and discard it.”
      ↩︎ Refusals arrive as successful responses
      “This category has no recommended fallback model, so change the prompt rather than retry the request.”
      ↩︎ Refusals arrive as successful responses
      “A refusal is an HTTP 200, so monitoring built on error rates or 5xx responses never sees it.”
      ↩︎ Exam trap 2
      “Re-sending a refused request to the same model usually earns another refusal.”
      ↩︎ Exam trap 3
    3. 3.
      “Explicitly instruct Claude to only use information from provided documents and not its general knowledge.”
      ↩︎ Isolating the integration layer from the model
      “Explicitly give Claude permission to admit uncertainty. This technique can drastically reduce false information.”
      ↩︎ Cause-level recoveries for model-output failures
      “Inconsistencies across outputs could indicate hallucinations.”
      ↩︎ Cause-level recoveries for model-output failures
      “For tasks involving long documents (>20k tokens), ask Claude to extract word-for-word quotes first before performing its task.”
      ↩︎ Checkpoint
    4. 4.
      “Claude is trained to treat instructions inside tool results as potentially untrusted third-party content.”
      ↩︎ Isolating the integration layer from the model
      “Move your instructions out of the tool result: send them in a user turn after the tool_result block”
      ↩︎ Checkpoint
    5. 5.
      “By default, implement changes rather than only suggesting them.”
      ↩︎ Cause-level recoveries for model-output failures
      “Place your long documents and inputs near the top of your prompt, above your query, instructions, and examples.”
      ↩︎ Cause-level recoveries for model-output failures
    6. 6.
      “Precisely define the output format you want using JSON, XML, or custom templates so that Claude follows every output formatting element you require.”
      ↩︎ Cause-level recoveries for model-output failures
    7. 7.
      “Even with temperature set to 0, the results will not be fully deterministic, and identical inputs might produce different outputs across API calls.”
      ↩︎ Cause-level recoveries for model-output failures
      “Even with temperature set to 0, the results will not be fully deterministic, and identical inputs might produce different outputs across API calls.”
      ↩︎ Exam trap 4

    Ready to test yourself?

    Practise Anthropic Claude Certified Developer – Foundations (CCDV-F) - December 7 onwards in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.