CertSafari
    Anthropic Claude Certified Developer – Professional (CCDV-P)· Lessons

    Domain 6 · Lesson 18/25

    Latency engineering for Claude: TTFT, streaming, parallel tools, caching and effort

    Latency Engineering

    19 min read
    4% of exam
    6 sources
    Published 19 Nov 2026
    Docs as of 18 Nov 2026

    What you will be able to do

    • Split a Claude request's latency into time-to-first-token and generation time, and match each part to the latency target that applies to it
    • Choose between streaming and output-length control depending on whether a person is watching the response or a pipeline is measuring throughput
    • Cut agent round trips by running independent tool calls concurrently and returning all their results in one message
    • Use cache breakpoints, the 1-hour TTL and max_tokens: 0 pre-warming to lower TTFT without accidentally invalidating the cache
    • Set effort for each step of a pipeline and use evals to check that quality holds when effort goes down

    Key concept

    Latency profile — A request's latency splits into the wait before the first token (prompt processing plus any thinking) and the time spent generating every token after it. Each optimization works on one of these parts, so first decide which part your target measures, then pick the lever.

    1.Time-to-first-token versus total latency

    A latency target only means something once you know which clock it refers to. Time to first token (TTFT) runs from the moment you send the prompt until the model produces the first token of its response. Total latency keeps running until the last token arrives. Model size, prompt complexity and infrastructure all affect both. They split apart because of output length: a response of 900 tokens can have a sub-second TTFT and still take many seconds to finish.

    TTFT is the number a person feels. It matters most for interactive applications, chatbots and real-time systems, where users expect something to appear quickly. Total latency is the number a pipeline feels. A nightly job that summarizes tickets into a database has nobody watching the screen, so only completion time per item counts toward its throughput target.

    Each half has its own levers. Prompt processing is shortened by a smaller prompt and by reading a cached prefix instead of reprocessing it. Thinking before the first visible token is shortened by lower effort. Generation is shortened by asking for fewer output tokens, and a faster model speeds up all of these. Before changing anything, read a trace and find which segment holds the seconds you need to remove.

    A useful latency profile records TTFT and total latency separately, as percentiles, for each step of the application. An hourly average of request duration hides both effects: a streaming rollout that users like, and a slow tail of requests that blows the target. The instrumentation itself is covered in telemetry and instrumentation. This lesson is about what to change once the profile shows where the time goes.

    Checkpoint 1 of 9· Check yourself

    A team runs two Claude workloads: a live support chat where customers watch replies appear, and a nightly job that summarizes tickets into a database. Which latency number should each workload's target measure?

    Sources12

    2.Streaming moves the first token forward, not the last

    Streaming is the main fix for a blank screen. When you set "stream": true on a Messages request, the response arrives as server-sent events. The stream begins with message_start. Each content block then sends content_block_start, a series of content_block_delta events and content_block_stop. One or more message_delta events follow, and the stream ends with message_stop. The UI can render each text delta as soon as it arrives.

    Streaming text with the Python SDK: each fragment is printed as it arrivespython
    client = anthropic.Anthropic()
    
    with client.messages.stream(
        max_tokens=1024,
        messages=[{"role": "user", "content": "Hello"}],
        model="claude-opus-5-5",
    ) as stream:
        for text in stream.text_stream:
            print(text, end="", flush=True)

    Streaming changes what the user sees. It does not change what the model does. Thinking still runs before the answer text, and it streams as thinking_delta events. With display: "omitted" no thinking text is streamed, but the thinking block still opens and closes before the answer appears. Six seconds of thinking takes six seconds whether or not anyone can watch it. Tool calls also have a gap: by default the API buffers each key and value of a tool's input until it is complete, so the stream can go quiet while the model produces a value.

    Streaming also matters in back-end jobs that never show tokens to anyone. For requests with a large max_tokens, the SDK requires streaming to avoid HTTP timeouts. stream.get_final_message() uses streaming internally and still returns the complete Message object. The reason there is connection reliability, not speed.

    Checkpoint 2 of 9· Fill the gap

    A support chat wants each fragment of Claude's reply to show up on screen as soon as it is generated, instead of after the whole reply is done. Which method completes this sample?

    client = anthropic.Anthropic()
    
    with client.messages. ? (
        max_tokens=1024,
        messages=[{"role": "user", "content": "Hello"}],
        model="claude-opus-5-5",
    ) as stream:
        for text in stream.text_stream:
            print(text, end="", flush=True)

    Sources3

    3.Output length sets total latency

    When total latency is the target, the quickest win is usually to generate fewer tokens. You have two levers. The prompt can ask for less: a short fixed schema, a sentence limit, or a direct instruction to be concise. max_tokens sets a hard limit on the response length, so the model cannot run long.

    These levers do different jobs, and mixing them up causes failures. A prompt that asks for a short format changes how long the answer naturally is. max_tokens only cuts the answer off. Set it close to the expected length and you get rationales that stop mid-word and JSON that fails to parse, so leave headroom and do the shortening in the prompt. Thinking counts toward max_tokens too, even when the thinking content is not returned. A cap that fits the visible answer can still truncate a request that thinks first.

    Checkpoint 3 of 9· Check yourself

    An engineer wants to cut total latency on a ticket-summary job where the model thinks before it answers. Summaries average about 150 tokens, so the engineer lowers max_tokens from 1024 to 150. What is the most likely result?

    Checkpoint 4 of 9· Exam question

    Zephyr Legal runs a production pipeline that summarizes support tickets from its client-portal platform through the Messages API. A nightly job processes the backlog and writes each result to a database; no human watches the run, and nothing downstream reads a summary until the whole record is stored. Each ticket is a distinct case thread that is summarized exactly once per night. Per-ticket total latency is 22 s, and the team has a throughput SLO of 8 s per ticket. Output averages 1,100 tokens, because the current prompt lets the model write free-form prose. Profiling shows the input is small next to the output, and the prompt already has a stable, cached instruction prefix. Which change most directly attacks the SLO?

    Sources14

    4.Parallel tool execution cuts round trips

    In agent traces, the most expensive item is often the number of model turns, not any single turn. Think of a refund agent that calls get_order, then get_customer, then get_shipping_status, then get_policy, each in its own turn. It pays for four model round trips plus four tool calls one after another, even though none of the lookups needs another's output.

    Claude can already ask for several tools at once. A response with stop_reason of tool_use may contain several tool_use blocks in one assistant turn. How you run them is up to you: concurrently with asyncio.gather or Promise.all, sequentially, or a mix. Independent, read-only lookups are usually safe to run in parallel. Tools that have side effects, share state or must run in order may be better run one after another.

    There are two steps to making the agent batch its calls:

    1. Ask for batching in the system prompt, for example: "For maximum efficiency, whenever you need to perform multiple independent operations, invoke all relevant tools simultaneously rather than sequentially." Claude 4 and later models already make parallel calls by default when a request benefits from them, and this instruction makes it more likely.
    2. Run the batch concurrently, then send one tool_result per tool_use block in a single user message, matched by tool_use_id and placed before any text. Sending results back in separate messages teaches the model to stop calling tools in parallel.

    Every call in the batch needs a result. If you run a batch sequentially and stop after a failure, still return a result for each skipped call, marked as an error:

    A tool_result for a call the application chose not to runjson
    {
      "type": "tool_result",
      "tool_use_id": "toolu_02",
      "is_error": true,
      "content": "Not executed: the preceding write_file call failed."
    }

    Checkpoint 5 of 9· Fill the gap

    An agent runs a batch of tool calls one at a time and stops after an earlier call fails. It still has to send a result for the call it skipped, so the model doesn't treat that call as successful. Which field completes this tool_result?

    {
      "type": "tool_result",
      "tool_use_id": "toolu_02",
      " ? ": true,
      "content": "Not executed: the preceding write_file call failed."
    }

    Checkpoint 6 of 9· Exam question

    Alder Logistics runs a production support agent that resolves refunds by calling get_order, get_customer, get_shipping_status and get_policy. Traces show four sequential model turns, each with a ~700 ms tool call, and none of the calls depends on another's output. The tools are read-only and the backend handles concurrent requests comfortably. The team already raised the per-turn output budget, but the turn count did not change. Streaming of the final answer is already live, yet customers still wait seconds before any text appears. Every refund needs all four lookups, so the model must stay free to request them together. Engineers want to remove both the repeated model round trips and the stacked tool waits. Select TWO changes that cut end-to-end latency.(Select 2)

    0 of 2 selected

    Sources5

    5.Cache checkpoints and pre-warming lower TTFT

    Prompt caching speeds up the prompt-processing part of a request. When a request's prefix, up to a cache_control breakpoint, matches a recent entry, the API reuses it instead of processing it again. The prefix covers tools, then system, then messages, in that order. That makes a large stable system prompt or tool set the natural place for a breakpoint. The pricing side of caching is covered in cost architecture at scale.

    Caching has a clear limit. It removes prompt processing, not generation. A report generator that reads a 100k-token cached prefix and then writes 3,000 tokens still spends most of its time writing. For the same reason, putting a breakpoint after a nightly job's unique ticket text only helps a re-run of that exact ticket. The first pass is still limited by its output.

    The cache lifetime has to fit your traffic. The default TTL is 5 minutes, and it is measured from the start of the request that wrote or read the entry, not from the end of its response. If a response takes 4 minutes to stream, the follow-up must start within about a minute or the entry expires. When latency matters and follow-ups may come more than 5 minutes apart, use "ttl": "1h".

    Pre-warming avoids a slow first request. Send a request with max_tokens: 0 at startup or on a schedule. It writes the cache at your breakpoint and returns at once with empty content. Put an explicit breakpoint on the last block the real requests share, not on the placeholder message. Use the same thinking configuration and output_config.effort as real traffic, because those values are rendered into the prompt and a mismatch writes an entry that real requests never hit. Do the same with tool_choice: a change in any of these between calls is enough to cause misses.

    A pre-warm request: the breakpoint is on the shared system prompt and max_tokens is 0python
    # Fire this before users arrive to warm the shared system-prompt cache.
    prewarm = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=0,
        system=[
            {
                "type": "text",
                "text": "You are an expert software engineer with deep knowledge of distributed systems...",
                "cache_control": {"type": "ephemeral"},
            }
        ],
        messages=[{"role": "user", "content": "warmup"}],
    )

    A max_tokens: 0 request is rejected if it also sets stream: true, enables extended thinking, requests structured output, or forces a tool with tool_choice. Each of those implies output that a zero-token budget cannot produce. Message Batches reject it too, because pre-warming targets TTFT and TTFT does not apply to batch processing.

    Checkpoint 7 of 9· Fill the gap

    Before the morning rush, a service sends one request to load its shared system prompt into the cache. The request should return at once and generate no output. Which value completes this pre-warm request?

    # Fire this before users arrive to warm the shared system-prompt cache.
    prewarm = client.messages.create(
        model="claude-opus-5-5",
        max_tokens= ? ,
        system=[
            {
                "type": "text",
                "text": "You are an expert software engineer with deep knowledge of distributed systems...",
                "cache_control": {"type": "ephemeral"},
            }
        ],
        messages=[{"role": "user", "content": "warmup"}],
    )

    Sources6

    6.Effort tuning for each pipeline step

    output_config.effort sets how many tokens Claude spends on a response. On models where adaptive thinking is always on, such as Claude Opus 5.5, effort is the main control for how much the model reasons. It applies to every output token: text, tool calls and their arguments, and thinking. Lower effort therefore also means fewer, terser tool calls.

    Effort levels and the work each one suits
    LevelBehaviourTypical use
    lowMost efficient; significant token savings with some capability reductionSimple tasks that need the best speed, such as subagents
    mediumBalanced; default on Claude Opus 5.5 and Claude Haiku 5.5Agentic tasks balancing speed, cost and performance
    highSpends as many tokens as the task needs; default on most other modelsComplex reasoning, difficult coding, agentic tasks
    xhighExtended capability for long-horizon workAgentic and coding tasks running over 30 minutes
    maxNo constraint on token spendingDeepest reasoning where time and tokens don't matter

    Effort is set per request, so set it per step and not per application. A triage classifier and the resolution step after it need different amounts of reasoning. Run the classifier at low or medium and keep the higher level where reasoning actually improves results. For chat and other latency-sensitive work on Claude Sonnet 5.5, start at medium or low. On speed-critical paths, Claude Haiku 5.5 at low effort is the fastest and cheapest combination.

    Effort and the prompt control different things. Effort barely changes how long replies are, so reply length and tone are set in the prompt. The reverse also holds: a prompt asking the model to "think quickly" does not replace lowering effort. Changing effort mid-conversation would normally break the cache, since effort is rendered into the prompt. Claude Opus 5.5 avoids that by accepting a per-message output_config that preserves the prompt cache.

    Treat lowering effort as a quality change, not just a configuration change. Run an effort sweep on your own evals rather than reusing settings from an earlier model, and step down only where the evals show quality holds. Then roll the change out gradually, watching task accuracy alongside latency percentiles, using the methods in online quality measurement.

    Checkpoint 8 of 9· Match them up

    Match each pipeline step to the effort level it should start from. Remember that effort is set per request, so each step can have its own level.

    Tap a term, then the definition that fits it.

    Checkpoint 9 of 9· Exam question

    Tamarack Utilities runs a customer support agent for outage and billing contacts. Its first step, triage, assigns each incoming message to one of five queues, and a second step then drafts the resolution for complex cases. Triage runs at high effort. A production trace shows 6 s of thinking before the first visible token, and agents complain that the queue label appears late. In an offline test on the team's eval set, quality at lower effort was identical, but that set has not been compared against recent live traffic, and leadership wants queue accuracy protected. The team has already asked the model in the prompt to answer briefly, and the thinking delay did not change. Select TWO appropriate actions.(Select 2)

    0 of 2 selected

    Sources41

    Sources

    (6)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “This metric measures the time it takes for the model to generate the first token of the response, from when the prompt was sent.”
      ↩︎ Time-to-first-token versus total latency
      “The fewer tokens the model has to process and generate, the faster the response will be.”
      ↩︎ Time-to-first-token versus total latency
      “Use the max_tokens parameter to set a hard limit on the maximum length of the generated response.”
      ↩︎ Output length sets total latency
      “Effort is its main control for speed and cost.”
      ↩︎ Effort tuning for each pipeline step
      “The fewer tokens the model has to process and generate, the faster the response will be.”
      ↩︎ Key concept
      “Streaming is a feature that allows the model to start sending back its response before the full output is complete.”
      ↩︎ Exam trap 1
      “This can significantly improve the perceived responsiveness of your application, as users can see the model’s output in real time.”
      ↩︎ Prediction
      “It’s particularly relevant when you’re using streaming (more on that later) and want to provide a responsive experience to your users.”
      ↩︎ Checkpoint
    2. 2.
      “particularly relevant for interactive applications, chatbots, and real-time systems where users expect quick initial feedback”
      ↩︎ Time-to-first-token versus total latency
    3. 3.
      “When creating a Message, you can set "stream": true to incrementally stream the response using server-sent events (SSE).”
      ↩︎ Streaming moves the first token forward, not the last
      “This is especially useful for requests with large max_tokens values, where the SDK requires streaming to avoid HTTP timeouts.”
      ↩︎ Streaming moves the first token forward, not the last
      “By default, the API buffers each key and value in input until it is complete.”
      ↩︎ Streaming moves the first token forward, not the last
    4. 4.
      “Thinking counts toward max_tokens even when the thinking content isn’t returned.”
      ↩︎ Output length sets total latency
      “Because effort applies to every output token, it works with or without thinking. Lower effort also means fewer and terser tool calls.”
      ↩︎ Effort tuning for each pipeline step
      “Claude Opus 5.5 also supports changing effort mid-conversation with a per-message output_config, which preserves the prompt cache.”
      ↩︎ Effort tuning for each pipeline step
      “Thinking counts toward max_tokens even when the thinking content isn’t returned.”
      ↩︎ Exam trap 2
      “Effort barely changes how long the model’s replies are, so use your prompt to change reply length and tone.”
      ↩︎ Exam trap 3
      “effort is the primary control for how much the model reasons and what a request costs”
      ↩︎ Prediction
      “low | Most efficient. Significant token savings with some capability reduction. | Simpler tasks that need the best speed and lowest costs, such as subagents”
      ↩︎ Checkpoint
    5. 5.
      “Independent, read-only operations are usually safe to run in parallel for lower latency.”
      ↩︎ Parallel tool execution cuts round trips
      “return one tool_result for each tool_use block, all together in the next user message.”
      ↩︎ Parallel tool execution cuts round trips
      “For maximum efficiency, whenever you need to perform multiple independent operations, invoke all relevant tools simultaneously rather than sequentially.”
      ↩︎ Parallel tool execution cuts round trips
    6. 6.
      “The lifetime is measured from the start of the request that writes or reads the cache entry, not from the end of its response.”
      ↩︎ Cache checkpoints and pre-warming lower TTFT
      “This can reduce the cache-miss latency penalty on the first user interaction, lowering time-to-first-token (TTFT) for latency-sensitive applications.”
      ↩︎ Cache checkpoints and pre-warming lower TTFT
      “When latency is important and your follow-up prompts might be sent beyond 5 minutes.”
      ↩︎ Cache checkpoints and pre-warming lower TTFT
      “Verify that tool_choice, image usage, the thinking configuration, and output_config.effort remain consistent between calls.”
      ↩︎ Exam trap 4

    Ready to test yourself?

    Practise Anthropic Claude Certified Developer – Professional (CCDV-P) in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.