Latency Levers: Fast Mode, Streaming, Retrieval Depth and Output Caps

1.What 'latency' means, and the basic levers

Before you can justify a latency trade-off, you need to know which latency matters. Baseline latency is the time the model takes to process the prompt and generate the response. Time to first token (TTFT) is the time from sending the prompt until the first token comes back. TTFT matters most when you stream and want an interface that feels responsive.

The most direct lever is model selection. For speed-critical applications, Claude Haiku 4.5 offers the fastest response times while keeping high intelligence. The second lever is token count: the fewer tokens the model has to process and generate, the faster the response. So keep prompts clear but concise, ask for shorter answers, and set sensible output limits. The third lever, streaming, does not make generation faster. It lets users watch the output arrive, which significantly improves perceived responsiveness.

Sources1

2.Fast mode: paying for output speed, not intelligence

Fast mode is unusual among latency levers because it costs no accuracy: you pay money for speed. It delivers up to 2.5x higher output tokens per second on Claude Opus 5.5, Claude Opus 5 and Claude Opus 4.8. You opt in with speed: "fast" and the fast-mode-2026-02-01 beta header. The gain is in output speed (OTPS), not TTFT, so it shows most clearly with streaming and on long outputs. If your bottleneck is the wait before the first token, fast mode is the wrong fix.

response = client.beta.messages.create(
    model="claude-opus-5-5",
    max_tokens=4096,
    speed="fast",
    betas=["fast-mode-2026-02-01"],
    messages=[
        {"role": "user", "content": "Refactor this module to use dependency injection"}
    ],
)

Fast mode has constraints you should account for. On Claude Opus 5.5 it costs $8 per million input tokens and $40 per million output tokens, and prompt caching multipliers apply on top. Switching between fast and standard speed invalidates the prompt cache, so flipping speed per request throws away the largest free win. It isn't available with the Batch API, which makes sense because batch work is not latency-sensitive anyway. When fast-mode rate limits are hit, the API returns a 429. You can either wait for the short retry-after delay or catch the error and fall back to standard speed.

Sources2

3.Agent runs: elapsed time, task budgets and max_tokens

In long agent loops, wall-clock time and cost come from how much the model searches, re-verifies and tests. Three controls act on this, and they do different jobs.

An elapsed-time clock is the direct latency lever: tell the model that time matters and show it the elapsed time. On three benchmarks, runs took 33% to 69% less time at 28% to 54% lower cost per task, with scores up to 1.9 points lower. That is a real accuracy-for-speed trade, and you justify it when finishing sooner matters more than the last point of score.

A task budget works because the model can see a live token countdown and regulates itself. On SWE-bench Pro, a generous budget cut cost per task by 44% for about 3 points of pass rate, and the tightest budget cut it by 58% for 6 points. Task budgets are in beta (beta header task-budgets-2026-03-13) on the most recent models. Budgets below the 20,000-token floor are rejected, and very tight budgets can produce refusal-like behavior. Set the budget once, on the first request, because a mid-task change invalidates the cache. Above all, the budget is advisory: it steers the model rather than stopping it, so verify adherence on your workload before you rely on it as a cost control.

max_tokens is different. It caps a single response without the model knowing, so the model doesn't economize. Turns that needed the room are simply cut off, and they are still billed. Thinking counts toward max_tokens even when the thinking content isn't returned. Lowering the cap cut cost per attempt but left cost per solved task about the same. So set max_tokens high (64,000 for agentic work, or 128,000 when a cut-off attempt is costly), treat stop_reason: max_tokens as a failure, and use effort and task budgets to save money.

Sources34

4.Retrieval depth and reranking: the same trade-off inside the pipeline

The same reasoning applies before the model is ever called. In a retrieval pipeline, every chunk you pass in adds input tokens, and fewer tokens to process means a faster response. Retrieval quality, though, depends on depth. Anthropic's Contextual Retrieval tests found that passing the top-20 chunks to the model beat the top-10 or top-5, and that reranking beat no reranking.

Reranking cuts both ways. On the model side it helps: it ensures only the most relevant chunks are passed on, which gives better responses and reduces cost and latency because the model is processing less information. On the pipeline side it has a runtime cost: it adds a step, and that step adds a small amount of latency even though the reranker scores all the chunks in parallel. The more chunks you rerank, the better the results tend to be and the higher the latency and cost of the rerank step itself. The guidance here matches the guidance for effort: there is no universal setting, so experiment on your own use case to find the balance. When justifying a retrieval configuration, state the latency you are paying for each added stage or added chunk and the accuracy gain you measured in return.

Sources15

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Fast mode is a lighter model that trades some intelligence for speed, and it reduces time to first token.Why is that wrong?

   Fast mode uses the same model weights, so accuracy is unchanged. You pay premium pricing, and the speed gain is in output tokens per second, not TTFT.

   Covered in Fast mode: paying for output speed, not intelligence

2. 2.Lowering max_tokens is a good way to make an agent spend less.Why is that wrong?

   The model can't see max_tokens, so cut-off turns are wasted but still billed. Keep it high and save money with effort and task budgets, which the model can see.

   Covered in Agent runs: elapsed time, task budgets and max_tokens