Accuracy-Latency Trade-offs: Model Choice and Effort

1.Say which latency you are optimising

Before you can justify a configuration, you have to say which latency you mean. Anthropic's latency guidance defines latency as the time the model takes to process a prompt and generate an output. Three things shape it: the size of the model, how complex the prompt is, and the infrastructure between the model and the user.

The guidance names two measurements. Baseline latency is the overall time the model takes to process the prompt and produce a response. It gives you a rough sense of the model's speed. Time to first token (TTFT) is the time from sending the prompt to receiving the first token of the response. TTFT is what users notice when output is streamed, because it decides how long they look at a blank screen.

The distinction matters because each lever moves a different metric. Some levers shorten total generation time. Others only change when the first visible text appears. A justification that says "this makes it faster" without naming the metric is incomplete, because a lever can improve one metric and leave the other untouched.

Sources1

2.Two kinds of lever: trades and free wins

Anthropic's cost-and-intelligence guide pictures configuration as a frontier. Some levers exchange capability for savings. Others remove waste and leave quality alone. The guide frames this in terms of cost, but the same split works for latency. Its list of real trade-offs is model choice, effort, output caps and task budgets, an elapsed-time clock, and multi-model architectures. Each of these makes a response cheaper or faster by making it less thorough. Its free wins, such as prompt caching and token hygiene, cut spend without touching quality.

That gives you a working rule. When you pull a trade-off lever, you need a measurement showing that quality held. When you pull a free lever, you don't. A scenario that asks you to cut latency "without affecting accuracy" is pointing you at the free group. A scenario that accepts some loss of capability is inviting the trade-offs, and the right answer usually names how you would check the loss.

Sources2

3.Model choice: speed-first or capability-first

The latency guidance calls model choice one of the most direct levers. For speed-critical applications it recommends Claude Haiku 4.5 as the fastest option that still keeps high intelligence.

client = anthropic.Anthropic()

# For time-sensitive applications, use Claude Haiku 4.5
message = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=100,
    messages=[
        {
            "role": "user",
            "content": "Summarize this customer feedback in 2 sentences: [feedback text]",
        }
    ],
)
print(message.content[0].text)

The model-selection guide gives two ways to start. Speed-first means you begin with Claude Haiku 4.5, test thoroughly, and upgrade only when you find a specific capability gap. It suits prototyping, tight latency requirements, cost-sensitive deployments and high-volume straightforward tasks. Capability-first means you implement with Claude Opus 5.5, optimise your prompts for it, and then gain efficiency over time by lowering effort or moving to a smaller model. If evals at xhigh or max effort still fall short on demanding reasoning or long agentic work, you step up to Claude Fable 5.1. Capability-first suits complex reasoning, nuanced understanding and applications where accuracy matters more than cost. The guide also notes that most workloads start with Claude Opus 5.5.

Sources13

4.Effort: the fine-grained dial inside one model

Model choice is the coarse dial and effort is the fine one. Effort sets how many tokens Claude spends on a response. That covers the text, tool calls and their arguments, and thinking when it is active. Lower effort also produces fewer and terser tool calls, which matters in agent loops where every call is a round trip. The model-selection guide says outright that tuning effort is often a better lever than switching models.

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=4096,
    messages=[
        {
            "role": "user",
            "content": "Analyze the trade-offs between microservices and monolithic architectures",
        }
    ],
    output_config={"effort": "medium"},
)

Effort also steers thinking. Thinking is adaptive: Claude decides for each request whether to reason and how deeply, and effort is the main control over that decision. Thinking improves results on maths, coding and long agentic work, but it has a cost. Reasoning tokens are billed as output tokens and count toward max_tokens, so high effort adds real latency. If you want Claude to think less often, lower effort before you try steering it with the prompt. On Claude Opus 5.5, adaptive thinking is always on and cannot be disabled, which makes effort the main lever there.

Three cautions come up when you defend an effort setting. First, defaults differ between models. Most models default to high, but Claude Opus 5.5 defaults to medium, so a request that omits effort runs one level lower than it did on Claude Opus 5. Second, don't carry settings over from an earlier model. Run a fresh effort sweep on your own evals. Third, effort is not a length control. On Claude Opus 5 it changes how much the model thinks, not how long the visible response is, so ask for brevity in the prompt instead. At the top of the range, the Claude Opus 4.7 guidance warns that max often costs far more for small gains.

Sources456

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.If a workload is too slow, the first fix is to move to a smaller model.Why is that wrong?

   Anthropic says adjusting effort on your current model is often a better lever than changing models. Try an effort sweep before you drop a tier.

   Covered in Model choice: speed-first or capability-first

2. 2.A request that leaves out effort runs at high on every model.Why is that wrong?

   Claude Opus 5.5 defaults to medium, so leaving effort out runs one level lower than it did on Claude Opus 5.

   Covered in Effort: the fine-grained dial inside one model

3. 3.Lowering effort is a reliable way to get shorter visible answers.Why is that wrong?

   On Claude Opus 5, effort changes how much the model thinks, not how long the visible response is. Ask for brevity in the prompt.

   Covered in Effort: the fine-grained dial inside one model