CertSafari
    Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards· Lessons

    Domain 3 · Lesson 15/39

    Relieving the binding constraint: accuracy, latency or throughput

    Relieving the Binding Performance Constraint

    16 min read
    2.63% of exam
    7 sources
    Published 18 Nov 2026
    Docs as of 18 Nov 2026

    What you will be able to do

    • Identify whether accuracy, latency or request throughput is the constraint a system is actually failing on
    • Choose among model tier, effort, token trimming, streaming, caching, batch processing and retrieval depth for the change that relieves that constraint at least cost to the others
    • Rule out changes that help a different constraint or make the binding one worse
    • Confirm a chosen change against a repeated baseline and a held-out split before adopting it

    Key concept

    Binding constraint — The binding constraint is the one requirement (accuracy, latency or throughput) that the system is currently failing to meet. The right change relieves that constraint while spending as little as possible of the requirements that still have headroom. A change that improves a constraint the system already meets only adds cost.

    1.Name the binding constraint before changing anything

    A production Claude system usually meets most of its targets and fails one. A triage classifier can score well above its accuracy target and still miss its latency budget. A policy assistant can answer instantly and still give wrong answers. The target it fails is the binding constraint, and it sets the direction for every configuration change. The targets it meets are slack. Slack is a budget you can spend, but only as much as the fix needs.

    The available changes fall into two kinds. Free wins lower cost without lowering quality: prompt caching, trimming tokens, and batch processing for work that can wait. Tradeoffs move along the frontier: a different model tier, a different effort level, output caps. Some free wins still carry a price on another axis. Batch processing, for example, trades latency for its discount, so it is free only when latency is slack.

    How common levers move cost, latency and quality, from Anthropic’s internal measurements
    LeverCost effectLatency effectQuality effect
    Prompt cachingCut by a factor of 4.8 to 7.2 on a research agentFasterNone
    Batch API50%Results within 24 hoursNone
    Lower effortMedium about 30% to 52% less than high on long coding tasksFasterUp to 34 points lower, depending on the task and the model
    Ask for shorter answersAbout 13% lessFasterNone
    Fast modeTwice the standard token pricesUp to 2.5x higher output tokens per secondNone

    Read the table by constraint, not by lever. If latency is binding, the useful rows are the ones marked faster, and the quality column shows what each will cost you. If accuracy is binding, a row that only saves money or time doesn't help. Batch processing shows how one lever can be right or wrong depending on which constraint binds. It has no quality cost, so it is ideal when latency is slack. When latency is binding, its 24-hour result window makes it unusable. The wider practice of trimming spend across a whole workload belongs to cost and latency optimization. Here the question is narrower: which single change clears the failing target.

    Checkpoint 1 of 8· Check yourself

    A synchronous triage classifier scores well above its accuracy target, but its responses miss the latency budget. Cost is acceptable. Which change relieves the binding constraint at the least cost to the other requirements?

    Sources1

    2.When latency is binding: tier, effort, tokens and calls

    Total task time breaks down into parts you can act on. The latency cookbook models the time to last token like this: TTLT ≈ calls × TTFT + output tokens ÷ OTPS + additional client- or server-side latency. Every model call pays its own time to first token (TTFT). Every output token, thinking included, has to be generated at the model’s output speed (OTPS). Any change that cuts latency shrinks one of these terms: fewer calls, a shorter uncached prompt, fewer output tokens, or a faster model.

    The model tier is the most direct lever. Claude Haiku 5.5 is the tier for the lowest latency and price, and its main speed control is effort. Effort trades intelligence for latency and cost within one model, so you can lower it without changing models. When a mid-tier model clears its accuracy target by a margin and misses its latency budget, that margin is the slack. Spend it by moving to a smaller, faster tier or lowering effort, then re-run the labelled eval set to confirm accuracy still clears the target. How each tier fits different workloads is covered in model tier selection.

    A latency-sensitive call on Claude Haiku 5.5 at low effort, the cheapest and fastest effort levelpython
    client = anthropic.Anthropic()
    
    # For time-sensitive applications, use Claude Haiku 5.5 at low effort
    message = client.messages.create(
        model="claude-haiku-5-5",
        max_tokens=1024,
        output_config={"effort": "low"},
        messages=[
            {
                "role": "user",
                "content": "Summarize this customer feedback in 2 sentences: [feedback text]",
            }
        ],
    )
    print(next(block.text for block in message.content if block.type == "text"))

    Token count is the next lever, and it works in both directions. The fewer tokens the model reads and writes, the faster it responds. Asking for concise answers and setting max_tokens both cut output. This is also why some tempting fixes make a latency problem worse. Adding twenty more few-shot examples lengthens every prompt, which buys accuracy the system already has and adds latency it can't afford. Moving to the largest tier with extended thinking adds thinking tokens to the output term, which slows the response further.

    Streaming relieves a latency complaint only when the complaint is about the wait before anything appears. In the cookbook’s insurer example, the phone line optimises for TTFT because a caller sits in silence until the agent speaks. The chat window can show progress, so it optimises for TTLT. A classifier whose p95 is measured on the completed label gets nothing from streaming. The Message Batches API is worse for that classifier: batch results can take up to 24 hours, which no synchronous console can wait for.

    Checkpoint 2 of 8· Fill the gap

    This call is for a latency-sensitive workload whose accuracy has headroom. Which effort value makes it the cheapest and fastest setting on Claude Haiku 5.5?

    client = anthropic.Anthropic()
    
    # For time-sensitive applications, use Claude Haiku 5.5 at low effort
    message = client.messages.create(
        model="claude-haiku-5-5",
        max_tokens=1024,
        output_config={"effort": " ? "},
        messages=[
            {
                "role": "user",
                "content": "Summarize this customer feedback in 2 sentences: [feedback text]",
            }
        ],
    )
    print(next(block.text for block in message.content if block.type == "text"))

    Checkpoint 3 of 8· Exam question

    You are the architect for a support-triage classifier that calls the Messages API synchronously from an agent console. The current mid-tier model scores 97% on the labelled eval set against a 95% target, but p95 latency is 4.2 seconds against a 1.5 second budget. Which change relieves the binding constraint at the least cost to the other?

    Sources2341

    3.When volume is binding and latency is slack: move work off the synchronous path

    A different failure shows up in bulk jobs. A nightly run sends hundreds of thousands of synchronous requests, keeps hitting rate limits, and has all night to finish. The binding constraint is request throughput. Per-request latency is slack. The fix that spends exactly that slack is the Message Batches API: it accepts many Messages requests together, processes them asynchronously, and lets you poll for results.

    Batch processing is designed for the situation where immediate responses are not required. In return for giving up immediacy, it raises throughput and halves cost. Most batches finish in under an hour, and results are guaranteed within 24 hours, so a job due by morning loses nothing it needed. If the job were feeding a live agent console, the same change would break the latency target. Whether batch is the right answer depends entirely on which constraint binds.

    Candidate fixes for a rate-limited overnight summarisation job
    ChangeEffect on the binding constraint (throughput)Why
    Message Batches APIRelieves itAsynchronous processing of large volumes, with higher throughput
    StreamingNoneChanges when output arrives, not how many requests or tokens are sent
    cache_control on each ticket bodyNoneA cache hit needs a repeated prefix, and each ticket is read once
    Larger model tierNone, and cost risesThe same number of requests, each one more expensive

    The caching row deserves a closer look, because caching is the strongest free win in most agent loops. Caching works by matching prefixes: the system checks whether the prompt prefix up to a breakpoint is already cached from a recent query. That fits repetitive tasks with consistent instructions, such as a shared system prompt sent with every ticket. It does nothing for the ticket bodies, which differ every time. Breakpoint placement is covered in cache breakpoints.

    Checkpoint 4 of 8· Check yourself

    Every night, hundreds of thousands of support tickets are summarised through synchronous Messages calls. The job keeps hitting rate limits, and nobody reads the summaries until the next morning. Which change fixes the requirement the job is failing?

    Checkpoint 5 of 8· Exam question

    A team runs a nightly job that summarises 200,000 support tickets through synchronous Messages API calls and keeps hitting rate limits. Results are only needed by the next morning, and per-ticket latency does not matter. Which change relieves the binding constraint at the least cost to the other?

    Sources56

    4.When accuracy is binding: fix what the model never sees

    Accuracy failures need a diagnosis before a lever. If failure analysis shows Claude reasoned badly over the right evidence, a stronger tier or higher effort is the tradeoff to try. Retrieval-augmented systems often fail in a different way: the correct passage exists in the index, but it ranks below the cut-off and never reaches the prompt. No model, however capable, can reason over a passage it was never sent. A system-prompt rule to answer only from the supplied passages doesn't help either, because the passage it needs isn't among them.

    When the right passage is missing from the prompt, the change belongs in retrieval, and reranking is the standard tool. It splits retrieval into two stages. The first stage casts a wide net, retrieving many more candidates than the prompt will hold, which raises recall. In the contextual-retrieval cookbook, hybrid search found the right chunk 88.86% of the time in the top 5 and 95.23% in the top 20. The second stage uses a specialised reranking model to score those candidates, and only the few best go to Claude.

    The cheaper path is to rerank, not to put all 50 candidates in the prompt. Pasting 50 chunks in would fix recall, but every query would then carry ten times the input tokens. More tokens mean slower responses, and the model has to sort relevant policy text from noise on its own. Retrieving 50 and reranking down to a handful spends a little of the latency headroom on the rerank step and keeps the prompt small. For the choice between RAG, long context and agentic search, see retrieval architecture.

    Checkpoint 6 of 8· Check yourself

    Failure analysis on a policy assistant built on RAG finds that the passage holding the right answer is in the index but ranks around 30th. Only the top 5 chunks go into the prompt. Answers must get more accurate, and the latency budget has only a little headroom. Which change fixes the failure at the least cost to latency?

    Checkpoint 7 of 8· Exam question

    A RAG assistant on the Messages API answers internal policy questions. Failure analysis shows the right passage is usually in the index but ranks outside the top 5 chunks that are sent to Claude; recall at 50 candidates is high. The latency budget has headroom. Choose two changes that relieve the accuracy constraint at the least added cost.(Select 2)

    0 of 2 selected

    Sources73

    5.Confirm the change against a noise bar

    Every change that spends slack needs to be measured, because the slack is real only if the eval says so. A good evaluation set built from your own prompts and data is the most important step in a model decision. Two habits make the measurement trustworthy. First, repeat the baseline before the first change: in Anthropic’s runs, identical runs of one configuration differed by several tasks, and those repeats give you a noise bar. Second, hold out a split that the tuning process never reads, so a change that only fits the tasks you looked at gets caught.

    These habits decide whether a speed-up is safe to ship. After moving a classifier down a tier, the question is whether accuracy stayed above the target by more than run-to-run noise, not whether it went down at all. In one search on tau2-bench retail, lowering effort and moving down a tier produced the savings, while the prompt edits produced no durable gain once checked on held-out data. How to design such comparisons is covered in experimental design.

    Checkpoint 8 of 8· Match them up

    Before shipping a cheaper configuration, such as a classifier moved down a tier, a team checks its eval results. Match each element of that check to the role it plays in the decision.

    Tap a term, then the definition that fits it.

    Sources41

    Sources

    (7)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Free wins cut spend without touching quality”
      ↩︎ Name the binding constraint before changing anything
      “Tradeoffs exchange cost for intelligence: model choice, effort, output caps and session budgets, an elapsed-time clock, and multimodel architectures.”
      ↩︎ Name the binding constraint before changing anything
      “batch processing trades latency for its discount”
      ↩︎ Name the binding constraint before changing anything
      “batch processing at 50% off for work that can wait up to 24 hours”
      ↩︎ When latency is binding: tier, effort, tokens and calls
      “Repeat the baseline before the first change.”
      ↩︎ Confirm the change against a noise bar
      “Hold out a split the search never reads.”
      ↩︎ Confirm the change against a noise bar
      “The effort and model changes produced the savings; the prompt edits brought no durable gain.”
      ↩︎ Confirm the change against a noise bar
      “Cost and intelligence are usually pictured as a frontier where one buys the other.”
      ↩︎ Key concept
      “batch processing trades latency for its discount”
      ↩︎ Exam trap 2
      “batch processing at 50% off for work that can wait up to 24 hours”
      ↩︎ Checkpoint
      “Identical runs of one starting configuration differed by several tasks in these measurements (reference 24). The repeats give the search’s adoption gate a noise bar.”
      ↩︎ Checkpoint
    2. 2.
      “TTLT ≈ calls × TTFT + output tokens ÷ OTPS + additional client- or server-side latency”
      ↩︎ When latency is binding: tier, effort, tokens and calls
      “Output tokens: these include thinking tokens.”
      ↩︎ When latency is binding: tier, effort, tokens and calls
    3. 3.
      “One of the most direct ways to reduce latency is to choose the appropriate model for your use case.”
      ↩︎ When latency is binding: tier, effort, tokens and calls
      “The fewer tokens the model has to process and generate, the faster the response will be.”
      ↩︎ When latency is binding: tier, effort, tokens and calls
      “The fewer tokens the model has to process and generate, the faster the response will be.”
      ↩︎ When accuracy is binding: fix what the model never sees
      “This can significantly improve the perceived responsiveness of your application, as users can see the model’s output in real time.”
      ↩︎ Exam trap 1
      “This can significantly improve the perceived responsiveness of your application, as users can see the model’s output in real time.”
      ↩︎ Prediction
    4. 4.
      “an effort parameter that trades intelligence for latency and cost within a single model”
      ↩︎ When latency is binding: tier, effort, tokens and calls
      “Having a good evaluation set is the most important step in the process.”
      ↩︎ Confirm the change against a noise bar
      “Several Claude models support an effort parameter that trades intelligence for latency and cost within a single model.”
      ↩︎ Checkpoint
    5. 5.
      “Process large volumes of Messages requests asynchronously with the Message Batches API, cutting costs by 50 percent and increasing throughput.”
      ↩︎ When volume is binding and latency is slack: move work off the synchronous path
    6. 6.
      “The system checks if a prompt prefix, up to a specified cache breakpoint, is already cached from a recent query.”
      ↩︎ Exam trap 3
    7. 7.
      “Reranking is a two-stage retrieval approach:”
      ↩︎ When accuracy is binding: fix what the model never sees
      “Stage 2 - Precise Selection: Use a specialized reranking model to score”
      ↩︎ When accuracy is binding: fix what the model never sees
      “Stage 1 - Broad Retrieval: Cast a wide net by retrieving more candidates than you need (e.g., retrieve 100 chunks)”
      ↩︎ Exam trap 4

    Ready to test yourself?

    Practise Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.