CertSafari
    Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards· Lessons

    Domain 4 · Lesson 26/39

    Cost and latency optimization from usage telemetry

    Cost and Latency Optimization

    18 min read
    2.43% of exam
    4 sources
    Published 18 Nov 2026
    Docs as of 18 Nov 2026

    What you will be able to do

    • Read the cache and token fields in Messages API usage to find which step, and which kind of token, the spend is concentrated in
    • Diagnose a cache that writes on every request but never reads, and fix it without changing answers
    • Choose the Message Batches API, a tier change, an effort change, or an output change to match the work and its quality floor
    • Gate every cost-cutting change on a fixed eval set measured against a repeated baseline
    • Specify an optimization as a target step, a lever, an expected saving and a quality floor, so the saving is accepted only if the eval clears the floor
    • Choose latency levers (model, output length, streaming, cache pre-warming) by whether they shorten real time or perceived time

    Key concept

    Free wins before tradeoffs — Cost levers fall into two groups. Free wins such as caching, trimming tokens and batching lower spend while output quality stays the same. Tradeoffs such as a smaller tier, lower effort or tighter output caps exchange quality for cost, so you use them only when an eval shows the result still clears the stated quality floor.

    1.Finding where spend and latency concentrate

    Cost optimization starts with telemetry, because the right lever depends on where the tokens actually go. Every Messages API response carries a usage block (in the message_start event when streaming). That block splits input into three fields: tokens read from the cache, tokens written to it, and tokens after the last breakpoint that were neither. Output tokens are counted separately. Each of the four fields is billed at a different rate, so a bill dominated by full-price input calls for a different fix than one dominated by output.

    A usage block from a cached request. The cache_creation object splits writes by TTL, and the two values sum to cache_creation_input_tokens.json
    {
      "usage": {
        "input_tokens": 2048,
        "cache_read_input_tokens": 1800,
        "cache_creation_input_tokens": 248,
        "output_tokens": 503,
    
        "cache_creation": {
          "ephemeral_5m_input_tokens": 148,
          "ephemeral_1h_input_tokens": 100
        }
      }
    }
    Usage fields and what each one tells you about spend
    FieldMeaningWhat a high value suggests
    cache_read_input_tokensTokens retrieved from the cache for this requestHealthy: the prefix is billed at the cache-read rate
    cache_creation_input_tokensTokens written to the cache when creating a new entryExpected on the first turn. On every turn, it means the prefix is being rewritten instead of matched
    input_tokensInput tokens after the last cache breakpoint, neither read nor writtenContent that is billed at full input price on every request
    output_tokensGenerated tokens, billed at the higher output rateVerbose responses; a target for asking for shorter answers

    The cache-read share is the first ratio to compute. Without caching, every turn of an agentic task resends the system prompt, the tool definitions and every earlier turn. A 40-turn task therefore sends its first turn 40 times, and task cost grows roughly with the square of the turn count. Over a full day of real traffic, agent loops read a median of 84% of their input from the cache. If a loop reads less than about 80%, something is probably breaking the cache. When cache_read_input_tokens is small compared with input_tokens plus cache_creation_input_tokens, check that caching is switched on and that the prefix stays the same between requests.

    For a multi-step pipeline, add up those fields per step, not per request. Suppose a pipeline has three steps and one of them accounts for most of the spend. Next, check whether that step's cost is mostly input or mostly output. A step that sends a large prompt and returns a one-word label spends almost nothing on output, so output levers cannot move its line. Its cost is its input tokens times the price per token of its model tier. When you compare models, compare cost per completed task rather than cost per token. A cheaper model that needs extra attempts may cost more per task. For deeper trace work on why a step misbehaves, see root-cause isolation from logs and traces.

    Checkpoint 1 of 8· Match them up

    You are reading the usage blocks from an agent loop's telemetry. Match each usage field to what a consistently high value suggests about spend.

    Tap a term, then the definition that fits it.

    Sources12

    2.Diagnosing a cache that writes but never reads

    One telemetry pattern always points to the same cause. cache_creation_input_tokens equals the whole prefix on every request, and cache_read_input_tokens stays at zero. The API is writing an entry each time, so caching is on and the prefix is long enough to cache. But no later request ever matches that entry. Cache hits require 100 percent identical prompt segments up to and including the block marked with cache_control. A single changing value inside that span changes the prefix hash on every call.

    The fix is to put the changing content after the breakpoint. Timestamps, per-request context and the incoming message belong in the varying suffix. The breakpoint goes on the last block that stays identical across requests, which is the end of the static prefix. Moving a timestamp from the system block to the start of the user message leaves the model the same content, only in a different position. Prompt caching has no effect on output token generation, so a quality floor of "answers must not change" still holds. Placing and layering breakpoints is covered in cache breakpoints and reusable packaging.

    Three tempting fixes do nothing for this pattern. A longer TTL only keeps unmatched entries alive longer, and every miss bills the whole prefix at the write price, which is higher for the 1-hour duration. A second breakpoint on the changing user block does not help either, because the lookback does not search for stable content behind a breakpoint. It only finds entries that earlier requests wrote at their own breakpoints. Making the prompt longer to reach a minimum cacheable size fixes nothing when writes are already happening.

    Checkpoint 2 of 8· Check yourself

    A router now sets output_config.effort to low or high on each request, depending on how hard it judges the request to be. The prompt text and breakpoint placement did not change. Since then, the step's cache_creation_input_tokens often equals its whole prefix and cache_read_input_tokens has dropped sharply. What explains this?

    Checkpoint 3 of 8· Exam question

    You own a support assistant built on the Claude Messages API. The system prompt is a large, static policy document, and the first line of that same system block is "Current time: <ISO timestamp>". A cache_control breakpoint sits at the end of the system block. Usage telemetry shows cache_read_input_tokens is 0 on nearly every request, while cache_creation_input_tokens equals the full system prompt size each time. The quality floor is that answers must not change. Which change restores cache hits?

    Sources21

    3.Batching work nobody is waiting for

    The Message Batches API is the free win for work that has a deadline but no one waiting on each response. Nightly enrichment, bulk classification and large-scale evaluations all fit. You submit many Messages requests as one batch. Each request is processed independently and asynchronously, and you poll for results. Most batches finish in less than an hour, and the price is 50% lower. The guarantee is results within 24 hours, which makes batching a free win only for work that can wait that long.

    Batching also fixes a throughput problem that a synchronous job does not. A nightly job that fans out thousands of synchronous calls hits rate limits at peak concurrency and spends its time on retries. A batch moves that scheduling to the API and increases throughput. The requests and the model stay the same, so the outputs match what synchronous calls would produce.

    Levers for a large overnight job where no user waits on any single response
    LeverEffect on spendWhy it does or does not fit
    Message Batches API50% lower, with higher throughputImmediate responses are not required, and quality is unchanged
    Streaming plus more parallel workersStandard token priceStreaming improves how responsive the output feels to a watching user; no user is watching, and more concurrency adds to rate-limit pressure
    cache_control after each itemNo savingEach item is a different prefix, and the next night is far beyond a 5-minute or 1-hour lifetime
    Lower max_tokensLittle or noneA hard cap only truncates. Responses that already fit are billed the same, and cut-off responses put quality at risk

    Checkpoint 4 of 8· Check yourself

    Every night, a job sends 150,000 support tickets to Claude for classification. The labels feed a report that is read the following afternoon, and no one looks at individual responses. The job fans out synchronous calls, hits rate limits at peak concurrency, and spends much of its run on retries. Which change cuts spend without lowering label quality?

    Checkpoint 5 of 8· Exam question

    A retailer enriches 40,000 product descriptions every night using synchronous Claude Messages API calls. Results are needed by 7 a.m., no user waits on individual responses, and the job currently hits rate-limit retries during peak concurrency. Telemetry shows the job is the largest line item on the monthly bill, and the output quality must stay identical. Which optimization fits?

    Sources31

    4.Trading quality for cost behind an eval gate

    Once the free wins are in place, the remaining levers are tradeoffs: model choice, effort, output caps and session budgets, an elapsed-time clock, and multimodel architectures. Each one can lower quality, so a stated quality floor decides whether a change stays. For high-volume work whose outputs you can check, try Claude Haiku 5.5 first. If costs are too high and quality is fine, try lower effort on your current model. If quality then falls short, restore the effort. Otherwise, try the next tier up at its default effort. Tier choice is covered in model tier selection, and effort in reasoning effort and context lifecycle.

    Tradeoff levers against their quality cost and latency
    LeverSavingQuality costLatency
    Lower effortMedium about 30% to 52% less than high on long coding tasksUp to 34 points lower, depending on task and modelFaster
    Ask for shorter answersAbout 13% less, from 39% fewer output tokensNoneFaster
    Raise max_tokensNone: cost per solved task did not rise at 64,000A gain on one coding benchmarkNeutral
    Fast modeNone: twice the standard token pricesNoneUp to 2.5x higher output tokens per second

    The max_tokens rows matter more than they look. max_tokens sets a hard limit on response length, and raising it to 64,000 did not raise cost per solved task. A cap is a ceiling, not a target. Lowering it on a step that already returns short labels changes nothing on the bill, and a cap that truncates real answers costs quality. Asking for shorter answers is different: it changes what the model generates, which is where the 13% came from.

    Apply a tradeoff to the step where spend is concentrated, one change at a time, and score it on the same fixed eval set. Adopt the change only if the result clears the floor. Two settings make that verdict trustworthy. First, repeat the baseline before the first change: identical runs of one configuration differed by several tasks, and the repeats give you a noise bar. Second, hold out a split that the search never reads. In one measured search, a cheaper configuration stayed within the noise band but ended below the stricter floor the search used for its own verdicts. Staying within run-to-run noise is not the same as clearing the floor. Afterwards, rerun the eval and compare per-step token spend with the original baseline to confirm both the saving and the quality. Moving every step to a smaller tier at once and spot-checking by hand breaks both rules: it changes several things together and it has no fixed bar. The method itself is taught in experimental design for configuration comparison.

    A finished optimization names four things: the step where spend is concentrated, the one lever applied to it, the saving expected, and the quality floor the eval must clear. Telemetry picks the step and the lever. Take a classification step on the top tier that holds 62% of spend and returns one-word labels. Its cost sits in input tokens at the tier's price, so output caps are out. Batching and caching come first if the work allows them, since they cost no quality. If the step still dominates, the specification reads: move this step to Claude Haiku 5.5 at low effort, expect a large cut in per-token price, and keep the change only if the held-out score is at or above the floor. When outputs can be checked by tests or a verifier, a second option exists: run at low effort and re-run only the failures at high effort, which cost about 40% less than running everything at high at the same pass rate. If the held-out score misses the floor, the specification falls back to the next tier up at its default effort rather than shipping the miss.

    Checkpoint 6 of 8· Check yourself

    After caching and batching are in place, a team moves the step that dominates spend to a smaller model tier and scores it on the fixed held-out eval set. Repeated baseline runs scored between 85% and 89%. The cheaper configuration scores 86%, and the stated quality floor is 87%. What should the team decide?

    Checkpoint 7 of 8· Exam question

    A contract-review pipeline on the Messages API has three steps. Per-step usage telemetry across 10,000 runs shows spend split as: clause classification on the Opus-tier model 62%, retrieval summarisation 29%, final memo drafting 9%. Classification emits short labels. The quality floor is at least 95% classification accuracy on a fixed 400-case eval set, and the current model scores 97%. Choose two actions that cut spend without breaching the floor.(Select 2)

    0 of 2 selected

    Sources1

    5.Cutting real and perceived latency

    Latency telemetry has two numbers, and they call for different fixes. Baseline latency is how long the model takes to process the prompt and generate the whole response. Time to first token (TTFT) is how long the user waits before anything appears. Choosing a faster model shortens real latency: for speed-critical applications, Claude Haiku 5.5 is the fastest, and effort is its main control for speed and cost. Fewer tokens also mean a faster response, whether you trim the prompt, ask for brevity or set an output limit.

    Streaming shortens perceived latency. The response starts arriving before generation is complete, so users see output in real time. The token count and the price stay the same, which makes streaming the right fix for a chat interface that feels slow and the wrong fix for a cost problem. Caching shortens both cost and processing time, because a cached prefix is processed faster than a fresh one.

    Cache pre-warming targets the first interaction, which is the slow one. Before a user's first request, send a request with max_tokens: 0 and an explicit breakpoint on the shared system prompt or tool definitions. The API writes the cache and returns without generating output, so the user's first real request reads the cache instead of paying the cache-miss latency. Use the same thinking configuration and effort as real traffic. A warm-up with different settings writes an entry that real requests never hit.

    Checkpoint 8 of 8· Match them up

    A support chat app's latency telemetry shows slow full responses, a long wait before any text appears, and a slow first request in each session. Match each lever to the latency problem it actually addresses.

    Tap a term, then the definition that fits it.

    Sources42

    Sources

    (4)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “If cache_read_input_tokens is small next to input_tokens plus cache_creation_input_tokens, check that caching is engaged and that the prefix stays the same between requests.”
      ↩︎ Finding where spend and latency concentrate
      “A 40-turn task sends its first turn 40 times, so task cost grows with roughly the square of turn count.”
      ↩︎ Finding where spend and latency concentrate
      “Below about 80%, look for something breaking the cache”
      ↩︎ Finding where spend and latency concentrate
      “Compare on cost per completed task, not per token”
      ↩︎ Finding where spend and latency concentrate
      “A miss on either duration bills the whole prefix at the write price instead of the read price.”
      ↩︎ Diagnosing a cache that writes but never reads
      “batch processing at 50% off for work that can wait up to 24 hours”
      ↩︎ Batching work nobody is waiting for
      “Tradeoffs exchange cost for intelligence: model choice, effort, output caps and session budgets, an elapsed-time clock, and multimodel architectures.”
      ↩︎ Trading quality for cost behind an eval gate
      “cost per solved task did not rise at 64,000”
      ↩︎ Trading quality for cost behind an eval gate
      “Identical runs of one starting configuration differed by several tasks in these measurements”
      ↩︎ Trading quality for cost behind an eval gate
      “Hold out a split the search never reads.”
      ↩︎ Trading quality for cost behind an eval gate
      “about 40% lower cost than everything at high, at the same pass rate”
      ↩︎ Trading quality for cost behind an eval gate
      “Free wins cut spend without touching quality”
      ↩︎ Key concept
      “ended below the stricter floor the search used for its own verdicts”
      ↩︎ Exam trap 4
      “Its pass rate stayed inside the band set before the search by a fraction of a task, and ended below the stricter floor”
      ↩︎ Checkpoint
    2. 2.
      “cache_read_input_tokens: Number of tokens retrieved from the cache for this request”
      ↩︎ Finding where spend and latency concentrate
      “Cache hits require 100 percent identical prompt segments”
      ↩︎ Diagnosing a cache that writes but never reads
      “The lookback does not find stable content behind the breakpoint; it only finds entries that earlier requests wrote at their own breakpoints.”
      ↩︎ Diagnosing a cache that writes but never reads
      “Prompt caching has no effect on output token generation.”
      ↩︎ Diagnosing a cache that writes but never reads
      “Verify that tool_choice, image usage, the thinking configuration, and output_config.effort remain consistent between calls.”
      ↩︎ Diagnosing a cache that writes but never reads
      “This can reduce the cache-miss latency penalty on the first user interaction, lowering time-to-first-token (TTFT) for latency-sensitive applications.”
      ↩︎ Cutting real and perceived latency
      “Place the breakpoint on the last block that stays identical across requests.”
      ↩︎ Exam trap 1
      “input_tokens: Number of input tokens that were not read from or used to create a cache (that is, tokens after the last cache breakpoint)”
      ↩︎ Checkpoint
      “Cache writes occur only at the breakpoint, and if that block changes (timestamps, per-request context, the incoming message), the prefix hash never matches.”
      ↩︎ Prediction
    3. 3.
      “most batches finishing in less than 1 hour while reducing costs by 50 percent and increasing throughput”
      ↩︎ Batching work nobody is waiting for
      “Immediate responses are not required”
      ↩︎ Batching work nobody is waiting for
    4. 4.
      “For speed-critical applications, Claude Haiku 5.5 offers the fastest response times while maintaining high intelligence.”
      ↩︎ Cutting real and perceived latency
      “This can significantly improve the perceived responsiveness of your application”
      ↩︎ Cutting real and perceived latency
      “Use the max_tokens parameter to set a hard limit on the maximum length of the generated response.”
      ↩︎ Exam trap 2
      “This can significantly improve the perceived responsiveness of your application”
      ↩︎ Exam trap 3
      “This can significantly improve the perceived responsiveness of your application, as users can see the model’s output in real time.”
      ↩︎ Checkpoint

    Ready to test yourself?

    Practise Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.