CertSafari
    Anthropic Claude Certified Developer – Professional (CCDV-P)· Lessons

    Domain 3 · Lesson 11/25

    Trace analysis at scale: sampling, aggregation and failure cohorts

    Trace Analysis at Scale

    17 min read
    4% of exam
    7 sources
    Published 19 Nov 2026
    Docs as of 18 Nov 2026

    What you will be able to do

    • Design a trace retention policy that keeps rare failures while per-run metrics carry the headline counts
    • Explain why rates computed over an error-biased trace store are wrong, and where trustworthy rates come from
    • Aggregate span data without double counting tokens or duplicate deliveries
    • Localize a regression by slicing runs into failure cohorts and ranking slices by excess failures
    • Spot loops and context growth by grouping a session's spans across traces

    Key concept

    Failure cohort — A failure cohort is the set of runs that share an attribute value, such as a model, prompt version, tenant or tool, and whose failure rate you compare with a baseline. Systemic issues appear as a cohort whose failure rate has moved, even when no single trace looks unusual.

    1.Sampling decides which runs you can investigate

    One trace shows how one request went. It can't tell you that tool timeouts doubled for one tenant, or that a new prompt version fails three times as often as the old one. Problems like that only show up across thousands of runs, and at that volume nobody stores every trace. Trace analysis at scale therefore starts with which runs you keep, and how keeping them changes every number you compute afterwards. Instrumenting the spans in the first place belongs to telemetry and instrumentation.

    The first decision is made before the request leaves your process. The SDK writes a W3C traceparent header, and the sampled flag in that header comes from your application's sampler. When the flag is off, the Claude API emits nothing for that request: no SERVER span, no model-request spans, no tool spans. An MCP server connected through the MCP connector also gets no traceparent. A head sampler therefore has to decide before the outcome is known. It can thin traffic uniformly, but it can't choose to keep the runs that will fail.

    Keeping rare failures cheaply takes two separate mechanisms. The first is metrics for every run. The Agent SDK's metrics signal is counters for tokens, cost, sessions and tool decisions, and it is configured separately from traces, so outcome counts, token histograms and per-step latency cover all traffic no matter how many traces you keep. Most dashboards need only those counts and percentiles. The second is a retention policy, applied to spans after they have been emitted, that keeps traces on purpose: every run with an ERROR span, every run slower than a latency threshold, and a small uniform baseline of everything else. The baseline matters. Without healthy runs to compare against, a stored failure has nothing to contrast with.

    A security-review workflow for agent sessions follows the same pattern. Cheap rules scan every session without any model calls and flag the suspicious ones. The expensive reader then sees the flagged sessions plus a random sample. Its own caveat applies to any sampled store: a random sample saves cost but loses coverage. Reading a sample of the unflagged sessions doesn't prove that the rest are clean.

    Cost-cutting changes judged by whether rare failures survive
    ChangeRare failures still findable?Why
    Per-run metrics plus retention of errors, slow runs and a uniform baselineYesCounters cover every run. Failures are kept on purpose, and the baseline gives a comparison set
    Keep all traces for only 24 hoursOnly if noticed the same dayA failure that is noticed later, or that builds up over a week, is already deleted
    Truncate every tool result to 200 charactersWeakenedTool arguments and results are what you read to diagnose a failure, and this cuts them on failed runs too
    Keep traces only for the largest tenantsNoA failure that only affects a small tenant disappears completely

    Checkpoint 1 of 9· Check yourself

    You need to cut tracing cost on a high-volume agent service. You still need to find rare failures later and to report accurate outcome counts and latency percentiles. Which design meets both goals?

    Checkpoint 2 of 9· Exam question

    Marlow Payments runs an agent that triages disputed card transactions in production, and an observability team stores a full trace for every run, which has become a major storage expense. A review of how the traces are used shows that the operations dashboards only ever read outcome counts and latency percentiles. Investigators open a trace in only a small fraction of runs, almost always after a rare failure or an unusually slow run, and they need span-level detail such as which tool output preceded a downstream error. Analysts also compare week-over-week cohorts and need a representative sample of healthy runs. Smaller merchants generate fewer runs but have contractual rights to investigation. The team wants to cut cost without losing the ability to find rare failures. Select two changes.(Select 2)

    0 of 2 selected

    Sources123

    2.Why rates computed from stored traces mislead

    A trace store filled by an error-biased policy is not a sample of your traffic, so a ratio computed over it describes the policy, not the product. Suppose every error is kept and 1% of successes are. If 0.8% of runs actually fail, roughly half of the stored traces are failures. A dashboard that divides stored failures by stored traces shows something like 31% and is reporting the keep rates, not the failure rate.

    Delivery adds further distortion. API spans arrive at least once, so duplicates do occur. They are also best effort: a span lost before delivery is never sent again. On top of that, an unsampled trace or a disabled destination produces no spans at all. So stored spans will not add up to the true total of anything, whether failures, runs or tokens. Headline numbers have to come from somewhere else.

    Which data source answers which question
    QuestionSourceReason
    What fraction of runs failed this week?Per-run counters (metrics signal)Recorded on every execution, so retention policy doesn't affect them
    What did we spend?Usage and Cost APISpans can be duplicated or dropped, so don't bill from them
    Why did these runs fail?Retained tracesTool arguments, results and stop reasons for individual runs
    Rate estimated from stored tracesRetained traces, each weighted by 1 / its keep probabilityUndoes the over-representation that the retention policy introduced

    If a rate has to come from the traces themselves, for example the failure rate inside a slice that the counters don't break out, weight each stored trace by the inverse of its retention probability. A success kept at 1% counts as 100 runs, and an error kept at 100% counts as one. Averaging over a longer window, lowering the error keep rate, or limiting the store to recent data only changes how distorted the result is. None of those corrects it.

    Checkpoint 3 of 9· Match them up

    Your trace store keeps every ERROR run and 1% of successful runs. Match each question about last week's traffic to the data source that answers it correctly.

    Tap a term, then the definition that fits it.

    Checkpoint 4 of 9· Exam question

    Fennel Mobility runs a ride-dispatch platform and, to control storage cost, adopted error-biased retention: nearly every failed task trace is kept, while only a small share of successful traces is stored. Engineers rely on the retained failures for debugging and want to keep them. After the change, a dashboard computed over stored traces reports a 31% task-failure rate, while the always-on run counter, which increments on every execution, shows about 0.8%. Switching the dashboard from a 7-day to a 30-day view left the figure essentially unchanged, and the sampling decision for each trace is logged alongside it. Leadership will cite these rates in a quarterly reliability review and needs them to match production reality. Which fix makes trace-derived rates trustworthy?

    Sources14

    3.Aggregating spans without double counting

    Even a correct cohort comparison goes wrong if each run is counted twice. With both SDK and API tracing enabled, one Messages API request produces two spans named anthropic.messages.create: a CLIENT span from your service and a SERVER span from claude-platform beneath it. Both carry the same anthropic.request.id. Summing tokens across both doubles the request's tokens, and summing the SERVER span together with its anthropic.model_request children counts them a third time.

    Use these rules for any span-based aggregate:

    • Remove duplicate deliveries by keying each span on trace ID plus span ID.
    • Sum one source per request: the SERVER spans or their anthropic.model_request children, never both and never alongside the CLIENT span. Add anthropic.compaction spans if any are present.
    • Don't add cache attributes to gen_ai.usage.input_tokens, because it already includes cache reads and writes.
    • For cost by cohort, prefer the API spans. They hold the billed count even when a request fails or is cancelled, while the SDK span carries what the response reported, which can be lower or missing.

    The last rule is the one that matters most for failure analysis. A cohort with many failed or cancelled requests looks cheaper on SDK spans than it really is.

    Checkpoint 5 of 9· Check yourself

    A cost-by-cohort dashboard shows that one tenant's runs look unusually cheap. That tenant also has many failed and cancelled Messages API requests. Both SDK and API tracing are on. Which way of aggregating gives that cohort's real token spend?

    Sources56

    4.Localizing a regression by slicing into cohorts

    When the overall failure rate moves, the fastest way to localize the cause is to slice every run by each attribute you already record, compare each slice's failure rate with the previous period, and rank the slices by excess failures: the change in rate multiplied by the slice's volume. A slice whose rate went from 2% to 30% on a fifth of traffic stands out. A slice that moved from 2% to 3% on a tiny tenant does not. These are aggregate questions, so the counts should be per-run counters or inverse-weighted traces, not raw stored traces.

    Span and resource attributes worth slicing on
    AttributeWhat a shifted slice suggests
    gen_ai.request.model vs gen_ai.response.modelA fallback model served the request, or an alias resolved to a different model
    service.version (resource attribute)A deploy changed prompts, tools or code
    tenant.id, enduser.id, anthropic.workspace.idThe problem is limited to particular customers or workspaces
    error.typerate_limit_error or overloaded_error points at capacity, while connection_error or client_timeout_error points at transport
    anthropic.message.stop_reasonA rise in max_tokens means answers are being truncated
    Tool span name, e.g. anthropic.tool_use bashOne tool's error rate is driving the change

    A dimension is only sliceable if it was recorded on the run. For Agent SDK workloads, set business dimensions as OpenTelemetry resource attributes when the request is made, so that every span and metric from that call carries them. The example below attaches the end user and tenant. A service.version set the same way lets you put runs from before and after a deploy side by side.

    Tagging Agent SDK telemetry with end-user and tenant IDs so runs can be sliced by thempython
    from urllib.parse import quote
    
    options = ClaudeAgentOptions(
        env={
            # ... exporter configuration from the Enable telemetry export example ...
            # request is the incoming request object from your web framework.
            "OTEL_RESOURCE_ATTRIBUTES": f"enduser.id={quote(request.user_id)},tenant.id={quote(request.tenant_id)}",
        },
    )

    Other first moves are slower or answer a different question. Hand-labelling a random sample of failed traces finds the most common kind of failure, which is often unrelated to what changed this week, because it has no comparison with last week. Replaying failed inputs at temperature 0 retests individual cases and says nothing about which part of the traffic moved. Adding new fields and waiting a week for data delays the diagnosis when the attributes you need are already on every trace. Slicing comes first. Reading traces from the top-ranked slice comes after it.

    Checkpoint 6 of 9· Check yourself

    After a release week, ranking slices by excess failures puts one cohort on top: runs where gen_ai.response.model differs from gen_ai.request.model. That cohort now fails at 28%, up from 3% the week before. What does this shifted slice point to first?

    Checkpoint 7 of 9· Fill the gap

    A regression may be limited to particular customers. To make that cohort sliceable, every span and metric from an Agent SDK call must carry the end user and tenant. Which environment variable completes this sample?

    from urllib.parse import quote
    
    options = ClaudeAgentOptions(
        env={
            # ... exporter configuration from the Enable telemetry export example ...
            # request is the incoming request object from your web framework.
            " ? ": f"enduser.id={quote(request.user_id)},tenant.id={quote(request.tenant_id)}",
        },
    )

    Checkpoint 8 of 9· Exam question

    Heron Pharma runs an agent that processes prior-authorization tasks for hospital tenants across several regions. Over the past week its task failure rate climbed from 2% to 5% and is still elevated today. The 2% baseline was never zero, and the team already knows several long-standing failure causes, such as malformed payer documents, that dominate any sample of failed traces. Every trace carries model version, prompt version, tool set, tenant and region, and the platform team changed several of these during the week. A trial replay of a few failed inputs showed that many of them also fail on last week's deployment. The on-call engineer must localize the regression today, using the data already collected rather than waiting on new instrumentation. What is the best first analysis to localize the regression?

    Sources7624

    5.Finding loops and context growth across a session

    Some systemic failures only appear when the turns of one session are read together. In a Managed Agents session, spans can belong to more than one trace. A turn started by another thread's reply has no parent request and begins a new trace. To rebuild the session, group its spans by anthropic.session.id, or group a thread by gen_ai.conversation.id and sort by start time to read it as a transcript.

    With the session rebuilt, two patterns are easy to count across the whole fleet:

    • Loops: runs that end with error_max_turns, or the same tool call retried after a refusal. Count them per session and per agent rather than reading them one at a time.
    • Context growth: gen_ai.usage.input_tokens on each anthropic.model_request rising turn after turn, followed by anthropic.compaction spans. A cohort whose input tokens per turn keep climbing pays for that context on every call.

    Once the rules flag sessions like these, give the flagged sessions plus a random sample to whoever investigates. When those patterns become alert conditions, that work belongs to production monitoring and alerting.

    Checkpoint 9 of 9· Check yourself

    In a multi-agent Managed Agents fleet, you want to count, per session, the runs that end in error_max_turns and the turns whose input tokens keep climbing. Some turns start because another thread replied. How should you assemble each session before counting?

    Sources643

    Sources

    (7)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “If the sampled flag in the header is off, the API doesn’t emit the request’s span or the spans under it.”
      ↩︎ Sampling decides which runs you can investigate
      “such as when its traceparent header has the sampled flag off, your MCP server gets no traceparent.”
      ↩︎ Sampling decides which runs you can investigate
      “At least once: A destination can receive the same span more than once. To remove duplicates, use the trace ID and span ID.”
      ↩︎ Why rates computed from stored traces mislead
      “Best effort: A span can be lost before delivery. The API can’t send a lost span again.”
      ↩︎ Why rates computed from stored traces mislead
      “The SDK writes the traceparent header with your application’s propagator, and the sampled flag in it follows your sampler.”
      ↩︎ Exam trap 1
      “Best effort: A span can be lost before delivery. The API can’t send a lost span again.”
      ↩︎ Exam trap 2
    2. 2.
      “Counters for tokens, cost, sessions, lines of code, and tool decisions”
      ↩︎ Sampling decides which runs you can investigate
      “service.version=1.4.0,deployment.environment=production”
      ↩︎ Localizing a regression by slicing into cohorts
    3. 3.
      “Sampling trades cost for coverage. It doesn't show the unread sessions are clean.”
      ↩︎ Sampling decides which runs you can investigate
      “The code scans every session, with no model calls, in two ways”
      ↩︎ Sampling decides which runs you can investigate
      “sdk_brief_04: error (error_max_turns) at event 32”
      ↩︎ Finding loops and context growth across a session
    4. 4.
      “Don't bill from spans. Delivery is at least once, so duplicates happen, and a disabled destination or an unsampled trace drops spans.”
      ↩︎ Why rates computed from stored traces mislead
      “response.model differs from request.model when you passed an alias or a fallback model served the request.”
      ↩︎ Localizing a regression by slicing into cohorts
      “ERROR means it raised, and error.type says why: an API error type such as rate_limit_error or overloaded_error, an HTTP status such as 502”
      ↩︎ Localizing a regression by slicing into cohorts
      “Group by it and sort by start time to read one thread as a transcript.”
      ↩︎ Finding loops and context growth across a session
      “Treat UNSET on stream and event-stream spans as normal.”
      ↩︎ Finding loops and context growth across a session
      “If your sampler drops a trace, the sampled bit in traceparent is off and the API emits nothing for that request.”
      ↩︎ Checkpoint
      “response.model differs from request.model when you passed an alias or a fallback model served the request”
      ↩︎ Checkpoint
      “a turn that starts because another thread replied has no parent request, so it begins a new trace that you find by anthropic.session.id.”
      ↩︎ Checkpoint
    5. 5.
      “To count each token once, sum the spans of only one source for each request.”
      ↩︎ Aggregating spans without double counting
      “the SDK span is CLIENT and its child, the API span, is SERVER.”
      ↩︎ Aggregating spans without double counting
      “To count each token once, sum the spans of only one source for each request.”
      ↩︎ Exam trap 3
    6. 6.
      “The input tokens, cache reads and writes included.”
      ↩︎ Aggregating spans without double counting
      “the token counts are the tokens that your organization is billed for, even when the request fails or the application cancels it.”
      ↩︎ Aggregating spans without double counting
      “for a failed or canceled request they can be lower than on the API span, or absent.”
      ↩︎ Aggregating spans without double counting
      “The ID of the workspace that the work runs in.”
      ↩︎ Localizing a regression by slicing into cohorts
      “to assemble a view of the session, such as a transcript, group its spans by these IDs.”
      ↩︎ Finding loops and context growth across a session
      “the SDK span for a stream that never completes, such as anthropic.sessions.events.stream, has the status unset or ERROR.”
      ↩︎ Finding loops and context growth across a session
    7. 7.
      “Break down by token type, user, team, model, skill.name, plugin.name, or agent.name”
      ↩︎ Localizing a regression by slicing into cohorts
      “Break down by token type, user, team, model, skill.name, plugin.name, or agent.name”
      ↩︎ Key concept
      “Break down by token type, user, team, model, skill.name, plugin.name, or agent.name”
      ↩︎ Exam trap 4

    Ready to test yourself?

    Practise Anthropic Claude Certified Developer – Professional (CCDV-P) in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.