What you will be able to do
- Design a trace retention policy that keeps rare failures while per-run metrics carry the headline counts
- Explain why rates computed over an error-biased trace store are wrong, and where trustworthy rates come from
- Aggregate span data without double counting tokens or duplicate deliveries
- Localize a regression by slicing runs into failure cohorts and ranking slices by excess failures
- Spot loops and context growth by grouping a session's spans across traces
Key concept
Failure cohort — A failure cohort is the set of runs that share an attribute value, such as a model, prompt version, tenant or tool, and whose failure rate you compare with a baseline. Systemic issues appear as a cohort whose failure rate has moved, even when no single trace looks unusual.
1.Sampling decides which runs you can investigate
One trace shows how one request went. It can't tell you that tool timeouts doubled for one tenant, or that a new prompt version fails three times as often as the old one. Problems like that only show up across thousands of runs, and at that volume nobody stores every trace. Trace analysis at scale therefore starts with which runs you keep, and how keeping them changes every number you compute afterwards. Instrumenting the spans in the first place belongs to telemetry and instrumentation.
The first decision is made before the request leaves your process. The SDK writes a W3C traceparent header, and the sampled flag in that header comes from your application's sampler. When the flag is off, the Claude API emits nothing for that request: no SERVER span, no model-request spans, no tool spans. An MCP server connected through the MCP connector also gets no traceparent. A head sampler therefore has to decide before the outcome is known. It can thin traffic uniformly, but it can't choose to keep the runs that will fail.
Keeping rare failures cheaply takes two separate mechanisms. The first is metrics for every run. The Agent SDK's metrics signal is counters for tokens, cost, sessions and tool decisions, and it is configured separately from traces, so outcome counts, token histograms and per-step latency cover all traffic no matter how many traces you keep. Most dashboards need only those counts and percentiles. The second is a retention policy, applied to spans after they have been emitted, that keeps traces on purpose: every run with an ERROR span, every run slower than a latency threshold, and a small uniform baseline of everything else. The baseline matters. Without healthy runs to compare against, a stored failure has nothing to contrast with.
A security-review workflow for agent sessions follows the same pattern. Cheap rules scan every session without any model calls and flag the suspicious ones. The expensive reader then sees the flagged sessions plus a random sample. Its own caveat applies to any sampled store: a random sample saves cost but loses coverage. Reading a sample of the unflagged sessions doesn't prove that the rest are clean.
| Change | Rare failures still findable? | Why |
|---|---|---|
| Per-run metrics plus retention of errors, slow runs and a uniform baseline | Yes | Counters cover every run. Failures are kept on purpose, and the baseline gives a comparison set |
| Keep all traces for only 24 hours | Only if noticed the same day | A failure that is noticed later, or that builds up over a week, is already deleted |
| Truncate every tool result to 200 characters | Weakened | Tool arguments and results are what you read to diagnose a failure, and this cuts them on failed runs too |
| Keep traces only for the largest tenants | No | A failure that only affects a small tenant disappears completely |
Checkpoint 1 of 9· Check yourself
You need to cut tracing cost on a high-volume agent service. You still need to find rare failures later and to report accurate outcome counts and latency percentiles. Which design meets both goals?
The head sampler decides before the outcome is known, and an unsampled request emits no API spans, so it cannot pick failures. Counters cover every run, and a retention policy applied after the spans are emitted keeps failures on purpose, with a baseline to compare them against.
“If your sampler drops a trace, the sampled bit in traceparent is off and the API emits nothing for that request.”Source: platform.claude.com
Checkpoint 2 of 9· Exam question
Marlow Payments runs an agent that triages disputed card transactions in production, and an observability team stores a full trace for every run, which has become a major storage expense. A review of how the traces are used shows that the operations dashboards only ever read outcome counts and latency percentiles. Investigators open a trace in only a small fraction of runs, almost always after a rare failure or an unusually slow run, and they need span-level detail such as which tool output preceded a downstream error. Analysts also compare week-over-week cohorts and need a representative sample of healthy runs. Smaller merchants generate fewer runs but have contractual rights to investigation. The team wants to cut cost without losing the ability to find rare failures. Select two changes.(Select 2)
Correct answers: A, B — Emit low-cardinality metrics for every run, such as outcome counts, token histograms and per-step latency, independent of trace retention.; Retain full traces by policy: all errors, runs beyond a latency threshold, and a small uniform baseline of everything else.
- A. Correct: metrics are cheap, cover 100% of runs and give exact rates, so full traces no longer have to serve the dashboards.
- B. Correct: this keeps the rare and slow cases that need span detail and a baseline for comparison, while dropping most healthy traces.
- C. Incorrect: a blanket short window cannot support week-over-week cohort comparison, and it still stores every healthy trace for that day.
- D. Incorrect: this cuts size but destroys evidence in the failing traces you most need, such as which output triggered a downstream error.
- E. Incorrect: a tenant filter leaves smaller tenants blind and biases every aggregate toward large accounts.
2.Why rates computed from stored traces mislead
A trace store filled by an error-biased policy is not a sample of your traffic, so a ratio computed over it describes the policy, not the product. Suppose every error is kept and 1% of successes are. If 0.8% of runs actually fail, roughly half of the stored traces are failures. A dashboard that divides stored failures by stored traces shows something like 31% and is reporting the keep rates, not the failure rate.
Delivery adds further distortion. API spans arrive at least once, so duplicates do occur. They are also best effort: a span lost before delivery is never sent again. On top of that, an unsampled trace or a disabled destination produces no spans at all. So stored spans will not add up to the true total of anything, whether failures, runs or tokens. Headline numbers have to come from somewhere else.
| Question | Source | Reason |
|---|---|---|
| What fraction of runs failed this week? | Per-run counters (metrics signal) | Recorded on every execution, so retention policy doesn't affect them |
| What did we spend? | Usage and Cost API | Spans can be duplicated or dropped, so don't bill from them |
| Why did these runs fail? | Retained traces | Tool arguments, results and stop reasons for individual runs |
| Rate estimated from stored traces | Retained traces, each weighted by 1 / its keep probability | Undoes the over-representation that the retention policy introduced |
If a rate has to come from the traces themselves, for example the failure rate inside a slice that the counters don't break out, weight each stored trace by the inverse of its retention probability. A success kept at 1% counts as 100 runs, and an error kept at 100% counts as one. Averaging over a longer window, lowering the error keep rate, or limiting the store to recent data only changes how distorted the result is. None of those corrects it.
Checkpoint 3 of 9· Match them up
Your trace store keeps every ERROR run and 1% of successful runs. Match each question about last week's traffic to the data source that answers it correctly.
Tap a term, then the definition that fits it.
Stored traces show the keep rates rather than the traffic, and spans can be duplicated or dropped. Headline rates therefore come from always-on counters and spend comes from the Usage and Cost API. Retained traces are for diagnosing failures, or for estimating a rate once they are weighted by inverse retention probability.
“Don't bill from spans. Delivery is at least once, so duplicates happen, and a disabled destination or an unsampled trace drops spans.”Source: platform.claude.com
Checkpoint 4 of 9· Exam question
Fennel Mobility runs a ride-dispatch platform and, to control storage cost, adopted error-biased retention: nearly every failed task trace is kept, while only a small share of successful traces is stored. Engineers rely on the retained failures for debugging and want to keep them. After the change, a dashboard computed over stored traces reports a 31% task-failure rate, while the always-on run counter, which increments on every execution, shows about 0.8%. Switching the dashboard from a 7-day to a 30-day view left the figure essentially unchanged, and the sampling decision for each trace is logged alongside it. Leadership will cite these rates in a quarterly reliability review and needs them to match production reality. Which fix makes trace-derived rates trustworthy?
Correct answer: C — Weight each stored trace by the inverse of its retention probability, or take headline rates from run counts recorded on each execution.
- A. Incorrect: a longer or smoothed window only averages the same over-sampled failures. Every stored trace is still drawn under the error-biased policy, so the rate stays near 31% rather than approaching the true value.
- B. Incorrect: a lower keep-rate or a per-service cap still retains failures at a higher rate than successes. The raw ratio from the stored traces remains inflated, and the failure evidence is reduced without correcting the bias.
- C. Correct: inverse-probability weights undo the unequal retention and restore the production mix. Counts recorded on each execution are unaffected by sampling, so they give the true rate directly.
- D. Incorrect: the error-biased policy is what over-represents failures, so traces stored since the change are the skewed ones. Restricting to them, or to a recent window under the same policy, keeps the inflated rate.
3.Aggregating spans without double counting
Even a correct cohort comparison goes wrong if each run is counted twice. With both SDK and API tracing enabled, one Messages API request produces two spans named anthropic.messages.create: a CLIENT span from your service and a SERVER span from claude-platform beneath it. Both carry the same anthropic.request.id. Summing tokens across both doubles the request's tokens, and summing the SERVER span together with its anthropic.model_request children counts them a third time.
Use these rules for any span-based aggregate:
- Remove duplicate deliveries by keying each span on trace ID plus span ID.
- Sum one source per request: the SERVER spans or their
anthropic.model_requestchildren, never both and never alongside the CLIENT span. Addanthropic.compactionspans if any are present. - Don't add cache attributes to
gen_ai.usage.input_tokens, because it already includes cache reads and writes. - For cost by cohort, prefer the API spans. They hold the billed count even when a request fails or is cancelled, while the SDK span carries what the response reported, which can be lower or missing.
The last rule is the one that matters most for failure analysis. A cohort with many failed or cancelled requests looks cheaper on SDK spans than it really is.
Checkpoint 5 of 9· Check yourself
A cost-by-cohort dashboard shows that one tenant's runs look unusually cheap. That tenant also has many failed and cancelled Messages API requests. Both SDK and API tracing are on. Which way of aggregating gives that cohort's real token spend?
API spans hold the billed count even when a request fails or is cancelled. SDK spans hold what the response reported, which can be lower or missing. Counting one source per request, after removing duplicate deliveries, stops the same tokens from being counted two or three times, and input_tokens already includes cache reads and writes.
“the token counts are the tokens that your organization is billed for, even when the request fails or the application cancels it.”Source: platform.claude.com
4.Localizing a regression by slicing into cohorts
When the overall failure rate moves, the fastest way to localize the cause is to slice every run by each attribute you already record, compare each slice's failure rate with the previous period, and rank the slices by excess failures: the change in rate multiplied by the slice's volume. A slice whose rate went from 2% to 30% on a fifth of traffic stands out. A slice that moved from 2% to 3% on a tiny tenant does not. These are aggregate questions, so the counts should be per-run counters or inverse-weighted traces, not raw stored traces.
| Attribute | What a shifted slice suggests |
|---|---|
| gen_ai.request.model vs gen_ai.response.model | A fallback model served the request, or an alias resolved to a different model |
| service.version (resource attribute) | A deploy changed prompts, tools or code |
| tenant.id, enduser.id, anthropic.workspace.id | The problem is limited to particular customers or workspaces |
| error.type | rate_limit_error or overloaded_error points at capacity, while connection_error or client_timeout_error points at transport |
| anthropic.message.stop_reason | A rise in max_tokens means answers are being truncated |
| Tool span name, e.g. anthropic.tool_use bash | One tool's error rate is driving the change |
A dimension is only sliceable if it was recorded on the run. For Agent SDK workloads, set business dimensions as OpenTelemetry resource attributes when the request is made, so that every span and metric from that call carries them. The example below attaches the end user and tenant. A service.version set the same way lets you put runs from before and after a deploy side by side.
from urllib.parse import quote
options = ClaudeAgentOptions(
env={
# ... exporter configuration from the Enable telemetry export example ...
# request is the incoming request object from your web framework.
"OTEL_RESOURCE_ATTRIBUTES": f"enduser.id={quote(request.user_id)},tenant.id={quote(request.tenant_id)}",
},
)Other first moves are slower or answer a different question. Hand-labelling a random sample of failed traces finds the most common kind of failure, which is often unrelated to what changed this week, because it has no comparison with last week. Replaying failed inputs at temperature 0 retests individual cases and says nothing about which part of the traffic moved. Adding new fields and waiting a week for data delays the diagnosis when the attributes you need are already on every trace. Slicing comes first. Reading traces from the top-ranked slice comes after it.
Checkpoint 6 of 9· Check yourself
After a release week, ranking slices by excess failures puts one cohort on top: runs where gen_ai.response.model differs from gen_ai.request.model. That cohort now fails at 28%, up from 3% the week before. What does this shifted slice point to first?
A gap between the requested model and the responding model means an alias resolved to another model or a fallback model handled the request. Truncation appears in anthropic.message.stop_reason=max_tokens, capacity problems appear in error.type, and a deploy appears in service.version.
“response.model differs from request.model when you passed an alias or a fallback model served the request”Source: platform.claude.com
Checkpoint 7 of 9· Fill the gap
A regression may be limited to particular customers. To make that cohort sliceable, every span and metric from an Agent SDK call must carry the end user and tenant. Which environment variable completes this sample?
from urllib.parse import quote
options = ClaudeAgentOptions(
env={
# ... exporter configuration from the Enable telemetry export example ...
# request is the incoming request object from your web framework.
" ? ": f"enduser.id={quote(request.user_id)},tenant.id={quote(request.tenant_id)}",
},
)Business dimensions such as enduser.id and tenant.id are set as OpenTelemetry resource attributes when the request is made, so every span and metric from that call carries them. OTLP headers only authenticate the exporter, and OTEL_SERVICE_NAME holds a single service name, not per-run dimensions.
Source: code.claude.comCheckpoint 8 of 9· Exam question
Heron Pharma runs an agent that processes prior-authorization tasks for hospital tenants across several regions. Over the past week its task failure rate climbed from 2% to 5% and is still elevated today. The 2% baseline was never zero, and the team already knows several long-standing failure causes, such as malformed payer documents, that dominate any sample of failed traces. Every trace carries model version, prompt version, tool set, tenant and region, and the platform team changed several of these during the week. A trial replay of a few failed inputs showed that many of them also fail on last week's deployment. The on-call engineer must localize the regression today, using the data already collected rather than waiting on new instrumentation. What is the best first analysis to localize the regression?
Correct answer: D — Slice runs by each logged attribute, compare each slice's failure rate with last week's, then rank slices by excess failures.
- A. Incorrect: failures were already occurring at 2% before the rise, so the most common cause in a failure sample is likely a long-standing one. Without comparing against healthy runs and last week's rates, the sample cannot reveal what changed.
- B. Incorrect: replaying failed inputs shows which cases fail again, but baseline failures reproduce too, so a repeat failure does not mark the regression. It also says nothing about which attribute values hold the extra failures.
- C. Incorrect: the traces already carry model version, prompt version, tool set, tenant and region, so analysis can begin now. Waiting a week for locale and request size delays diagnosing a regression that is still running.
- D. Correct: computing failure rates per slice against last week's baseline, with run counts as denominators, shows where the rise is concentrated. Ranking by excess failures shows how much of the increase each slice accounts for.
5.Finding loops and context growth across a session
Some systemic failures only appear when the turns of one session are read together. In a Managed Agents session, spans can belong to more than one trace. A turn started by another thread's reply has no parent request and begins a new trace. To rebuild the session, group its spans by anthropic.session.id, or group a thread by gen_ai.conversation.id and sort by start time to read it as a transcript.
With the session rebuilt, two patterns are easy to count across the whole fleet:
- Loops: runs that end with
error_max_turns, or the same tool call retried after a refusal. Count them per session and per agent rather than reading them one at a time. - Context growth:
gen_ai.usage.input_tokenson eachanthropic.model_requestrising turn after turn, followed byanthropic.compactionspans. A cohort whose input tokens per turn keep climbing pays for that context on every call.
Once the rules flag sessions like these, give the flagged sessions plus a random sample to whoever investigates. When those patterns become alert conditions, that work belongs to production monitoring and alerting.
No, not on its own. A stream's span ends when the stream ends, and if the application closes it early the SDK span is unset. A stream that never completes shows unset or ERROR. Treat UNSET on stream spans as normal and count ERROR status.
Checkpoint 9 of 9· Check yourself
In a multi-agent Managed Agents fleet, you want to count, per session, the runs that end in error_max_turns and the turns whose input tokens keep climbing. Some turns start because another thread replied. How should you assemble each session before counting?
A turn started by another thread's reply has no parent request, so it begins a new trace. Grouping by trace ID therefore splits the session. anthropic.session.id brings all of its spans back together, and gen_ai.conversation.id sorted by start time gives one thread as a transcript. anthropic.request.id only pairs the two spans of a single request, and tenant.id mixes many sessions together.
“a turn that starts because another thread replied has no parent request, so it begins a new trace that you find by anthropic.session.id.”Source: platform.claude.com
Sources
(7)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“If the sampled flag in the header is off, the API doesn’t emit the request’s span or the spans under it.”
↩︎ Sampling decides which runs you can investigate“such as when its traceparent header has the sampled flag off, your MCP server gets no traceparent.”
↩︎ Sampling decides which runs you can investigate“At least once: A destination can receive the same span more than once. To remove duplicates, use the trace ID and span ID.”
↩︎ Why rates computed from stored traces mislead“Best effort: A span can be lost before delivery. The API can’t send a lost span again.”
↩︎ Why rates computed from stored traces mislead“The SDK writes the traceparent header with your application’s propagator, and the sampled flag in it follows your sampler.”
↩︎ Exam trap 1“Best effort: A span can be lost before delivery. The API can’t send a lost span again.”
↩︎ Exam trap 2 - 2.
“Counters for tokens, cost, sessions, lines of code, and tool decisions”
↩︎ Sampling decides which runs you can investigate“service.version=1.4.0,deployment.environment=production”
↩︎ Localizing a regression by slicing into cohorts - 3.
“Sampling trades cost for coverage. It doesn't show the unread sessions are clean.”
↩︎ Sampling decides which runs you can investigate“The code scans every session, with no model calls, in two ways”
↩︎ Sampling decides which runs you can investigate“sdk_brief_04: error (error_max_turns) at event 32”
↩︎ Finding loops and context growth across a session“another try after a refusal”
↩︎ Finding loops and context growth across a session - 4.
“Don't bill from spans. Delivery is at least once, so duplicates happen, and a disabled destination or an unsampled trace drops spans.”
↩︎ Why rates computed from stored traces mislead“response.model differs from request.model when you passed an alias or a fallback model served the request.”
↩︎ Localizing a regression by slicing into cohorts“ERROR means it raised, and error.type says why: an API error type such as rate_limit_error or overloaded_error, an HTTP status such as 502”
↩︎ Localizing a regression by slicing into cohorts“Group by it and sort by start time to read one thread as a transcript.”
↩︎ Finding loops and context growth across a session“Treat UNSET on stream and event-stream spans as normal.”
↩︎ Finding loops and context growth across a session“If your sampler drops a trace, the sampled bit in traceparent is off and the API emits nothing for that request.”
↩︎ Checkpoint“response.model differs from request.model when you passed an alias or a fallback model served the request”
↩︎ Checkpoint“a turn that starts because another thread replied has no parent request, so it begins a new trace that you find by anthropic.session.id.”
↩︎ Checkpoint - 5.
“To count each token once, sum the spans of only one source for each request.”
↩︎ Aggregating spans without double counting“the SDK span is CLIENT and its child, the API span, is SERVER.”
↩︎ Aggregating spans without double counting“To count each token once, sum the spans of only one source for each request.”
↩︎ Exam trap 3 - 6.
“The input tokens, cache reads and writes included.”
↩︎ Aggregating spans without double counting“the token counts are the tokens that your organization is billed for, even when the request fails or the application cancels it.”
↩︎ Aggregating spans without double counting“for a failed or canceled request they can be lower than on the API span, or absent.”
↩︎ Aggregating spans without double counting“The ID of the workspace that the work runs in.”
↩︎ Localizing a regression by slicing into cohorts“to assemble a view of the session, such as a transcript, group its spans by these IDs.”
↩︎ Finding loops and context growth across a session“the SDK span for a stream that never completes, such as anthropic.sessions.events.stream, has the status unset or ERROR.”
↩︎ Finding loops and context growth across a session - 7.https://code.claude.com/docs/en/monitoring-usageOfficial docs
“Break down by token type, user, team, model, skill.name, plugin.name, or agent.name”
↩︎ Localizing a regression by slicing into cohorts“Break down by token type, user, team, model, skill.name, plugin.name, or agent.name”
↩︎ Key concept“Break down by token type, user, team, model, skill.name, plugin.name, or agent.name”
↩︎ Exam trap 4