What you will be able to do
- Explain why transport-level signals such as status codes, latency and token spend miss failures where a tool returns a successful but useless result
- Choose the per-tool, per-agent monitoring strategy that would have surfaced a failure that went undetected in a multi-agent architecture
- Enable metrics, log events and traces for the Agent SDK and Claude Code, including the beta flag that traces require
- Choose the content gate that exposes tool commands and paths without recording prompt text
- Pin every installation to one central OTLP collector through managed settings
Key concept
Instrumentation gap — A failure that no collected signal reacts to, because every metric being watched measures whether a call happened and how fast, not what came back. To close the gap, find the signal the failure would actually change and monitor that one.
1.Why a scaled agent can fail without raising an alarm
Most agent dashboards watch the transport layer: did the API call succeed, how long did it take, how many tokens did it cost. Each claude_code.llm_request span carries exactly these attributes: success, status_code, an error_class such as api_timeout or server_overload, duration_ms (wall-clock time including retries) and ttft_ms. These are the right signals for overloads, timeouts and retry storms, which production reliability controls are built to handle. None of them tells you whether a tool's answer was any use.
A transcript and an outcome are different things. A flight-booking agent can say "Your flight has been booked" while no reservation exists in the database. Monitoring that checks only the shape of a transcript (calls were made, calls returned) has the same blind spot. A tool that quietly degrades produces a transcript that looks normal, and replies that look fine to the customer, for as long as nobody measures what the tool returned.
Scale makes this worse. With one agent and one tool, a person reading a few sessions might notice the empty searches. With an orchestrator delegating to six subagents, each with its own tools, one tool's failure is a small fraction of total traffic, so aggregate dashboards stay flat.
| Signal | Reacts to | Blind to |
|---|---|---|
| success, status_code, error_class on claude_code.llm_request | Failed API requests, timeouts, server overload | A successful request that was fed an empty or wrong tool result |
| duration_ms, ttft_ms | Slow requests and retries | Wrong answers delivered at normal speed |
| claude_code.token.usage | Cost spikes, unusual token consumption | Failures that leave spend roughly unchanged |
| tool_result events, tool success rates, error patterns by tool type | Per-tool behaviour: what each tool did and returned | Whether the result was correct, unless tool content or details are captured |
Checkpoint 1 of 8· Match them up
Each failure below happens in a scaled multi-agent deployment. Match each one to the Claude Code signal that would change when it happens.
Tap a term, then the definition that fits it.
Transport attributes such as error_class and duration_ms, and token metrics, react only to failed, slow or costly calls. A tool that quietly returns empty results leaves all of them unchanged, so only per-tool result signals can show it.
“tool_decision and tool_result: each tool call, whether it was allowed or denied, and its result”Source: code.claude.com
2.Selecting the monitor that would have surfaced the failure
Start from the failure and work backwards. Ask which measurable quantity actually changed, then check whether anything was watching it. In the empty-search case, latency was normal, every call returned HTTP 200 and spend didn't move. The one thing that changed was the content of one tool's results: they went from populated to empty. The monitor that would have caught it tracks that quantity per tool, such as an empty-result rate, and alerts when it departs from that tool's own normal level.
Claude Code's monitoring guidance lists per-tool views as analysis targets alongside cost spikes: most frequently used tools, tool success rates, average tool execution times, and error patterns by tool type. Managed Agents, a beta Claude API feature in which agents run as stateful sessions that you inspect in the Claude Console, and which is separate from the Agent SDK and Claude Code, has the same per-tool view in the session viewer's Tools tab, which shows each configured tool along with call counts, failures, and median duration. To judge whether a result was empty, you need the result itself. In the Agent SDK, OTEL_LOG_TOOL_CONTENT=1 adds a tool.output span event with file contents, Bash output, and whatever MCP tools, WebFetch and WebSearch return. It requires tracing to be enabled.
Attribution makes a per-tool signal usable at scale. On each model request span, agent_id identifies the subagent or teammate that issued it (it's absent on the main session), parent_agent_id identifies the agent that spawned it, and query_source_safe names the issuing subsystem. Token metrics can be broken down by agent.name, so spend can be attributed to specific subagent types. Within one process, the Agent SDK can give each agent its own service identity through per-call env options. In the snippet below, the comment line stands in for the exporter setup: CLAUDE_CODE_ENABLE_TELEMETRY=1, an OTEL_*_EXPORTER selector for each signal, and the OTLP protocol, endpoint and headers.
options = ClaudeAgentOptions(
env={
# ... exporter configuration from the Enable telemetry export example ...
"OTEL_SERVICE_NAME": "support-triage-agent",
"OTEL_RESOURCE_ATTRIBUTES": "service.version=1.4.0,deployment.environment=production",
},
)Signals tied to the cause help, but they only fire when the failure shows up at their layer. Managed Agents sends a vault_credential.refresh_failed webhook when MCP OAuth credentials cannot refresh, and mcp_server_connection events record server connection failures. Use them to catch a credential that breaks the connection itself. A tool whose backing index silently returns nothing behind a healthy connection still needs the result-level monitor. Webhooks can also be missed, which is why the guidance is to also list sessions on a schedule.
Checkpoint 2 of 8· Check yourself
After an index migration, the document-lookup MCP tool used by one of six subagents starts returning no matches. The MCP connection stays up, every call returns HTTP 200 with normal latency, and spend does not change. Which monitor would have caught this?
The only thing that changed was what one tool returned, so the monitor has to measure each tool's results against that tool's baseline. Capturing tool output (for example with OTEL_LOG_TOOL_CONTENT) makes this possible. Latency and spend did not move, and connection or credential signals stay quiet because the connection was healthy.
“A tool.output span event on claude_code.tool with file contents, Bash output, and what MCP tools, WebFetch, and WebSearch return”Source: code.claude.com
Checkpoint 3 of 8· Exam question
A support platform runs an orchestrator that delegates to six subagents, each calling its own tools. For three weeks the retrieval subagent's search tool returned empty results because its index credential had expired. Every API call returned HTTP 200, latency stayed normal, and the agent answered from general model knowledge, so customers got plausible but ungrounded replies. Which monitoring strategy would have surfaced this failure?
Correct answer: A — Alert on the empty-result rate of each tool call, firing when a tool's rate departs from its trailing four-week baseline.
- A. Correct. The fault was a search tool returning nothing while everything else looked healthy. A per-tool empty-result rate compared with its own baseline shows the step change that began when the credential expired.
- B. Incorrect. Latency stayed normal throughout the three weeks, so a latency baseline alert on tool calls would not have fired. An empty result from an expired credential does not make a call slower.
- C. Incorrect. A budget-curve alert fires only when spend rises above forecast. Empty retrievals put less content into the subagent's context, so spend would stay flat or drop, and the alert would never have fired for this failure.
- D. Incorrect. Every call returned HTTP 200, so the tool's error rate stayed at zero, well below the one percent threshold. The failure showed up in the content of the response, not in its status code.
3.Enabling metrics, log events and traces
The Agent SDK exports three OpenTelemetry signals, and each one answers different questions. Metrics are counters for tokens, cost, sessions, lines of code and tool decisions. Log events are structured records for each prompt, API request, API error and tool result. Traces are spans for each interaction, model request, tool call and hook. Each has its own exporter variable. Traces are in beta and also need CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1. Setting OTEL_TRACES_EXPORTER alone produces no spans, and with no spans, OTEL_LOG_TOOL_CONTENT has nothing to attach the tool output to.
OTEL_ENV = {
"CLAUDE_CODE_ENABLE_TELEMETRY": "1",
# Required for traces, which are in beta. Metrics and log events do not need this.
"CLAUDE_CODE_ENHANCED_TELEMETRY_BETA": "1",
# Choose an exporter per signal. Use otlp for the SDK; see the Note below.
"OTEL_TRACES_EXPORTER": "otlp",
"OTEL_METRICS_EXPORTER": "otlp",
"OTEL_LOGS_EXPORTER": "otlp",
# Standard OTLP transport configuration.
"OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
"OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example.com:4318",
"OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer your-token",
}The sample selects otlp for all three signals, which sends them to the collector named by OTEL_EXPORTER_OTLP_ENDPOINT. Other selector values exist (console, and none to disable a signal), but the SDK sample uses otlp throughout. There are two places to set these variables. For production, set them in the process environment (shell, container or orchestrator) before the application starts, and every query() call picks them up with no code change. Use per-call env when agents in the same process need different telemetry settings, for example a different service name per subagent. The two SDKs differ here: in Python, env is merged on top of the inherited environment, but in TypeScript it replaces the environment entirely. A TypeScript agent that passes only the telemetry variables loses PATH and its API key unless it spreads ...process.env first.
claude_code.tool.blocked_on_user covers the permission wait, and claude_code.tool.execution covers the execution itself. A tool that looks slow because it's waiting on a human shows up in the first, not the second.
Checkpoint 4 of 8· Fill the gap
A team's collector receives metrics and log events from their Agent SDK service, but no spans show up, so OTEL_LOG_TOOL_CONTENT never attaches any tool.output. Which variable fills the blank in this configuration?
OTEL_ENV = {
"CLAUDE_CODE_ENABLE_TELEMETRY": "1",
# Required for traces, which are in beta. Metrics and log events do not need this.
" ? ": "1",
# Choose an exporter per signal. Use otlp for the SDK; see the Note below.
"OTEL_TRACES_EXPORTER": "otlp",
"OTEL_METRICS_EXPORTER": "otlp",
"OTEL_LOGS_EXPORTER": "otlp",
# Standard OTLP transport configuration.
"OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
"OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example.com:4318",
"OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer your-token",
}Traces are in beta. Setting OTEL_TRACES_EXPORTER is not enough on its own: you also need CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1. Without it, no spans are produced, and OTEL_LOG_TOOL_CONTENT has no span to attach tool output to. Metrics and log events work without this flag, which is why they still arrived.
Source: code.claude.com4.Content gates: exposing tool commands without exposing prompts
By default, telemetry records that things happened, not what they contained. The user_prompt attribute on interaction spans reads <REDACTED> unless its gate is set. A tool_result event shows that a Bash tool ran, but not which command it ran. Each kind of content has its own opt-in variable, so a team can expose exactly the content it needs for an investigation and nothing more.
| Variable | Adds | Note |
|---|---|---|
| OTEL_LOG_USER_PROMPTS=1 | Prompt text on claude_code.user_prompt events and the claude_code.interaction span | Has no effect on tool arguments |
| OTEL_LOG_TOOL_DETAILS=1 | Tool input arguments (file paths, shell commands, search patterns) on tool_result events; real agent, skill, plugin and MCP server names on cost and token metrics | The gate for which command ran |
| OTEL_LOG_TOOL_CONTENT=1 | tool.output span event with file contents, Bash output and MCP, WebFetch and WebSearch results, truncated at 60 KB by default | Requires tracing to be enabled |
| OTEL_LOG_RAW_API_BODIES | Full Messages API request and response JSON, including the entire conversation history | Implies consent to everything the other three reveal |
Raw API bodies are the wrong tool for a no-prompts rule. They contain the entire conversation history, so the prompt text is already collected at capture time. Filtering it out later doesn't undo that. Enabling raw bodies counts as consent to everything the other three gates would reveal. For questions like "which command ran" and "which file was touched", set OTEL_LOG_TOOL_DETAILS=1 and leave OTEL_LOG_USER_PROMPTS unset. For MCP calls, tool details also replace the redacted literal mcp_tool with the real server and tool names.
Checkpoint 5 of 8· Check yourself
A security lead has a plan for a team bound by a no-prompt-text retention policy. They want to set OTEL_LOG_RAW_API_BODIES=1 and add a collector rule that deletes prompt text before anything is stored. They argue this keeps the policy while giving incident reviewers every tool call in full. Why is this the wrong choice?
Raw API bodies record the full request and response, including the whole conversation, so the prompt text has already been collected before any downstream filter runs. The right gate for showing which command ran is OTEL_LOG_TOOL_DETAILS=1, with OTEL_LOG_USER_PROMPTS left unset.
“Bodies include the entire conversation history and have extended-thinking content redacted. Enabling this implies consent to everything the three variables above would reveal”Source: code.claude.com
Checkpoint 6 of 8· Exam question
A compliance team requires that Claude Code telemetry never store prompt text. In an incident review, the security team found that tool-result events showed a shell tool had run, but not which command or file paths were involved. The platform owner must close this gap before the next audit and keep compliance's sign-off. Which change should the architect propose?
Correct answer: A — Enable the tool-detail logging switch in the centrally managed settings and leave the prompt-logging switch off, so reviewers can see the shell commands and file paths involved.
- A. Correct. Tool arguments such as shell commands and file paths are controlled by their own logging switch, separate from the one that records prompt text. Enabling it in managed settings fills the gap in tool-result events while prompt text stays out of telemetry.
- B. Incorrect. The prompt-logging switch records prompt text, which compliance forbids storing, and a limited window or restricted access does not change that. It also does not govern tool arguments, which have their own separate switch.
- C. Incorrect. A self-hosted collector changes only where telemetry is delivered and who can read it. The fields in each event are decided by the logging switches on the client, so tool-result events would still lack commands and paths.
- D. Incorrect. Full request and response bodies contain the prompts, so writing them to any store means prompt text is stored, even if a later job removes it. That breaks the rule that telemetry never keeps prompt text, and it bypasses the tool-detail switch built for this need.
5.Pinning every installation to one collector
A gap can also come from telemetry that leaves the machine but lands in the wrong place. If each developer chooses their own exporter destination, some installations will report somewhere the security team never looks. In Claude Code, setting OTEL_EXPORTER_OTLP_ENDPOINT in managed settings removes every per-signal endpoint a developer has set. Developers then can't point one signal at a different collector, and administrators don't need to set each per-signal endpoint variable separately. A setting committed to a repository or written by an onboarding script doesn't give that guarantee. It's an ordinary file that developers can edit, so the override only comes from the managed setting.
Once telemetry reaches one place, put identity on it. OTEL_RESOURCE_ATTRIBUTES can carry enduser.id and tenant.id, and the security events then support the questions that matter after an incident: tool_decision records whether each call was allowed or denied and where the decision came from, permission_mode_changed records escalations, and mcp_server_connection records connections and connection failures. For tracing one specific incident through these records, see root-cause isolation from logs and traces.
Checkpoint 7 of 8· Match them up
All installations now report to one central collector. During a post-incident review, the security team has four questions. Match each record or attribute to the question it answers.
Tap a term, then the definition that fits it.
Once telemetry lands in one place, each security event answers a different question: tool_decision covers allow/deny and where the decision came from, permission_mode_changed covers escalations, and mcp_server_connection covers server connections and failures. Identity resource attributes such as enduser.id link all of these records to a specific user.
“Tool call allowed or denied, and where the decision came from, such as configuration, a hook, or the user | tool_decision”Source: code.claude.com
Checkpoint 8 of 8· Exam question
A security architect wants every Claude Code installation in the company to send telemetry to one central OTLP collector, and wants developers unable to redirect it with their own per-signal endpoints or credentials. Which configuration achieves this?
Correct answer: A — Set OTEL_EXPORTER_OTLP_ENDPOINT and headers in the managed settings file, which removes developer-set per-signal endpoints and credentials.
- A. Correct. Managed settings apply organization-wide and take precedence over developer-set endpoints and credentials, which locks the destination.
- B. Incorrect. Claude Code ignores OTEL exporter variables in repository settings files, so no telemetry would be sent this way.
- C. Incorrect. Local settings files are also ignored for OTEL exporter variables, and developers could edit them anyway.
- D. Incorrect. Claude Code does not pass OTEL variables to Bash, hooks or MCP servers, and a hook is not a configuration channel for the exporter.
Sources
(5)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://code.claude.com/docs/en/monitoring-usageOfficial docs
“Short error class token when the request failed, such as api_timeout or server_overload.”
↩︎ Why a scaled agent can fail without raising an alarm“Wall-clock duration including retries”
↩︎ Why a scaled agent can fail without raising an alarm“Error patterns by tool type”
↩︎ Selecting the monitor that would have surfaced the failure“Identifier of the subagent or teammate that issued the request. Absent on the main session”
↩︎ Selecting the monitor that would have surfaced the failure“Attributing spend to specific skills, plugins, or subagent types via the skill.name, plugin.name, and agent.name attributes”
↩︎ Selecting the monitor that would have surfaced the failure“Traces exporter types, comma-separated. Use none to disable”
↩︎ Enabling metrics, log events and traces“Prompt text. Value is <REDACTED> unless the gate is set”
↩︎ Content gates: exposing tool commands without exposing prompts“Claude Code removes every developer-set per-signal endpoint”
↩︎ Pinning every installation to one collector“Tool call allowed or denied, and where the decision came from, such as configuration, a hook, or the user | tool_decision”
↩︎ Checkpoint - 2.
“The outcome is the final state in the environment at the end of the trial.”
↩︎ Why a scaled agent can fail without raising an alarm“The outcome is the final state in the environment at the end of the trial.”
↩︎ Key concept“The outcome is the final state in the environment at the end of the trial.”
↩︎ Exam trap 1 - 3.
“along with call counts, failures, and median duration”
↩︎ Selecting the monitor that would have surfaced the failure“Inspect a session in the Claude Console, read its token usage and list cost, and debug unexpected agent behavior.”
↩︎ Selecting the monitor that would have surfaced the failure - 4.
“A tool.output span event on claude_code.tool with file contents, Bash output, and what MCP tools, WebFetch, and WebSearch return”
↩︎ Selecting the monitor that would have surfaced the failure“OTEL_TRACES_EXPORTER plus CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1”
↩︎ Enabling metrics, log events and traces“This is the recommended approach for production deployments.”
↩︎ Enabling metrics, log events and traces“In TypeScript, env replaces the inherited environment entirely, so include ...process.env in the object you pass.”
↩︎ Enabling metrics, log events and traces“Bodies include the entire conversation history and have extended-thinking content redacted.”
↩︎ Content gates: exposing tool commands without exposing prompts“Enabling this implies consent to everything the three variables above would reveal”
↩︎ Content gates: exposing tool commands without exposing prompts“tool_decision and tool_result: each tool call, whether it was allowed or denied, and its result”
↩︎ Pinning every installation to one collector“OTEL_TRACES_EXPORTER plus CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1”
↩︎ Exam trap 2“Tool input arguments such as file paths, shell commands, and search patterns on claude_code.tool_result events”
↩︎ Exam trap 3“Enabling this implies consent to everything the three variables above would reveal”
↩︎ Exam trap 4“tool_decision and tool_result: each tool call, whether it was allowed or denied, and its result”
↩︎ Checkpoint“Tool input arguments such as file paths, shell commands, and search patterns on claude_code.tool_result events”
↩︎ Prediction“Bodies include the entire conversation history and have extended-thinking content redacted. Enabling this implies consent to everything the three variables above would reveal”
↩︎ Checkpoint - 5.
“MCP OAuth credentials that cannot refresh”
↩︎ Selecting the monitor that would have surfaced the failure“You can miss a webhook delivery, so also list your sessions on a schedule and check the idle ones.”
↩︎ Selecting the monitor that would have surfaced the failure