Agent Observability: Challenges, Signals and Traces

1.Why agent runs resist ordinary debugging

In ordinary software, a bug usually reproduces: run the same input again and you get the same failure. Agents don't work that way. Anthropic's account of building its multi-agent research system says that agents "make dynamic decisions and are non-deterministic between runs, even with identical prompts." They also run for a long time and keep state across many tool calls, so a small early mistake carries through to the end. Users reported that agents were "not finding obvious information", and the team could not tell which of several causes was to blame: "Were the agents using bad search queries? Choosing poor sources? Hitting tool failures?"

What fixed this was instrumentation, not more careful prompting: "Adding full production tracing let us diagnose why agents failed and fix issues systematically." The team also monitored "agent decision patterns and interaction structures" while deliberately not monitoring "the contents of individual conversations, to maintain user privacy." That tension between enough detail to diagnose and little enough to protect users runs through every strategy decision in this subdomain.

For Claude-based agents, the supported route is OpenTelemetry. The Agent SDK exports traces, metrics and events to your own backend, so you can see which tools a run called, how long each model request took, how many tokens it spent, and where it failed.

Running this across a fleet adds volume and privacy concerns, and Claude Code gives you levers for each. Three are worth knowing before you look at specific signals.

First, which attributes ride on every metric datapoint. OTEL_METRICS_INCLUDE_SESSION_ID, OTEL_METRICS_INCLUDE_ACCOUNT_UUID and OTEL_METRICS_INCLUDE_RESOURCE_ATTRIBUTES all default to true. OTEL_METRICS_INCLUDE_VERSION, OTEL_METRICS_INCLUDE_ENTRYPOINT and OTEL_METRICS_INCLUDE_REPOSITORY default to false. Setting a default-true flag to false removes that attribute from metrics.

Second, content volume. Content-bearing attributes such as model responses and tool content are truncated at 60 KB by default, and CLAUDE_CODE_OTEL_CONTENT_MAX_LENGTH can be lowered to cut telemetry volume. Prompt text and tool details are off unless you enable them.

Third, who controls the configuration. When managed settings set OTEL_EXPORTER_OTLP_ENDPOINT, Claude Code removes every developer-set per-signal endpoint, so developers can't point one signal at a different collector. Exporter selectors still follow normal per-key precedence, so set them in managed settings too if you need them locked.

Sources123

2.Metrics, log events and traces answer different questions

Claude Code and the Agent SDK emit three OpenTelemetry signals. Each one has its own exporter selector, and you turn on only the ones you need. Choose by the question you're trying to answer. Metrics are aggregate counters, good for spotting cost spikes or unusual token consumption across a fleet. Log events are one structured record per occurrence, which supports tool success rates, average tool execution times and error patterns by tool type. Traces connect those occurrences into the shape of a single run.

One caveat on the table: the hook span is the exception to its "plus CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1" rule. The interaction, model-request and tool spans need only the enhanced beta flag, but claude_code.hook requires detailed beta tracing, which means ENABLE_BETA_TRACING_DETAILED=1 and BETA_TRACING_ENDPOINT. That pair also changes where your logs and traces go.

OTEL_ENV = {
    "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
    # Required for traces, which are in beta. Metrics and log events do not need this.
    " ? ": "1",
    # Choose an exporter per signal. Use otlp for the SDK; see the Note below.
    "OTEL_TRACES_EXPORTER": "otlp",
    "OTEL_METRICS_EXPORTER": "otlp",
    "OTEL_LOGS_EXPORTER": "otlp",
    # Standard OTLP transport configuration.
    "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
    "OTEL_EXPORTER_OTLP_ENDPOINT": "http://collector.example.com:4318",
    "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Bearer your-token",
}

Sources23

3.Following one run, and its subagents, through spans

A trace is a tree of spans. The root, claude_code.interaction, covers a single turn of the agent loop. Under it, each Claude API call and each tool call get their own spans. Tool spans are split into time spent waiting for permission and time spent executing, so a slow tool and a slow human approval look different. When the Agent tool starts a subagent, the subagent's own model-request and tool spans sit under that tool span.

In a multi-agent system, the attributes on the llm_request span are what make a run readable. agent_id identifies the subagent that issued a request and is absent on the main session. parent_agent_id identifies the agent that spawned it. query_source_safe gives a bounded subsystem label, such as repl_main_thread or agent.builtin.general-purpose, and is emitted even without detailed tracing. workflow.run_id ties agents back to the Workflow tool run that created them. For latency and failure analysis, the same span carries ttft_ms, attempt, error_class and stop_reason.

Sources23

4.Where telemetry config lives, and tagging it per tenant

There are two places to put the variables. In production, set them in the process environment: your shell, container or orchestrator. Every query() call then picks them up without any code change. Use per-call options (ClaudeAgentOptions.env in Python, options.env in TypeScript) when different agents in the same process need different telemetry settings. The two languages handle this differently. Python merges env on top of the inherited environment. TypeScript replaces the inherited environment entirely, so you have to spread process.env into the object, or you lose PATH and ANTHROPIC_API_KEY.

At scale, you also need to know whose run you're looking at. OTEL_SERVICE_NAME names the agent, and OTEL_RESOURCE_ATTRIBUTES can carry deployment tags or per-request identity. By default these keys are also copied onto metric datapoints, controlled by OTEL_METRICS_INCLUDE_RESOURCE_ATTRIBUTES. Separately, token metrics already carry skill.name, plugin.name and agent.name, which lets you attribute spend to specific subagent types.

That default has a cost. If you put identifiers such as enduser.id or tenant.id in OTEL_RESOURCE_ATTRIBUTES, they land on every metric datapoint, so your metrics backend stores them and metrics grow with the number of distinct users or tenants. Set OTEL_METRICS_INCLUDE_RESOURCE_ATTRIBUTES to false to keep them off metrics, and note that OTEL_METRICS_INCLUDE_SESSION_ID and OTEL_METRICS_INCLUDE_ACCOUNT_UUID are also true by default. The same identity tags still appear as resource attributes on the other signals. Decide per tenant and per signal how much identity you want in aggregate metrics, versus in logs and traces you query for a single run.

from urllib.parse import quote

options = ClaudeAgentOptions(
    env={
        # ... exporter configuration from the Enable telemetry export example ...
        # request is the incoming request object from your web framework.
        "OTEL_RESOURCE_ATTRIBUTES": f"enduser.id={quote(request.user_id)},tenant.id={quote(request.tenant_id)}",
    },
)

Sources23

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Setting CLAUDE_CODE_ENABLE_TELEMETRY=1 and OTEL_TRACES_EXPORTER=otlp is enough to get traces.Why is that wrong?

   Traces are in beta and also need CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1. Metrics and log events don't.

   Covered in Metrics, log events and traces answer different questions

2. 2.Per-call env behaves the same in the Python and TypeScript SDKs.Why is that wrong?

   Python merges env on top of the inherited environment. TypeScript replaces it entirely, so you must spread process.env.

   Covered in Where telemetry config lives, and tagging it per tenant