Tracing and Logging AI Apps with TruLens and the AI Observability Event Table

1.What AI observability covers, and where TruLens fits

Snowflake AI observability is a set of features inside Cortex products. It answers questions like these: what happened in one production request, how well an application scores on test data, and what a request cost. The features don't all expose the same surfaces. Native features such as Cortex Agents log their traces automatically. Other features show up only in Account Usage views. Custom applications you build yourself send telemetry into your account through the TruLens SDK. Those apps can run on Snowflake compute or outside Snowflake.

The deciding question is whether a workload writes to the shared event table. Cortex Agents do this automatically: they log threads, turns, and execution spans. Cortex Search writes one row per request, but only when REQUEST_LOGGING is enabled on the service. Cortex Analyst is the odd one out. Its direct request and response logs go to its own table, and they reach the shared event table only when an agent calls Analyst as a tool. The REST API and built-in functions produce usage rows and no traces.

Use TruLens when you own the application from end to end. Examples include a RAG pipeline that combines Cortex Search with AI_COMPLETE, an agent running on Snowpark Container Services, or a workload on another cloud. If what you built is a native Cortex Agent, you don't need TruLens. Snowflake's own agent monitoring already covers it. TruLens is an open source SDK. To use it with Snowflake, install version 2.1.2 or later of trulens-core, trulens-connectors-snowflake, and trulens-providers-cortex.

Sources12

2.Tracing: instrumenting functions and span types

TruLens exports OpenTelemetry-style traces into your account. You choose what gets traced by adding the @instrument() decorator to your functions. Each decorated call becomes a span that records the function's inputs, outputs, and latency. You can also give each span a type, which makes the trace easier to read in Snowsight. Use RETRIEVAL for search or retrieval functions, GENERATION for LLM inference calls, and RECORD_ROOT for the application's main entry point.

@instrument(span_type=SpanAttributes.SpanType.RETRIEVAL)
def retrieve_context(self, query: str) -> list:
    return self.retrieve(query)

@instrument(span_type=SpanAttributes.SpanType.GENERATION)
def generate_completion(self, query: str, context_str: list) -> str:
    return response

@instrument(span_type=SpanAttributes.SpanType.RECORD_ROOT)
def answer_query(self, query: str) -> str:
    context_str = self.retrieve_context(query)
    return self.generate_completion(query, context_str)

Span attributes bring specific values to the surface of a trace. You can set them with a static mapping, or with a lambda that receives the function's return value, any exception it raised, and its arguments. That way an attribute can be computed from what actually happened during the call. Hand-decorating every function isn't the only option. TruLens ships wrappers that instrument popular frameworks automatically: TruChain for LangChain and LCEL chains, TruGraph for LangGraph applications, and TruLlama for LlamaIndex query engines and retrievers.

Sources2

3.Registering the app as an External Agent

Instrumentation alone writes nothing to Snowflake. You also have to register the application with TruApp. Registration lets TruLens write traces to AI_OBSERVABILITY_EVENTS, and it creates an External Agent object for governance. That object holds only the application name, version, and run name. Code, prompts, traces, and scores are not stored in it. The app_version label is what later lets you run experiments and compare versions. If you use a framework wrapper (TruChain, TruGraph, TruLlama), registration happens as part of wrapping the app.

tru_app = TruApp(
    app: Any,
    app_name: str,
    app_version: str,
    connector: SnowflakeConnector,
    main_method: callable  # for example app.query
)

One naming rule catches people out. External Agent objects share a namespace with model objects in the same schema. If a model you logged with log_model() already uses a name, you can't create an External Agent with that name. You have to rename or drop one of the two objects. For live production tracing, not just batch tests, put the @trace_with_run decorator on the function that calls your app, such as an API handler. Each call then records spans into a live run.

@ ? (app=tru_app, run_name=run_name)
def process_message(message: str) -> str:
    return agent_app.ask(message)

Sources23

4.The event table: querying traces and controlling access

Traces and evaluation results for TruLens apps are stored in SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS. Rows can't be modified after ingestion. Two application roles control the table. SNOWFLAKE.AI_OBSERVABILITY_READER gives read-only access, and SNOWFLAKE.AI_OBSERVABILITY_ADMIN can also delete rows. To read the traces for one application, call GET_AI_OBSERVABILITY_EVENTS with agent_type EXTERNAL AGENT, then filter on RECORD_ATTRIBUTES (for example, by run name) to narrow the results.

SELECT *
  FROM TABLE(SNOWFLAKE.LOCAL.GET_AI_OBSERVABILITY_EVENTS(
    '<database_name>',
    '<schema_name>',
    '<external_agent_name>',
    'EXTERNAL AGENT'
  ));

For an External Agent, USAGE on the object is enough to call this function. The MONITOR privilege doesn't apply here, and changing or dropping the agent requires OWNERSHIP. The separate account privilege READ UNREDACTED AI OBSERVABILITY EVENTS TABLE decides whether a role sees full tool inputs, outputs, and conversation text. Without it, the role still sees metadata such as tool names, latency, and token usage. The event table is also the wrong place to total up spend. Trace delivery is best effort, so authoritative cost figures come from the Account Usage views.

Sources21

5.Logging: warnings and errors from TruLens runs

Traces show what your application did. Logs show what went wrong while a TruLens run was executing. For those you use a second table function, GET_AI_OBSERVABILITY_LOGS. It takes the same database, schema, External Agent name, and 'EXTERNAL AGENT' arguments. Filter it on severity and run name to find the warnings and errors from one evaluation.

SELECT *
  FROM TABLE(SNOWFLAKE.LOCAL.GET_AI_OBSERVABILITY_LOGS(
    '<database_name>',
    '<schema_name>',
    '<external_agent_name>',
    'EXTERNAL AGENT'
  ))
  WHERE record:"severity_text" IN ('ERROR', 'WARN')
    AND record_attributes:"snow.ai.observability.run.name" = '<run_name>';

Only two fields in these logs are guaranteed to stay stable: record:"severity_text" and record_attributes:"snow.ai.observability.run.name". Other fields may change, so durable alerting queries should filter only on these two. Logging means something slightly different for native features. Cortex Search request logging, for example, is a per-service setting. Once you turn it on, each request lands in the same event table, where you can query it with agent_type CORTEX SEARCH SERVICE.

Sources2

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.You can add up spend accurately by summing token or cost data from AI_OBSERVABILITY_EVENTS.Why is that wrong?

   Trace delivery is best effort, so totals from the event table can under-report. Spend reporting belongs on the Account Usage views.

   Covered in The event table: querying traces and controlling access

2. 2.Querying an External Agent's traces requires the MONITOR privilege, as it does for Cortex Search.Why is that wrong?

   When agent_type is EXTERNAL AGENT, USAGE on the External Agent is enough. MONITOR doesn't apply.

   Covered in The event table: querying traces and controlling access

3. 3.Without READ UNREDACTED AI OBSERVABILITY EVENTS TABLE, a role sees nothing in a trace.Why is that wrong?

   Without the grant, metadata such as tool names, latency, and token usage is still visible. Only the full inputs, outputs, and conversation text are redacted.

   Covered in The event table: querying traces and controlling access