What you will be able to do
- Explain what Snowflake AI Observability does for a custom generative AI application and where it stops
- Instrument app functions with TruLens @instrument() and choose RECORD_ROOT, RETRIEVAL or GENERATION span types
- Register an instrumented app as an External Agent and say what that object does and does not store
- Query traces and run logs in SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS and explain who can read them
- Create, invoke and score evaluation runs with LLM-as-a-judge and custom metrics, then compare runs
- Grant the privileges a role needs to register apps, execute runs and compute judge metrics
Key concept
External Agent plus the AI observability event table — A TruLens-instrumented application is registered in Snowflake as an External Agent object. That object only holds metadata such as name and version. Every trace, span and evaluation score the app produces goes into the shared SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS table.
1.What AI Observability does, and for which apps
Snowflake AI Observability shows you what a generative AI application did and how good its answers were. It is a set of features across Cortex products that answers questions like these: what happened in one specific production request, how well the app performs on test data, and what a request cost. For an application you build yourself, it does two jobs. It traces each execution step by step, and it evaluates output quality with LLM-as-a-judge metrics. Both jobs are built on TruLens, an open source observability SDK that Snowflake integrates.
Which tool you use depends on who built the app. Native features handle their own telemetry. A Cortex Agent deployed through the Agent API logs threads, turns and spans automatically, so it needs no TruLens. TruLens is for applications you own end to end. Examples are a RAG pipeline that combines Cortex Search with AI_COMPLETE, a graph of Cortex Agents, or a custom retrieval chain. The app can run on Snowpark Container Services, on another cloud, or on-premises, and it still streams its traces into your Snowflake account.
Traces and scores are evidence that you read and act on. They are not a control mechanism, and they are not a billing ledger. Trace delivery is best effort, so totals added up from trace data can under-report consumption. Spend reporting and caps belong to the Account Usage views and budgets (cost monitoring), not to the observability event table.
Checkpoint 1 of 10· Check yourself
Your team needs step-by-step traces and LLM-judge quality scores. Which workload is the one where you should instrument and register the app with TruLens?
TruLens is for custom applications you own end to end, and they can run outside Snowflake and still stream traces into your account. A Cortex Agent logs its threads, turns and spans automatically, so it needs no TruLens. Spend reporting and caps belong to the Account Usage views and budgets. AI Observability is not a control mechanism for data access.
“Custom AI applications you host through Snowflake products, or even outside of Snowflake, can stream telemetry information into your account with TruLens.”Source: docs.snowflake.com
Checkpoint 2 of 10· Exam question
What is the primary purpose of Snowflake AI Observability?
Correct answer: A — Trace executions and compute LLM-as-judge quality metrics for generative AI apps, built on TruLens
- A. AI Observability captures traces of an application's inputs, outputs, and intermediate steps and uses LLM-as-judge scoring to compute quality metrics, built on the open-source TruLens framework integrated with Snowflake.
- B. Row access policies are a data-governance feature unrelated to observability; they restrict row visibility for queries but do not trace or evaluate AI application behavior.
- C. Observability records traces and metrics for analysis by developers, but it does not automatically retrain or fine-tune the underlying LLMs based on feedback.
- D. Resource monitors control credit spend by suspending warehouses; observability is a separate capability focused on quality and tracing, not spend enforcement.
2.Instrumenting app functions with TruLens span types
Instrumentation begins with three Python packages at version 2.1.2 or later: trulens-core, trulens-connectors-snowflake and trulens-providers-cortex. You then put the @instrument() decorator on each function you want traced. A decorated function becomes a span that records its inputs, its outputs and its latency.
A span type tells Snowflake what role the function plays in the app. There are three:
RECORD_ROOTmarks the main entry point. One call to it is one record: the whole invocation, from the user's question to the final answer.RETRIEVALmarks a retrieval or search function.GENERATIONmarks an LLM inference call.
@instrument(span_type=SpanAttributes.SpanType.RETRIEVAL)
def retrieve_context(self, query: str) -> list:
return self.retrieve(query)
@instrument(span_type=SpanAttributes.SpanType.GENERATION)
def generate_completion(self, query: str, context_str: list) -> str:
return response
@instrument(span_type=SpanAttributes.SpanType.RECORD_ROOT)
def answer_query(self, query: str) -> str:
context_str = self.retrieve_context(query)
return self.generate_completion(query, context_str)Span types are what make metrics possible, not just tidier traces. A metric is computed from span attributes, and each span type carries a fixed set of them. You map a function parameter, or its return value, to an attribute through the attributes argument of @instrument(). For example, the query parameter can map to RETRIEVAL.QUERY_TEXT and return to RETRIEVAL.RETRIEVED_CONTEXTS. The attributes argument also accepts a lambda that receives ret, exception, *args and **kwargs, so you can compute an attribute from the call, such as pulling just the text field out of each returned document.
| Span type | Marks | Supported attributes | Metrics that depend on it |
|---|---|---|---|
| RECORD_ROOT | Main entry point: overall input and output of one invocation | INPUT, OUTPUT, GROUND_TRUTH_OUTPUT | answer_relevance, correctness, coherence; OUTPUT also feeds groundedness |
| RETRIEVAL | Retrieval or search function | QUERY_TEXT, RETRIEVED_CONTEXTS | context_relevance; RETRIEVED_CONTEXTS also feeds groundedness |
| GENERATION | LLM inference call | None | None; the span adds trace readability |
Apps built on a common framework don't need hand-placed decorators. TruLens provides wrappers that instrument them automatically: TruChain for LangChain and LCEL chains, TruGraph for LangGraph, and TruLlama for LlamaIndex query engines and retrievers.
Checkpoint 3 of 10· Fill the gap
In this RAG app, answer_query takes the user's question and returns the final answer. Which span type completes its decorator so that each call counts as one record, and so that answer_relevance, correctness and coherence can be computed?
@instrument(span_type=SpanAttributes.SpanType.RETRIEVAL)
def retrieve_context(self, query: str) -> list:
return self.retrieve(query)
@instrument(span_type=SpanAttributes.SpanType.GENERATION)
def generate_completion(self, query: str, context_str: list) -> str:
return response
@instrument(span_type=SpanAttributes.SpanType. ? )
def answer_query(self, query: str) -> str:
context_str = self.retrieve_context(query)
return self.generate_completion(query, context_str)RECORD_ROOT marks the app's main entry point, and one call to it is one record. It carries the INPUT, OUTPUT and GROUND_TRUTH_OUTPUT attributes that answer_relevance, correctness and coherence need. GENERATION is for the LLM call and supports no attributes. RETRIEVAL is for the search step.
Source: docs.snowflake.comCheckpoint 4 of 10· Exam question
An engineer instruments the top-level entry-point function of a custom RAG application with TruLens and tags its span type as RECORD_ROOT. What is the purpose of the RECORD_ROOT span type?
Correct answer: A — It marks the overall input and output of a single application invocation, anchoring record-level metrics such as answer relevance and correctness
- A. RECORD_ROOT designates the top-level call whose input and output represent the whole record, which is where record-level metrics like answer relevance and correctness attach.
- B. Retrieval functions are tagged with the RETRIEVAL span type, not RECORD_ROOT, so that context relevance can be computed against the retrieved chunks specifically.
- C. Generation functions are tagged with the GENERATION span type; RECORD_ROOT is not limited to token or latency capture and covers the whole record instead.
- D. Writing to the event table is an internal platform mechanism, not something a developer tags with a span type, and RECORD_ROOT has no connection to storage cost attribution.
3.Registering the app as an External Agent
Instrumented code sends nothing to Snowflake until you register the app. You register it with TruApp, and registration does two things. It lets TruLens write traces to the event table, and it creates an External Agent object, which you govern like any other schema object. The framework wrappers register the app automatically when they wrap it.
tru_app = TruApp(
app: Any,
app_name: str,
app_version: str,
connector: SnowflakeConnector,
main_method: callable # for example app.query
)The parameters:
app_nameis the name Snowflake stores for the application.app_versionis the label you later compare experiments by.connectoris aSnowflakeConnectorthat manages the Snowpark session.main_methodis optional when the entry point already carriesRECORD_ROOTinstrumentation.
Traces are exported only for calls made while the recorder is active.
An External Agent object is small. It holds only the application name, version and run name. It contains no code, prompts, traces or scores. Two consequences matter in practice. First, External Agents share a namespace with model objects in the same schema, so registration fails if a model in that schema already uses the name. Second, the role that registers needs CREATE EXTERNAL AGENT on the schema, and changing or dropping the object later with SQL requires OWNERSHIP.
Checkpoint 5 of 10· Check yourself
A data engineer calls TruApp with app_name="SUPPORT_RAG" to register an app in schema APPS.ML. The app's entry point is decorated with RECORD_ROOT, so main_method is omitted. The role has CREATE EXTERNAL AGENT on APPS.ML, and no External Agent named SUPPORT_RAG exists there. Registration still fails. What is the most likely cause?
External Agents share a namespace with model objects in the same schema, so an existing model with that name blocks registration. CREATE EXTERNAL AGENT is the privilege needed to register; OWNERSHIP is only needed later to change or drop the object. main_method is optional when RECORD_ROOT marks the entry point, and apps hosted outside Snowflake can still register and stream traces.
“External Agent objects share a namespace with model objects in the same schema.”Source: docs.snowflake.com
4.The AI observability event table, logs and access
SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS is a Snowflake-managed table, and TruLens apps are only one of the features that write to it. You don't create it or configure it. Its rows can't be modified after ingestion. To read it, you call the table function SNOWFLAKE.LOCAL.GET_AI_OBSERVABILITY_EVENTS with the database, schema and External Agent name and the agent type 'EXTERNAL AGENT'. Results use the event table's column layout. To narrow them to one evaluation run, filter on RECORD_ATTRIBUTES, for example by run name.
| Feature | Where its telemetry goes |
|---|---|
| TruLens External Agents | AI_OBSERVABILITY_EVENTS (traces and evaluation scores) |
| Cortex Agents | AI_OBSERVABILITY_EVENTS, logged automatically |
| Cortex Search | AI_OBSERVABILITY_EVENTS, one row per request, only when REQUEST_LOGGING is enabled |
| Direct Cortex Analyst requests | SNOWFLAKE.LOCAL.CORTEX_ANALYST_REQUESTS_RAW, not the shared event table |
| Built-in functions such as AI_COMPLETE, and the REST API | No traces; usage only in Account Usage views |
Run logs are separate from traces. Warnings and errors raised during a TruLens run come from a second function, GET_AI_OBSERVABILITY_LOGS, which takes the same arguments. Only two log fields are guaranteed to stay stable: record:"severity_text" and the run name in record_attributes. Other fields may change, so filter on those two.
SELECT *
FROM TABLE(SNOWFLAKE.LOCAL.GET_AI_OBSERVABILITY_LOGS(
'<database_name>',
'<schema_name>',
'<external_agent_name>',
'EXTERNAL AGENT'
))
WHERE record:"severity_text" IN ('ERROR', 'WARN')
AND record_attributes:"snow.ai.observability.run.name" = '<run_name>';Access to this data has three layers:
- Calling the function. For an External Agent,
USAGEon the agent is enough to call the table functions.MONITORdoes not apply. - Reading or deleting rows directly. The
SNOWFLAKE.AI_OBSERVABILITY_READERapplication role gives read-only access to the table.SNOWFLAKE.AI_OBSERVABILITY_ADMINcan delete rows. - Seeing sensitive content. The account privilege
READ UNREDACTED AI OBSERVABILITY EVENTS TABLEdecides whether a role sees unredacted tool inputs, outputs and conversation text in Snowsight and in the UDTFs. Without it, the role still sees metadata such as tool names, latency and token usage. The privilege does not change how evaluation runs execute or how their scores appear.
Checkpoint 6 of 10· Fill the gap
Several records in a TruLens evaluation run failed, and you want only the warnings and errors raised during that run. Which table function completes this query?
SELECT *
FROM TABLE(SNOWFLAKE.LOCAL. ? (
'<database_name>',
'<schema_name>',
'<external_agent_name>',
'EXTERNAL AGENT'
))
WHERE record:"severity_text" IN ('ERROR', 'WARN')
AND record_attributes:"snow.ai.observability.run.name" = '<run_name>';Run logs are kept apart from traces. Warnings and errors from a TruLens run come from GET_AI_OBSERVABILITY_LOGS, which takes the same arguments as GET_AI_OBSERVABILITY_EVENTS. GET_AI_OBSERVABILITY_EVENTS returns traces and scores. AI_OBSERVABILITY_EVENTS is the underlying table, not the function you call. CORTEX_ANALYST_REQUESTS_RAW holds only direct Cortex Analyst requests.
Source: docs.snowflake.comCheckpoint 7 of 10· Exam question
Which combination correctly identifies where AI Observability trace and evaluation data are stored, and how an engineer queries them directly in SQL?
Correct answer: A — Data lands in SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS, queried through the GET_AI_OBSERVABILITY_EVENTS table function
- A. AI Observability writes traces and evaluation results into SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS, and the GET_AI_OBSERVABILITY_EVENTS table function is the supported way to query that data with SQL.
- B. The account-level event table set with ALTER ACCOUNT SET EVENT_TABLE is a general logging/tracing destination for Snowflake objects, not the dedicated store queried by SNOWFLAKE.EVENTS for AI Observability results.
- C. Observability data is stored inside Snowflake-managed storage, not exported to a customer external stage that must be reloaded with COPY INTO.
- D. There is no SHOW_TRACES() Streamlit widget, and observability results are queryable with SQL, not limited to a browser cache.
5.Datasets and LLM-as-a-judge evaluation metrics
Evaluation starts with a dataset: a set of test inputs, and optionally the answers you expect (the ground truth). You can supply it as a Snowflake table or as a pandas dataframe. Each column must map to a reserved attribute:
RECORD_ROOT.INPUT: the prompt.RECORD_ROOT.INPUT_ID: an identifier for the prompt, generated automatically if you leave it out.RETRIEVAL.QUERY_TEXT: the user query in a RAG app.RECORD_ROOT.GROUND_TRUTH_OUTPUT: the expected response.
Your instrumented functions supply the output attributes RETRIEVAL.RETRIEVED_CONTEXTS and RECORD_ROOT.OUTPUT.
Scoring uses the LLM-as-a-judge approach. A judge LLM gives each output a score between 0 and 1, along with an explanation of the score. Any LLM available in Cortex AI can be the judge, and llama3.1-70b is used when you don't name one. Each metric checks one specific relationship, so it needs specific attributes. If you leave an attribute unmapped, the metric has nothing to score.
| Metric | What the judge decides | Required attributes |
|---|---|---|
| context_relevance | Is the retrieved context relevant to the query? | RETRIEVAL.QUERY_TEXT, RETRIEVAL.RETRIEVED_CONTEXTS |
| groundedness | Is the response supported by the retrieved context? (uses chain-of-thought reasoning) | RETRIEVAL.RETRIEVED_CONTEXTS, RECORD_ROOT.OUTPUT |
| answer_relevance | Is the response relevant to the query? (no ground truth) | RECORD_ROOT.INPUT, RECORD_ROOT.OUTPUT |
| correctness | How closely does the response align with the ground truth? | RECORD_ROOT.INPUT, RECORD_ROOT.GROUND_TRUTH_OUTPUT, RECORD_ROOT.OUTPUT |
| coherence | Is the response free of logical gaps and contradictions? | RECORD_ROOT.OUTPUT |
Cost and latency come from the traces, not from a judge. Cost is calculated for each Cortex LLM call from the prompt_tokens and completion_tokens that AI_COMPLETE returns, and you can see it per call in the trace. Latency is measured for each instrumented function and then added up for the whole application on each input. Each run also reports an average latency across all inputs.
Snowflake computes the metrics above on the server when you request them by name. Custom metrics run on the client: you wrap any Python function, or any TruLens feedback function, in a Metric object and pass it in the same list as the named metrics. A feedback function such as provider.relevance_with_cot_reasons can be tuned to your domain with criteria, additional_instructions, scored examples, and a custom score range set by min_score_val and max_score_val.
word_count = Metric(
name="word_count",
implementation=lambda text: len(text.split()),
selectors={"text": Selector.select_record_output()},
)Checkpoint 8 of 10· Match them up
Match each server-side evaluation metric to the question its LLM judge answers.
Tap a term, then the definition that fits it.
Each metric checks one relationship. Context relevance compares the query with the retrieved context. Groundedness compares the response with the retrieved context. Answer relevance compares the response with the query and never uses ground truth. Correctness compares the response with GROUND_TRUTH_OUTPUT. Coherence needs only RECORD_ROOT.OUTPUT.
“Groundedness determines if the generated response is supported by and grounded in the retrieved context from the retriever or the search service.”Source: docs.snowflake.com
6.Creating, invoking and comparing runs
A run is one batch evaluation of one application version against one dataset. Building a gen AI app means trying different LLMs, prompts and inference parameters. Each combination you try is a separate version, and each run measures one of them. You create a run with RunConfig and tru_app.add_run(). The dataset_spec maps attribute names to dataset columns, and llm_judge_name picks the judge model.
run_config = RunConfig(
run_name=run_name,
description="desc",
label="custom tag useful for grouping comparable runs",
source_type="DATAFRAME",
dataset_name="My test dataframe name",
dataset_spec={
"RETRIEVAL.QUERY_TEXT": "user_query_field",
"RECORD_ROOT.INPUT": "user_query_field",
"RECORD_ROOT.GROUND_TRUTH_OUTPUT": "golden_answer_field",
},
llm_judge_name="mistral-large2"
)
run = tru_app.add_run(run_config=run_config)A run has four phases, and the first two happen without any scoring. The phases are creation, invocation, computation and visualization:
- Invocation.
run.start()reads each input, calls your app, and stores the traces. Userun.start(input_df=...)for a dataframe source. The call blocks until ingestion completes or times out. - Computation. Scoring is a separate, asynchronous call:
run.compute_metrics([...]). You can call it after invocation reachesINVOCATION_COMPLETEDorINVOCATION_PARTIALLY_COMPLETED.
You can call compute_metrics again later with different metrics, but you can't recompute a metric the run already has.
| Status | Meaning |
|---|---|
| CREATED | Created but not started |
| INVOCATION_IN_PROGRESS | Generating outputs and traces |
| INVOCATION_COMPLETED | All outputs and traces created; metrics not yet computed |
| INVOCATION_PARTIALLY_COMPLETED | Some app invocations or trace generation failed |
| COMPUTATION_IN_PROGRESS | Metric computation running |
| COMPLETED | Metric computation done |
| PARTIALLY_COMPLETED | Some metric computations failed |
| CANCELLED | Run cancelled |
Comparison is the reason runs exist. Run the same dataset against version 1 and version 2. Then, in Snowsight under AI & ML » Evaluations, select the External Agent, select both runs, and choose Compare. You see differences in aggregate scores and record by record, which shows whether a prompt change helped and which version to deploy. Selecting a single record opens its traces, latency, span detail and the judge's explanation. A label groups runs that are meant to be compared, so you can filter for them.
Runs also cover production monitoring and clean-up:
- Live runs. Put
@trace_with_run(app=tru_app, run_name=...)on the function that calls your app, typically a web or API handler. Every production call is then traced into that run. - Clean-up.
run.cancel()stops a run.run.delete()removes only the run's metadata, and its traces stay inAI_OBSERVABILITY_EVENTS.
Checkpoint 9 of 10· Put it in order
Version 1 of a RAG app already has a scored run. You now want to evaluate version 2 on the same dataset and decide which version to deploy. Put the steps for version 2 in order.
- 1.After the status reaches INVOCATION_COMPLETED, call run.compute_metrics() with the metrics to score
- 2.Call run.start() so the app is invoked on each input and its outputs and traces are stored
- 3.Create a run for version 2 with RunConfig and tru_app.add_run(), pointing dataset_spec at the same test dataset
- 4.In Snowsight under AI & ML » Evaluations, select the version 1 and version 2 runs and choose Compare
A run goes through creation, invocation, computation and visualization, in that order. Invocation only produces outputs and traces. Scores exist only after a separate compute_metrics call, so the two versions can be compared only after that call.
“Computation: After invocation, trigger computation by specifying metrics to be computed.”Source: docs.snowflake.com
7.Privileges for registering apps and running evaluations
Evaluation calls a Cortex model, so the role that evaluates must pass the Cortex gates. Access to Cortex functions depends on two things: the account-level USE AI FUNCTIONS privilege and the SNOWFLAKE.CORTEX_USER database role (The two gates: USE AI FUNCTIONS privilege and the SNOWFLAKE.CORTEX_USER database role). AI Observability's judge is AI_COMPLETE, which means the role computing metrics must be able to call AI_COMPLETE. That role needs CORTEX_USER to use AI Observability at all. It also needs USE AI FUNCTIONS, or the per-function USE AI FUNCTION AI_COMPLETE privilege. PUBLIC holds USE AI FUNCTIONS by default. If your account revoked it from PUBLIC, metric computation fails until you grant it explicitly.
Runs need further privileges because they execute as tasks:
USAGEon the External Agent.CREATE TASKon the schema where the app is registered.- The global
EXECUTE TASKprivilege, to run the task that invokes the application.
Registration on its own needs only CREATE EXTERNAL AGENT on the schema.
USE ROLE ACCOUNTADMIN;
CREATE ROLE observability_user_role;
GRANT DATABASE ROLE SNOWFLAKE.CORTEX_USER TO ROLE observability_user_role;
GRANT CREATE EXTERNAL AGENT ON SCHEMA app_schema TO ROLE observability_user_role;
GRANT CREATE TASK ON SCHEMA app_schema TO ROLE observability_user_role;
GRANT EXECUTE TASK ON ACCOUNT TO ROLE observability_user_role;
GRANT USE AI FUNCTIONS ON ACCOUNT TO ROLE observability_user_role;
GRANT ROLE observability_user_role TO USER some_user;Checkpoint 10 of 10· Fill the gap
An administrator is building one role that can register a TruLens app, execute runs as tasks and have the AI_COMPLETE judge compute metrics. Which database role completes the first grant?
USE ROLE ACCOUNTADMIN;
CREATE ROLE observability_user_role;
GRANT DATABASE ROLE SNOWFLAKE. ? TO ROLE observability_user_role;
GRANT CREATE EXTERNAL AGENT ON SCHEMA app_schema TO ROLE observability_user_role;
GRANT CREATE TASK ON SCHEMA app_schema TO ROLE observability_user_role;
GRANT EXECUTE TASK ON ACCOUNT TO ROLE observability_user_role;
GRANT USE AI FUNCTIONS ON ACCOUNT TO ROLE observability_user_role;
GRANT ROLE observability_user_role TO USER some_user;A role needs the SNOWFLAKE.CORTEX_USER database role to use AI Observability at all, because the AI_COMPLETE judge is a Cortex function. AI_OBSERVABILITY_READER only gives read-only access to the event table. AI_OBSERVABILITY_ADMIN only lets a role delete rows. Neither of them lets a role compute metrics.
Source: docs.snowflake.comSources4
Sources
(4)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“AI observability is a collection of features inside Cortex products that help you answer questions about AI workloads in your account”
↩︎ What AI Observability does, and for which apps“Custom AI applications you host through Snowflake products, or even outside of Snowflake, can stream telemetry information into your account with TruLens.”
↩︎ What AI Observability does, and for which apps“Base spend reporting on the usage views rather than on AI_OBSERVABILITY_EVENTS.”
↩︎ What AI Observability does, and for which apps“Cortex Agents deployed through the Agent API or Snowflake CoWork log conversation threads, turns, and execution spans automatically into SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS.”
↩︎ The AI observability event table, logs and access“Built-in functions such as AI_COMPLETE and AI_CLASSIFY do not write traces to the observability event table.”
↩︎ The AI observability event table, logs and access“Snowflake registers each TruLens application as an External Agent object. That object stores metadata only.”
↩︎ Key concept“Trace delivery is best effort, so totals computed from the event table can under-report consumption.”
↩︎ Exam trap 1 - 2.https://docs.snowflake.com/en/user-guide/snowflake-cortex/ai-observability/evaluate-applications-trulensOfficial docs
“Evaluations invoke your app against a dataset, store traces, and compute LLM-as-a-judge metrics (such as context relevance, groundedness, and correctness).”
↩︎ What AI Observability does, and for which apps“RECORD_ROOT: INPUT, OUTPUT, GROUND_TRUTH_OUTPUT”
↩︎ Instrumenting app functions with TruLens span types“Filter on RECORD_ATTRIBUTES (for example run name) to scope results to one evaluation run.”
↩︎ The AI observability event table, logs and access“Client-side metrics can be any TruLens feedback function or any Python function.”
↩︎ Datasets and LLM-as-a-judge evaluation metrics“run.start() is blocking until invocation and ingestion complete or time out.”
↩︎ Creating, invoking and comparing runs“To compute server-side metrics after invocation status is INVOCATION_COMPLETED or INVOCATION_PARTIALLY_COMPLETED:”
↩︎ Creating, invoking and comparing runs“To compare runs that share a dataset, select multiple runs and choose Compare.”
↩︎ Creating, invoking and comparing runs“For online tracing (production monitoring), create a live run with the @trace_with_run decorator on the function that calls your app”
↩︎ Creating, invoking and comparing runs - 3.https://docs.snowflake.com/en/user-guide/snowflake-cortex/ai-observability/trace-applications-trulensOfficial docs
“Install TruLens packages (version 2.1.2 or later): trulens-core, trulens-connectors-snowflake, trulens-providers-cortex.”
↩︎ Instrumenting app functions with TruLens span types“RECORD_ROOT: Main entry point for the application”
↩︎ Instrumenting app functions with TruLens span types“Register the application so TruLens can write traces to AI_OBSERVABILITY_EVENTS and create an External Agent object for governance”
↩︎ Registering the app as an External Agent“main_method (optional): Entry point for tracing (for example, app.answer_query). Not required if the entry point uses RECORD_ROOT instrumentation.”
↩︎ Registering the app as an External Agent“OWNERSHIP is required to modify or drop the External Agent with SQL.”
↩︎ Registering the app as an External Agent“The SNOWFLAKE.AI_OBSERVABILITY_READER application role grants read-only access to the table.”
↩︎ The AI observability event table, logs and access“Without the grant, metadata (tool names, latency, token usage, and similar fields) is still available.”
↩︎ The AI observability event table, logs and access“Snowflake registers each TruLens application as an External Agent object. That object stores metadata only (application name, version, run name).”
↩︎ Exam trap 2“External Agent objects share a namespace with model objects in the same schema.”
↩︎ Exam trap 3“When agent_type is EXTERNAL AGENT, USAGE on the External Agent is sufficient to call the function; MONITOR does not apply.”
↩︎ Exam trap 5“External Agent objects share a namespace with model objects in the same schema.”
↩︎ Checkpoint“Traces and evaluation results for TruLens applications are stored in SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS.”
↩︎ Prediction - 4.
“To register an application, your role must have CREATE EXTERNAL AGENT privileges on the schema.”
↩︎ Registering the app as an External Agent“Each column in the dataset must be mapped to one of the following reserved attributes”
↩︎ Datasets and LLM-as-a-judge evaluation metrics“If no LLM judge is specified, llama3.1-70b is used as the default judge.”
↩︎ Datasets and LLM-as-a-judge evaluation metrics“Cost is calculated for each LLM invocation call that relies on Cortex LLMs based on the token usage information”
↩︎ Datasets and LLM-as-a-judge evaluation metrics“You can trigger multiple runs with the same dataset for different versions.”
↩︎ Creating, invoking and comparing runs“AI Observability uses the AI_COMPLETE function as the LLM judge, so the role that runs the computation must be able to call AI_COMPLETE.”
↩︎ Privileges for registering apps and running evaluations“To use AI Observability, your role must have the CORTEX_USER database role.”
↩︎ Privileges for registering apps and running evaluations“The EXECUTE TASK global privilege to execute the task that runs the application.”
↩︎ Privileges for registering apps and running evaluations“The run invocation completed with all outputs and traces created.”
↩︎ Exam trap 4“Correctness determines how aligned the generated response is with the ground truth.”
↩︎ Prediction“Groundedness determines if the generated response is supported by and grounded in the retrieved context from the retriever or the search service.”
↩︎ Checkpoint“Computation: After invocation, trigger computation by specifying metrics to be computed.”
↩︎ Checkpoint