What you will be able to do
- Describe what Unity Catalog lineage captures, when it is captured, and what it is used for
- Navigate the Lineage tab and lineage graph down to column level, and predict what a user with partial privileges sees
- Add external lineage by creating an external metadata object and a lineage relationship
1.What Unity Catalog lineage records
Data lineage shows where data in Databricks came from and where it goes. For a table, that means the queries and files that populate it, the jobs and notebooks that transform it, and the dashboards that read from it. You don't have to set anything up. Unity Catalog records lineage automatically for queries run on Databricks, down to individual columns. It combines lineage from every workspace attached to the same metastore.
Lineage answers practical questions:
- Impact analysis: before you drop or rename a column, find the downstream tables, jobs and dashboards that depend on it.
- Root cause: when a report looks wrong, trace upstream to where the data diverged.
- Compliance: show where regulated data originates, how it is transformed and who consumes it.
- Ownership: find which teams own your upstream sources or consume your tables.
Lineage covers more than tables. It also includes views, columns, ML model versions, file paths, UDFs, model services, and external assets you register yourself. Some conditions apply for anything to be captured. The tables must be registered in a Unity Catalog metastore. Queries must go through the Spark DataFrame API or a Databricks SQL interface such as a notebook or the SQL query editor. External assets must be registered as external metadata objects. When a temporary table sits between two permanent tables, lineage links the permanent tables directly.
Checkpoint 1 of 3· Check yourself
An external ETL tool copies rows from MySQL into a Unity Catalog table. A Databricks notebook then transforms that table into a second Unity Catalog table, and a Tableau report reads the second table. Nobody has set anything else up. What does the lineage graph show?
Unity Catalog records lineage automatically, down to the column level, for queries run on Databricks, so the notebook's table-to-table flow shows up without any setup. MySQL and Tableau sit outside Databricks, so they appear only after you register them as external metadata objects.
“External assets (those not registered in the Unity Catalog metastore) must be added as external metadata objects in Unity Catalog”Source: docs.databricks.com
Sources1
2.Reading the Lineage tab and lineage graph
In Catalog Explorer, open a table and select its Lineage tab. The panel lists related tables, and you can switch to the notebooks, jobs, pipelines, queries and dashboards connected to the table. For example, select Jobs and then Downstream to see which jobs read the table. Click Dashboards to see which dashboards use it.
For a picture of the connections, click See Lineage Graph. The graph shows one level of connections at first. Click the icon on a node to expand it. Click the icon on an edge between two nodes to open the Lineage details panel, which shows the source and target and the notebooks, jobs, pipelines or queries behind the connection. Click a column to highlight the upstream columns it was derived from, such as the inputs behind a revenue column.
Lineage follows the same permission model as the rest of Unity Catalog. To view lineage at all, you need at least BROWSE on the table's parent catalog, and that catalog must be reachable from your workspace. Nodes for objects you have no rights on stay closed to you. Notebooks, jobs and dashboards follow their own workspace access controls, and pipelines need CAN VIEW. Lineage recorded in one workspace shows up in another workspace on the same metastore, but details of notebooks and dashboards from other workspaces are masked.
Checkpoint 2 of 3· Match them up
You are tracing a table's lineage in Catalog Explorer. Match each action to what it shows you.
Tap a term, then the definition that fits it.
The Lineage tab lists related assets, and you can filter it by type and direction. In the graph, a node icon expands that node's connections, an edge icon opens the Lineage details panel for that connection, and clicking a column traces where that column came from.
“Click the icon on a connecting edge in the lineage graph to open the Lineage details panel.”Source: docs.databricks.com
Sources1
3.Adding external lineage for systems outside Databricks
Automatic capture only sees work that runs on Databricks. If data starts in Salesforce or MySQL before ingestion, or ends up in a Tableau or Power BI report, those steps are missing from the graph until you record them. External lineage fills in those first and last steps. The external systems then appear as nodes in the same graph as the built-in lineage.
Recording external lineage takes two steps. First, create an external metadata object, which is a Unity Catalog securable representing one thing in the outside system, such as a Tableau dashboard. Second, create a lineage relationship that links it upstream or downstream of a table, model, path or another external metadata object. The object appears in the lineage graph only after the relationship exists.
You can start in Catalog Explorer in two ways. From an existing lineage graph, click Create external lineage in its upper-right corner. Or go to External data → External Metadata tab → Create external metadata. Fill in a name that follows Unity Catalog naming rules, a system type (choose Custom if yours isn't listed) and an entity type such as table or dashboard. Optionally, add a URL so people viewing the graph can click through, and column names for column-level mapping. Then open the object, click Create lineage relationship, choose upstream or downstream and pick the target. The relationship then appears in the Lineage tab of the related objects. The same object can also be created with the Databricks SDK for Python:
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.catalog import ExternalMetadata, SystemType
w = WorkspaceClient()
external_metadata = w.external_metadata.create_external_metadata(
external_metadata=ExternalMetadata(
name="salesforce_accounts",
system_type=SystemType.SALESFORCE,
entity_type="table",
url="https://my-org.salesforce.com/lightning/o/Account/list",
description="Salesforce Account records ingested into Databricks.",
)
)| Task | Required privilege |
|---|---|
| Create an external metadata object | CREATE EXTERNAL METADATA on the metastore |
| Define any lineage relationship for that object | MODIFY on the external metadata object |
| Make a Unity Catalog object downstream of it | Read access on that object, for example SELECT on a table |
| Make a Unity Catalog object upstream of it | Write access on that object, for example MODIFY on a table |
You don't always have to record external lineage by hand. Lakeflow Connect managed ingestion pipelines, one of the approaches for bringing data into Databricks, record lineage from the source tables to the destination tables in Unity Catalog automatically. External metadata objects can also be tagged like other securables. That feature is in Public Preview.
Checkpoint 3 of 3· Check yourself
Your team wants a lineage graph that runs from outside sources to outside consumers. Which of these links shows up even though nobody has created a lineage relationship by hand?
Lakeflow Connect managed ingestion pipelines record lineage from source to destination tables automatically. Other external systems appear only after you create an external metadata object and a lineage relationship. An object without a relationship does not appear in the graph.
“Automatically, using Lakeflow Connect managed ingestion pipelines, which record source lineage from your source tables to the destination tables in Unity Catalog.”Source: docs.databricks.com
Sources
(3)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Unity Catalog captures lineage automatically for queries run on Databricks, down to the column level”
↩︎ What Unity Catalog lineage records“Tables must be registered in a Unity Catalog metastore.”
↩︎ What Unity Catalog lineage records“By default, one level is displayed in the graph.”
↩︎ Reading the Lineage tab and lineage graph“You must have at least the BROWSE privilege on the parent catalog of the table or view.”
↩︎ Reading the Lineage tab and lineage graph“Detailed information about workspace-level objects like notebooks and dashboards in other workspaces is masked.”
↩︎ Reading the Lineage tab and lineage graph“To view column-level lineage, click a column in the graph to show links to related columns.”
↩︎ Reading the Lineage tab and lineage graph“If a user does not have the BROWSE or SELECT privilege on a table, they cannot explore its lineage.”
↩︎ Exam trap 2“External assets (those not registered in the Unity Catalog metastore) must be added as external metadata objects in Unity Catalog”
↩︎ Checkpoint“If a user does not have the BROWSE or SELECT privilege on a table, they cannot explore its lineage.”
↩︎ Prediction“Click the icon on a connecting edge in the lineage graph to open the Lineage details panel.”
↩︎ Checkpoint - 2.
“Unity Catalog lets you add external lineage metadata to augment the Databricks data lineage it captures automatically”
↩︎ Adding external lineage for systems outside Databricks“To start from an existing lineage graph, click Create external lineage in the upper-right corner of the graph.”
↩︎ Adding external lineage for systems outside Databricks“you must have the CREATE EXTERNAL METADATA privilege on the metastore.”
↩︎ Adding external lineage for systems outside Databricks“You can now see the external lineage relationship in the Lineage tab of the related objects.”
↩︎ Adding external lineage for systems outside Databricks“When you have created lineage relationships, the external metadata object appears in the lineage graph view.”
↩︎ Exam trap 1“Automatically, using Lakeflow Connect managed ingestion pipelines, which record source lineage from your source tables to the destination tables in Unity Catalog.”
↩︎ Checkpoint - 3.
“Tagging on external metadata objects is in Public Preview.”
↩︎ Adding external lineage for systems outside Databricks