Designing a Unity Catalog Delta Table for Chunked Text

1.Where the chunk write sits in the pipeline

Databricks describes the data pipeline behind a RAG application as a chain of components. It starts with corpus composition and ingestion. Next comes data preprocessing, which covers parsing, enrichment, metadata extraction, deduplication and filtering. After that come chunking, embedding, and finally indexing and storage. This lesson covers one hand-off in that chain: the point where chunked text stops being an in-memory DataFrame and becomes a governed Delta Lake table in Unity Catalog. Embedding and indexing both read from that table, so its shape, and how you write to it, decide how smoothly the rest of the pipeline runs.

The chunk table is not the first table in the pipeline. Databricks recommends ingesting data in a scalable and incremental way, and it states a best practice for what happens next: raw source data should be ingested and stored in a target table. The guidance says this approach ensures data preservation, traceability, and auditing. In practice that gives you two tables. First a document table holding raw (or parsed) content, then a chunk table derived from it. The pipeline guidance expects you to experiment and tune chunking, indexing and parsing. When you change a chunking parameter, you rebuild the chunk table from the preserved source table instead of going back to the original systems.

Sources1

2.What a chunk row contains

The ai_prep_search SQL function documentation gives a concrete example of a chunk table. It reads files from a Unity Catalog volume, parses them with ai_parse_document, prepares them with ai_prep_search, and then flattens the output into individual chunk rows and writes them to a Delta table. The key operation is the explode (variant_explode): one input document becomes many rows, one for each chunk. The final projection of that query is shown below.

SELECT
  chunk.value:chunk_id::STRING AS chunk_id,
  chunk.value:chunk_position::INT AS chunk_position,
  chunk.value:chunk_to_retrieve::STRING AS chunk_to_retrieve,
  chunk.value:chunk_to_embed::STRING AS chunk_to_embed,
  chunk.value:metadata AS metadata,
  prepped_documents.path AS source_uri
FROM
  prepped_documents,
  LATERAL variant_explode(prepped_documents.result:document.contents) AS chunk;

The same page says what two of these columns are for. The table can be an AI Search index source, with chunk_to_embed as the embedding column and chunk_id as the primary key. The source_uri value repeats on every chunk from the same file, so it links a chunk back to its document but cannot identify a single row. Only chunk_id can do that.

The metadata column matters too. Databricks says that storing metadata alongside chunked documents or their embeddings is essential for optimal performance, because it lets retrieval narrow down results. Combining vector similarity search with keyword-based filtering on that metadata is what the guidance calls a hybrid search pipeline. Chunks produced by other libraries should land in the same shape: a unique key, the text to embed, and source and metadata columns.

Sources21

3.Creating the managed table under a three-level name

Databricks calls Unity Catalog managed tables the default and recommended table type for Delta Lake. Unity Catalog handles reads, writes, storage and optimization for these tables. Managed tables also bring addressing rules that show up on the exam. All reads and writes to a managed table must use its full catalog_name.schema_name.table_name. Path-based access is not supported (except in Compatibility Mode), because it bypasses Unity Catalog access controls and can cause data corruption or loss. The Delta tutorial says the same thing from the other direction: path-based access works only for volumes and external tables. Source files can live in a volume path, but the chunk table itself is always referenced by name.

To create the table you need three privileges: USE CATALOG on the parent catalog, USE SCHEMA on the parent schema, and CREATE TABLE on the parent schema.

-- Create a managed Delta table
CREATE TABLE <catalog-name>.<schema-name>.<table-name>
(
  <column-specification>
);

In Python, you can create the same empty table by writing an empty DataFrame with a schema through saveAsTable(), or you can use the DeltaTableBuilder API. The tutorial points out that, compared with the DataFrame writers, the builder makes it easier to specify column comments, table properties and generated columns. That is handy if you want to document what chunk_to_embed holds, or set a table property up front.

from delta.tables import DeltaTable

(
  DeltaTable. ? (spark)
    .tableName("workspace.default.people_10k_2")
    .addColumn("id", "INT")
    .addColumn("firstName", "STRING")
    .addColumn("lastName", "STRING", comment="surname")
    .addColumn("gender", "STRING")
    .addColumn("age", "INT")
    .execute()
)

Sources34

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.You can write chunks to a Unity Catalog managed table by pointing the writer at its storage path.Why is that wrong?

   Managed tables must be read and written by their catalog.schema.table name. Path-based access is supported only for volumes and external tables.

   Covered in Creating the managed table under a three-level name