What you will be able to do
- Choose between createOrReplace, saveAsTable and overwrite mode for a first or full load of chunks
- Choose between append mode and MERGE when documents are re-processed
- Prepare a chunk table for a Delta Sync AI Search index by meeting the change data feed requirement
1.First load: create, replace, or overwrite
Once chunking produces a DataFrame of chunk rows (a unique key, the text to embed, and source and metadata columns), the first job is to land it in a Unity Catalog managed table, referenced by its full catalog.schema.table name. No format option is needed: Delta Lake is the default for every table creation, read and write command in Databricks. A plain Spark write therefore produces a Delta table.
# Create the table if it does not exist. Otherwise, replace the existing table.
df.writeTo("workspace.default.people_10k").createOrReplace()
# If you know the table does not already exist, you can use this command instead.
# df.write.saveAsTable("workspace.default.people_10k")The comments in that sample are the distinction to remember. writeTo(...).createOrReplace() creates the table if it is missing and replaces it otherwise, so a notebook that uses it can be rerun. df.write.saveAsTable(...) is the option for when you know the table does not already exist. When a chunking experiment regenerates the entire corpus, use mode("overwrite"), which replaces all the data in the table. Every one of these writes rewrites the table. None of them reconciles new chunks against old ones row by row; that job belongs to the operations in the next section.
| Operation | Python form | Effect on existing rows |
|---|---|---|
| Create or replace | df.writeTo(...).createOrReplace() | Creates the table if absent, otherwise replaces it |
| Save as new table | df.write.saveAsTable(...) | For a table you know does not already exist |
| Overwrite | df.write.mode("overwrite").saveAsTable(...) | Replaces all data in the table |
| Append | df.write.mode("append").saveAsTable(...) | Adds rows; does not check for duplicate records |
| Upsert | DeltaTable.merge(...) | Updates matching rows, inserts non-matching rows |
Checkpoint 1 of 6· Match them up
Match each write operation to its behaviour on a chunk table that already holds data.
Tap a term, then the definition that fits it.
Of these four, only MERGE compares incoming rows with existing ones. Append never checks for duplicates, and overwrite and createOrReplace both replace the table's contents.
“Unlike upserting, writing to a table does not check for duplicate records.”Source: docs.databricks.com
Sources1
2.Re-ingesting documents: append versus MERGE
After the first load, new documents and edited documents keep arriving. To add new data to an existing Delta table, the standard operation is append mode.
Checkpoint 2 of 6· Fill the gap
Which write mode adds new chunk rows to an existing table without replacing what is already there?
# Append the new data to the target table.
df.write.mode(" ? ").saveAsTable("workspace.default.people_10k")Append mode adds the new rows to the existing table. Overwrite would replace every row already in it.
Source: docs.databricks.comAppend has a catch: unlike upserting, it does not check for duplicate records. If a document is parsed and chunked a second time and its chunks are appended, the table holds two copies of the same chunks. The data pipeline guidance warns that duplicates can leave you with highly redundant chunks in your final index, which can hurt application performance.
The fix is an upsert. Delta Lake's DeltaTable.merge in Python (MERGE INTO in SQL) updates rows that match a key and inserts rows that do not. For a chunk table, the merge condition is the chunk key. That only works if re-processing a document produces the same key for the same chunk.
(deltaTable.alias("people_10k")
.merge(
people_10k_updates.alias("people_10k_updates"),
"people_10k.id = people_10k_updates.id")
.whenMatchedUpdateAll()
.whenNotMatchedInsertAll()
.execute()
)Two MERGE rules matter for chunks. First, only one source row can match a given target row. If the incoming batch contains the same chunk_id twice, the merge cannot apply it, so make keys unique before merging. Second, an edited document may produce fewer chunks than before, which leaves its old trailing chunks behind. In Databricks SQL and Databricks Runtime 12.2 LTS and above, the WHEN NOT MATCHED BY SOURCE clause can delete target rows that have no corresponding source row. If your batch holds only some documents, an unconditional delete would also remove every other document's chunks. Databricks recommends adding a conditional clause to avoid fully rewriting the target table.
Checkpoint 3 of 6· Check yourself
A nightly job re-parses documents edited that day and writes their chunks to the existing chunk table. Retrieval starts returning the same passage twice. Which change fixes the write?
Append adds the re-chunked rows next to the existing ones because it never checks for duplicates. A MERGE on the chunk key updates chunks that already exist instead.
“Unlike upserting, writing to a table does not check for duplicate records.”Source: docs.databricks.com
Checkpoint 4 of 6· Exam question
A team created an AI Search endpoint and attempted to create a Delta Sync Index pointing at `catalog.schema.chunks`, but index creation fails with an error stating the source table cannot be synced incrementally. What is the most likely cause and correct fix?
Correct answer: A — The Delta table was created without Change Data Feed enabled; run `ALTER TABLE catalog.schema.chunks SET TBLPROPERTIES (delta.enableChangeDataFeed = true)` and then create the index.
- A. A Delta Sync Index relies on Change Data Feed to identify which rows were added, updated, or deleted since the last sync; without `delta.enableChangeDataFeed = true` set on the source table, incremental sync cannot function and index creation fails with exactly this kind of error.
- B. Z-ordering is a file-layout optimization for query performance and has no bearing on whether a Delta Sync Index can track incremental changes, so it would not resolve a sync-capability error.
- C. The write pattern used to populate the table (Auto Loader streaming versus batch inserts) is irrelevant to Delta Sync Index compatibility; what matters is whether Change Data Feed is enabled on the resulting Delta table, not how rows were written.
- D. Delta Sync Index sync operations are managed by the vector search service reading the Delta table's change feed directly, not by polling through a SQL warehouse, so attaching a warehouse would not address an incremental-sync failure.
3.Preparing the chunk table for an AI Search index
The last step in the sequence is making the chunk table usable as an index source. Databricks AI Search was formerly known as Databricks Vector Search. Its indexes run similarity searches over a source table, and supported sources include Delta Lake tables. There is one write-side requirement: for standard endpoints, the source table must use a change data feed (CDF). CDF tracks row-level changes between table versions. Each change record carries the row plus metadata saying whether it was inserted, updated or deleted.
No. Tables with row tracking enabled use a change data feed automatically, without manual configuration. Automatic CDF needs Databricks Runtime 19 or above and a Unity Catalog managed Delta table with row tracking. Without that, you turn on legacy CDF by setting a table property, and you cannot use legacy and automatic CDF together.
ALTER TABLE myDeltaTable
SET TBLPROPERTIES (delta.enableChangeDataFeed = true)CDF adds metadata columns named _change_type, _commit_version and _commit_timestamp. If your chunk schema already has columns with those names, you cannot use CDF on the table, so keep them out of your schema design. With CDF in place, the index is created against the table by name. In the call below, source_table_name is the three-level table name, primary_key is the unique key column, and embedding_source_column is the text Databricks embeds. For a chunk table, those map to chunk_id and chunk_to_embed. Creating the index also requires CREATE TABLE privileges on the catalog schema where the index will live.
client = AISearchClient()
index = client.create_delta_sync_index(
endpoint_name="vector_search_demo_endpoint",
source_table_name="vector_search_demo.vector_search.en_wiki",
index_name="vector_search_demo.vector_search.en_wiki_index",
pipeline_type="TRIGGERED",
primary_key="id",
embedding_source_column="text",Checkpoint 5 of 6· Put it in order
Put the steps for getting chunked text into a standard-endpoint AI Search index in order.
- 1.Make sure the table uses a change data feed
- 2.Create the managed chunk table under catalog.schema.table
- 3.Create the Delta Sync index on the table with its primary key and embedding source column
- 4.Write the chunk rows with append or MERGE
The index reads from an existing table, and a standard endpoint requires that table to already use a change data feed. Table creation and the chunk writes therefore come before index creation.
“For standard endpoints, the source table must use a change data feed.”Source: docs.databricks.com
Checkpoint 6 of 6· Exam question
An engineer has parsed raw PDF documents into plain text and needs to end up with a queryable AI Search index built on chunks stored in Unity Catalog. Order the following operations into the correct sequence.(Select 4)
Correct answers: A, B, C, D — Split the parsed document text into chunks using a chosen chunking strategy (e.g., recursive character splitting) and attach a unique identifier and source metadata to each chunk.; Write the chunk records to a managed Delta table in Unity Catalog using a three-level `catalog.schema.table` name, including a primary-key column.; Enable Change Data Feed on the Delta table by setting `delta.enableChangeDataFeed = true` in the table properties.; Create an AI Search endpoint and a Delta Sync Index that references the Delta table as its source, specifying the primary key and embedding source column.
- A. Chunking must happen first, since the raw parsed text has to be segmented into retrieval-sized units with identifiers and metadata before anything can be written to a table.
- B. Once chunks exist as structured records, they are written into a Unity Catalog managed Delta table so they persist and are governed under the catalog's access controls before any indexing occurs.
- C. Change Data Feed must be enabled on the Delta table before a Delta Sync Index is created against it, since the index relies on the change feed to perform incremental syncs from that point forward.
- D. Only after the table exists with Change Data Feed enabled can the endpoint and Delta Sync Index be created, since the index creation step requires a valid, sync-ready source table to reference.
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Append mode skips chunks that are already in the table, so re-running a chunking job with append is safe.Why is that wrong?
Append never checks for duplicates. Re-processed documents need an upsert (MERGE) on the chunk key to avoid duplicate chunks.
Covered in Re-ingesting documents: append versus MERGE
2.Any Delta table of chunks can back a standard-endpoint AI Search index as it is.Why is that wrong?
For a standard endpoint, the source table must use a change data feed, either automatically through row tracking or through the legacy table property.
3.Naming a chunk-table column _change_type to record your own edit history is harmless.Why is that wrong?
Change data feed reserves _change_type, _commit_version and _commit_timestamp. A schema that uses those names blocks CDF, and therefore blocks a standard-endpoint index.
Practise it for real
Write a small set of chunks to a Unity Catalog managed Delta table, re-ingest them without creating duplicates, and prepare the table for a Delta Sync AI Search index.
1.Run CREATE TABLE <catalog-name>.<schema-name>.<table-name> with columns chunk_id STRING, chunk_to_embed STRING and source_uri STRING.
Why: Managed tables are addressed by a three-level name, and chunk_id gives each chunk row a unique key.
You should see: The empty table appears under your schema in Catalog Explorer.
2.Build a DataFrame of a few chunks and write it with df.write.mode("append").saveAsTable("<catalog-name>.<schema-name>.<table-name>").
Why: Append mode adds new rows to an existing Delta table.
You should see: The table's row count equals the number of chunks you wrote.
3.Run the same append again, count rows, then restore the table and instead MERGE the batch on chunk_id using whenMatchedUpdateAll() and whenNotMatchedInsertAll().
Why: This shows that append does not check for duplicates while an upsert does.
You should see: A second append doubles the row count. The MERGE leaves the count unchanged for chunks that already exist.
4.If the table does not have row tracking, run ALTER TABLE <table> SET TBLPROPERTIES (delta.enableChangeDataFeed = true), then query SELECT * FROM table_changes('<table_name>', 0).
Why: A standard-endpoint index requires its source table to use a change data feed.
You should see: table_changes returns rows with _change_type, _commit_version and _commit_timestamp columns.
5.Call create_delta_sync_index with source_table_name set to the chunk table, primary_key="chunk_id" and embedding_source_column="chunk_to_embed".
Why: The index syncs from the chunk table using its unique key and text column.
You should see: The index is created on your endpoint and lists the chunk table as its source.
Stuck? Get a nudge
If index creation is rejected, check the source table's change data feed and your CREATE TABLE privilege on the index's schema before you look anywhere else.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.https://docs.databricks.com/aws/en/delta/tutorialOfficial docs
“Delta Lake is the default for all table creation, read, and write commands in Databricks.”
↩︎ First load: create, replace, or overwrite“To replace all the data in a table, use the overwrite mode.”
↩︎ First load: create, replace, or overwrite“To add new data to an existing Delta Lake table, use the append mode.”
↩︎ Re-ingesting documents: append versus MERGE“Unlike upserting, writing to a table does not check for duplicate records.”
↩︎ Exam trap 1“Unlike upserting, writing to a table does not check for duplicate records.”
↩︎ Checkpoint“Unlike upserting, writing to a table does not check for duplicate records.”
↩︎ Checkpoint - 2.https://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/quality-data-pipeline-ragOfficial docs
“you can end up with highly redundant chunks in your final index that can decrease the performance of your application.”
↩︎ Re-ingesting documents: append versus MERGE - 3.https://docs.databricks.com/aws/en/delta/mergeOfficial docs
“Only a single row from the source table can match a given row in the target table.”
↩︎ Re-ingesting documents: append versus MERGE“Databricks recommends adding an optional conditional clause to avoid fully rewriting the target table.”
↩︎ Re-ingesting documents: append versus MERGE - 4.
“For standard endpoints, the source table must use a change data feed.”
↩︎ Preparing the chunk table for an AI Search index“Tables with row tracking enabled use a change data feed automatically, without requiring manual configuration.”
↩︎ Preparing the chunk table for an AI Search index“For standard endpoints, the source table must use a change data feed.”
↩︎ Exam trap 2 - 5.
“Set the table property delta.enableChangeDataFeed = true in the ALTER TABLE command.”
↩︎ Preparing the chunk table for an AI Search index“If the schema contains columns with the same names as these metadata columns, you can't use change data feed on a table.”
↩︎ Exam trap 3