What you will be able to do
- Name the two halves of a Databricks RAG application and say what each one does
- Lay out the data pipeline stages, from ingesting a document corpus into a table through to indexing
- Pick between a Delta Sync Index and a Direct Vector Access Index, and list the Unity Catalog prerequisites for creating one
- Explain why the same embedding model has to be used for chunks and for queries
Key concept
Delta Sync Index — A Delta Sync Index is an AI Search index tied to a governed source table. When rows in that table change, the index updates incrementally, so the agent retrieves from the same corpus Unity Catalog governs and nobody has to push vectors in by hand.
1.A RAG application has two halves
Retrieval-augmented generation (RAG) runs in three steps. It retrieves supporting data from a knowledge base, combines that data with the user's request into a prompt, and passes the prompt to an LLM. Databricks separates the work into a data pipeline, which pre-processes and indexes documents ahead of time, and a RAG chain, which runs at request time to retrieve, augment and generate. Evaluation and monitoring sit on top, along with governance that tracks lineage and access controls for every component. This page covers the data pipeline: how a document corpus becomes an index an agent can query. Databricks AI Search is the indexing product here. It was formerly called Databricks Vector Search, and many SDK class names and API paths still use the old name.
Governance belongs in the design from the start. The RAG overview lists access control as one of RAG's benefits: the retrieval step can be built to return personal or proprietary information only to users whose credentials allow it. If the corpus and index both live in Unity Catalog, the permissions on those objects determine what the retriever can surface.
Checkpoint 1 of 5· Check yourself
According to Databricks, which stage of a RAG application pre-processes and indexes documents for fast and accurate retrieval?
Preparing and indexing happen in the offline data pipeline. The RAG chain runs at request time and queries what the pipeline built.
“Data pipeline: Pre-process and index documents, tables, or other data for fast and accurate retrieval.”Source: docs.databricks.com
Sources1
2.Ingest the corpus into a table, then clean it
The pipeline begins by choosing the corpus. Which documents belong in it depends on the use case, so domain experts should help curate it. Databricks recommends ingesting scalably and incrementally, and storing the raw source data in a target table before any transformation. Keeping the raw data in a table preserves it, makes it traceable, and leaves an audit trail. Inside Unity Catalog, that table is also the governed object every later stage reads from.
Preprocessing comes next. Parse the raw files into clean text, enrich it with metadata, remove duplicates, and filter out documents the agent should never see. Filtering is a governance step as well as a quality step: some documents are irrelevant or out of date, and others contain sensitive information you do not want an agent to expose, such as PII that a detection algorithm can flag. Store the metadata you extract alongside the chunks or their embeddings, because the index can then use it to narrow retrieval later.
If a document is never indexed, no retrieval call can return it, whoever is asking. A chunk that is in the index can be retrieved and injected into the prompt, and from that point the model has seen it. The documentation recommends a filtering step, for example running PII detection, to remove such documents before they reach the agent.
Checkpoint 2 of 5· Check yourself
Why does Databricks recommend storing ingested raw source data in a target table as a best practice?
The documentation justifies the target table on preservation, traceability and auditing grounds, which is what a governed corpus needs. Parsing still happens afterwards.
“This approach ensures data preservation, traceability, and auditing.”Source: docs.databricks.com
Sources2
3.Chunk, then embed with a model you will reuse at query time
Once the documents are clean, split them into chunks. Chunking exists because of the context window: small, focused chunks fit in the LLM's context and carry less irrelevant text than whole documents do. How to choose chunk sizes and strategies is a topic of its own. For pipeline design, the point is that chunking is a separate stage between preprocessing and embedding.
An embedding model then turns each chunk into a vector that captures its meaning. This choice ties the pipeline to the RAG chain: at query time the retrieval query has to be embedded with the same model that embedded the chunks, otherwise the similarity comparison does not work. Check the model's maximum token limit too. A chunk longer than the limit is truncated and can lose important information. For example, bge-large-en-v1.5 has a 512-token limit.
Checkpoint 3 of 5· Put it in order
Put these data pipeline stages in the order Databricks describes them
- 1.Indexing and storage
- 2.Corpus composition and ingestion
- 3.Data preprocessing (parsing, enrichment, deduplication, filtering)
- 4.Embedding
- 5.Chunking
The pipeline ingests, cleans, chunks, embeds and then indexes. Each stage takes the previous stage's output as its input.
“Indexing and storage: Create efficient vector indices for optimized search performance.”Source: docs.databricks.com
Sources2
4.Create the AI Search index on the governed table
AI Search indexes run real-time similarity search over a source table, and AI Search endpoints serve them. Supported sources are Delta Lake tables, streaming tables, and managed tables using Iceberg v3 or above. The prerequisites tie the index to Unity Catalog: the workspace must be Unity Catalog enabled, you need CREATE TABLE on the schema where the index will live, and the index name uses a three-level <catalog>.<schema>.<name> namespace. On standard endpoints, the source table must also have change data feed enabled. Tables with row tracking get one automatically.
| Aspect | Delta Sync Index | Direct Vector Access Index |
|---|---|---|
| How data gets in | Syncs automatically from a source table, updating incrementally | You read and write vectors and metadata directly |
| Who keeps it current | The sync (Continuous or Triggered) | You, through the REST API or Python SDK |
| Create in the UI? | Yes (Catalog Explorer > Create > Vector search index) | No, REST API or SDK only |
When you create a Delta Sync Index, decide where the embeddings come from. With Compute embeddings, you pick a text column and Databricks embeds it, using a Databricks-managed model by default. With Use existing embeddings, you point the index at an array[float] column of precomputed vectors. For freshness, Continuous sync keeps the index within seconds of the table but costs more. Triggered sync runs when you start it from the SDK or REST API, and it is the only mode storage-optimized endpoints support. Finally, *Columns to index* controls what the retriever can see: only indexed columns can be returned in results or used as filters. If you leave it blank, every column is indexed.
client.create_endpoint(
name="vector_search_endpoint_name",
endpoint_type="STANDARD" # or "STORAGE_OPTIMIZED"
)Checkpoint 4 of 5· Check yourself
A team creates a Delta Sync Index and indexes only chunk_id, chunk_text, and the embedding column. Later they want the agent to filter results by a department column in the source table. What happens?
A column left out of the index cannot be filtered on or returned. Add department to the indexed columns.
“Only indexed columns can be returned in search results or used as filters.”Source: docs.databricks.com
Checkpoint 5 of 5· Exam question
A context engineer is designing a RAG pipeline that must retrieve chunks from a Unity Catalog-governed document corpus and inject them into an agent's context. The source PDF files already exist in a Unity Catalog volume and have been parsed into plain text. What is the correct order of the remaining pipeline stages?
Correct answer: A — Store the parsed text in a Delta table, chunk the text, build an AI Search index from the chunked table, then retrieve and inject chunks into the agent's context
- A. Correct. The standard RAG pipeline flow on Databricks is source ingestion into a Delta table, chunking the stored text, building an AI Search index from the chunked table, and then retrieving and injecting matching chunks into the agent's context at query time.
- B. Incorrect. AI Search indexes are built from Delta tables, not directly from a UC volume, and chunking must happen before indexing so each indexed row corresponds to a retrievable chunk rather than being computed after the fact.
- C. Incorrect. Chunking must operate on parsed text, not raw PDF bytes, and embeddings for retrieval are produced by an embedding model tied to the index, not by the LLM used for response generation.
- D. Incorrect. Retrieval must occur before generation so the retrieved chunks can ground the response; indexing the agent's own output after the fact does not build a corpus the agent can query for answers.
Sources3
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Any index type, including a Direct Vector Access Index, can be created from Catalog Explorer in the UI.Why is that wrong?
Only Delta Sync Indexes can be created in the UI. A Direct Vector Access Index has to be created and kept up to date through the REST API or the Python SDK.
2.You can embed the chunks with one model and embed user queries with another, cheaper model.Why is that wrong?
The retrieval query has to be embedded with the same model that embedded the chunks, or the semantic similarity comparison is meaningless.
Covered in Chunk, then embed with a model you will reuse at query time
3.Any source-table column can be returned or filtered at query time, whatever the index configuration.Why is that wrong?
Only the columns selected under Columns to index can be returned or used as filters. The primary key and embedding columns are always included.
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“The retrieval step can be designed to selectively retrieve personal or proprietary information based on user credentials.”
↩︎ A RAG application has two halves“Databricks AI Search was formerly known as Databricks Vector Search.”
↩︎ A RAG application has two halves“Data pipeline: Pre-process and index documents, tables, or other data for fast and accurate retrieval.”
↩︎ Checkpoint - 2.https://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/quality-data-pipeline-ragOfficial docs
“As a best practice, raw source data should be ingested and stored in a target table.”
↩︎ Ingest the corpus into a table, then clean it“other documents may contain sensitive information you do not want to expose through your agent.”
↩︎ Ingest the corpus into a table, then clean it“Storing metadata alongside chunked documents or their corresponding embeddings is essential for optimal performance.”
↩︎ Ingest the corpus into a table, then clean it“ensures that retrieved data fits in the LLM's context while minimizing the inclusion of distracting or irrelevant information”
↩︎ Chunk, then embed with a model you will reuse at query time“If you pass chunks that exceed this limit, they will be truncated, potentially losing important information.”
↩︎ Chunk, then embed with a model you will reuse at query time“transformed at query time using the same embedding model used to embed chunks in the data pipeline”
↩︎ Exam trap 2“This approach ensures data preservation, traceability, and auditing.”
↩︎ Checkpoint“Indexing and storage: Create efficient vector indices for optimized search performance.”
↩︎ Checkpoint - 3.
“you must have CREATE TABLE privileges on the catalog schema where the index will be created”
↩︎ Create the AI Search index on the governed table“For standard endpoints, the source table must use a change data feed.”
↩︎ Create the AI Search index on the governed table“Continuous keeps the index in sync with seconds of latency.”
↩︎ Create the AI Search index on the governed table“automatically syncs with a source table, incrementally updating the index as the underlying data in the source table changes”
↩︎ Key concept“This type of index cannot be created using the UI.”
↩︎ Exam trap 1“Only indexed columns can be returned in search results or used as filters.”
↩︎ Exam trap 3“Only indexed columns can be returned in search results or used as filters.”
↩︎ Checkpoint