Configuring and Measuring the Databricks AI Search Reranker

1.Turning on the reranker in a query

In AI Search, you turn on re-ranking by adding a reranker argument to similarity_search. You don't need to rebuild the index. Two parameters do two different jobs. columns sets which fields are returned to the caller. columns_to_rerank sets which fields the reranker reads when it scores each candidate. The two lists can differ.

results = index.similarity_search(
    query_text="How to create an AI Search index",
    num_results=10,
    columns=["id", "text", "parent_doc_summary"],
    reranker={
        "model": "databricks_reranker",
        "parameters": {
            "columns_to_rerank": ["text", "parent_doc_summary"]
        }
    }
)

Two details about how this behaves come up in exam questions. First, num_results controls only how many results are returned at the end. Databricks says it "does not affect the number of results used for reranking." Second, the order of columns_to_rerank matters. The reranker reads the columns in the order you list them and stops after the first 2000 characters. If a long text column comes first, it can use up the whole budget before the reranker reaches a summary column listed after it.

The SDK also has a typed helper, DatabricksReranker, which you can combine with any query type, including hybrid:

from databricks.ai_search.reranker import DatabricksReranker

results = index.similarity_search(
    query_text = "How to create an AI Search index",
    columns = ["id", "text", "parent_doc_summary", "date"],
    num_results = 10,
    query_type = "hybrid",
    reranker=DatabricksReranker(columns_to_rerank=["text", "parent_doc_summary", "other_column"])
    )

results = index.similarity_search(
    query_text="How to create an AI Search index",
    num_results=10,
    columns=["id", "text", "parent_doc_summary"],
    reranker={
        "model": "databricks_reranker",
        "parameters": {
            " ? ": ["text", "parent_doc_summary"]
        }
    }
)

Sources1

2.Giving the reranker more context with metadata

A chunk read on its own often lacks context. It may not say which manual it came from or which procedure it belongs to. Databricks recommends adding semantic context during data preparation, and one stated reason is that it "Gives rerankers more context for scoring." You can do this in two ways: prepend the summary and section to the chunk text, or store them as separate metadata columns.

Separate columns don't help unless the query uses them. The guidance says to use reranking with columns_to_rerank for semantic metadata such as summaries. For keyword-only metadata, it says to use hybrid (full-text) search. If a column isn't listed in columns_to_rerank, the reranker doesn't read it.

Sources21

3.Confirming the reranker ran and measuring its effect

Set debug_level to at least 1 and the response includes timing for each stage. The example below shows the trade-off from the previous section in real numbers: ANN retrieval took 29 ms and reranking took 619 ms.

'debug_info': {'response_time': 693.0, 'ann_time': 29.0, 'reranker_time': 619.0}

If the reranker call fails, the query still returns results. They are the first-stage results in their original order, and the debug info includes a RERANKER_TEMPORARILY_UNAVAILABLE warning saying "Results returned have not been processed by the reranker." Since the query doesn't raise an error, you have to check the debug info to know whether re-ranking actually happened.

To decide whether re-ranking pays off on your own data, use AI Search's built-in retrieval-quality evaluation (Beta). It generates queries from your documents, runs ANN, hybrid and full-text retrieval, and evaluates each strategy with and without the reranker on the same set of queries. An LLM judge then scores how relevant the results are. The results dashboard shows a bar chart comparing DCG@10 for each query type with and without the reranker, plus a table comparing base and reranker performance.

Sources13

4.Finetuning a reranker on your own corpus

The built-in reranker is a general-purpose model. Reranker finetuning (Beta) "trains a custom reranker on your own index data," and Databricks handles training data generation, training, registration and deployment. It supports managed Delta Sync indexes only. Training data is built by sampling queries, from payload logs, a table you provide, or synthesized from your data. Databricks then retrieves candidates for each query and has an LLM judge their relevance. A base reranker model is trained on that data, registered in Unity Catalog, and deployed to Model Serving.

Querying with the finetuned reranker has a stricter column rule than the built-in one. columns_to_rerank must be exactly the index's embedding source column, given as a single-element list. In the REST API you also have to pass the Model Serving endpoint name with model_type set to MODEL_TYPE_FINETUNED, because REST doesn't work out the endpoint for you.

Sources4

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Raising num_results makes the reranker score more candidates.Why is that wrong?

   num_results only sets how many final results are returned. It doesn't change how many results the reranker scores.

   Covered in Turning on the reranker in a query

2. 2.The reranker reads every column in columns_to_rerank in full, so order doesn't matter.Why is that wrong?

   The reranker reads the columns in the order listed and stops after the first 2000 characters, so a long column listed first can push out the ones after it.

   Covered in Turning on the reranker in a query

3. 3.A finetuned reranker accepts the same multi-column columns_to_rerank list as the built-in reranker, including metadata columns.Why is that wrong?

   For a finetuned reranker, columns_to_rerank must be exactly the embedding source column, given as a single-element list.

   Covered in Finetuning a reranker on your own corpus

Practise it for real

Measure what the built-in reranker changes, in both order and latency, on one of your AI Search indexes

1. 1.Run index.similarity_search with a query_text, num_results=10, the columns you want returned, and debug_level=1, without a reranker argument.

   Why: This gives you the first-stage order and timing as a baseline.

   You should see: Ten results, and debug_info showing response_time and ann_time but no reranker_time.

2. 2.Run the same query again with reranker={"model": "databricks_reranker", "parameters": {"columns_to_rerank": ["text"]}}.

   Why: Changing only the reranker means any difference in order comes from re-ranking.

   You should see: The same number of results, possibly in a different order, and debug_info now including reranker_time.

3. 3.Add a summary or section metadata column after text in columns_to_rerank and run the query again.

   Why: Metadata columns give the reranker more context, but only if they are listed and fit within its 2000-character budget.

   You should see: The order may change again. If the text column is long, the extra column may have little effect.

4. 4.Inspect debug_info for a warnings entry on each reranked run.

   Why: If the reranker is unavailable, the query returns un-reranked results without raising an error.

   You should see: No warnings if re-ranking succeeded. A RERANKER_TEMPORARILY_UNAVAILABLE warning if it fell back.