What you will be able to do
- Explain why every prompt revision costs tokens on every request and adds upkeep for later editors
- Apply the revision ladder (reword, add a constraint, add an example, restructure) to a failure pattern and stop at the first rung that resolves it
- Pick the Genie knowledge-store edit (table description, column description and synonyms, entity matching) that fixes a failure more cheaply than a general text instruction
- Confirm a revision on the original evaluation set and keep it reversible as a registered prompt version
Key concept
Prompt revision ladder — Name the observed failure pattern first. Then try the cheapest edit aimed at it: reword an existing instruction, add one constraint, add one example, and restructure only as a last resort. Stop at the first rung that makes the failure go away.
1.What a prompt revision costs
A miscalibrated system prompt usually fails in a pattern, not everywhere. The agent might pick the wrong table for one family of questions, skip one required element in its answers, or read one business term the wrong way. The revision should be sized to that pattern, because each word you add is paid for twice.
The first payment is in tokens. The system prompt goes out with every request. Say you add a 400-token policy block to fix a failure that hits 3% of questions: the other 97% carry it too, on every call, for as long as the block stays in the prompt. A one-line rule aimed at the failing case costs a small fraction of that.
The second payment is maintenance. Every rule is one more thing a later editor has to keep consistent with all the other rules and examples. Genie is the Databricks natural-language interface to data: a user asks a business question, and Genie answers by writing SQL over tables registered in Unity Catalog, guided by the table descriptions, instructions and example queries an author has curated. Genie's curation guidance gives a concrete case: if text instructions say to round decimals to two digits, the example SQL queries have to round to two digits as well. A broad rule creates many of these links between parts of the prompt. A narrow rule creates few, and you can delete it as a unit once the underlying cause is fixed.
Length also weakens the prompt itself. Databricks recommends keeping prompts clear and minimal, because extra material brings contradictions and distraction. Genie's guidance says too many instructions reduce effectiveness, especially in long conversations, because the model struggles to prioritise. A sprawling fix can therefore break guidance that was already working, so it is not just expensive but risky.
Checkpoint 1 of 7· Check yourself
A Databricks agent misreads the business term "active customer" in about 3% of questions. All other questions are answered correctly. Which revision fixes the failure at the lowest token cost and maintenance burden?
The system prompt goes out with every request, so a broad block costs tokens on the 97% of questions that never needed it. It also adds rules that later editors have to keep consistent and can bring in contradictions. A narrow rule fixes the pattern, costs few tokens, and can be deleted on its own later.
“Keep prompts clear and minimal to avoid contradictory instructions, distracting information and reduce hallucinations.”Source: docs.databricks.com
2.Climbing the revision ladder
Start from the failure, not from the prompt. Read the failing traces and describe the wrong behaviour in one sentence, for example 'emails end without a concrete next step'. That sentence tells you which part of the prompt to touch. Databricks' evaluation tutorial lists the usual moves: refine prompts to address specific failure patterns, add guidelines for edge cases, include examples showing correct handling.
Climb from the bottom rung and stop as soon as the failure goes away:
- Reword. An instruction for the behaviour already exists but is vague. Replace it with a precise one. The token cost barely changes. Genie's guidance gives the pattern: 'Ask clarification questions when asked about sales' becomes a rule that names exactly which missing details to ask for.
- Add a constraint. Nothing in the prompt covers the behaviour. Add one rule scoped to the failure, as the sales-email tutorial did for missing next steps.
- Add an example. The rule exists but is followed inconsistently, or the correct output is a shape that is tedious to describe in words. Add one targeted example.
- Restructure. The failure comes from how the prompt is organised: instructions that contradict each other, or priorities buried under less important text. Only then is a rewrite worth its cost and its re-testing.
The choice between rung 2 and rung 3 is a real trade-off. An example usually costs more tokens than a one-line rule. It also has to be updated whenever the schema or the output format changes. It wins when the failure is about form: a short sample of the output or query you want can convey the intent in fewer tokens than a long description of it. How much a given example adds is its marginal contribution. Here the only question is whether that contribution beats a cheaper instruction edit.
| Rung | Failure looks like | Added tokens per request | Upkeep |
|---|---|---|---|
| Reword | An existing rule is vague or ambiguous | About zero | None new: same rule, clearer wording |
| Add a constraint | The behaviour is unspecified | One line | One rule to keep consistent |
| Add an example | A rule is ignored, or the output shape is hard to describe | Tens to hundreds | Must track schema and format changes |
| Restructure | Contradictions or buried priorities | Unbounded | Every behaviour must be re-tested |
Compare that with a version that changes many things at once. The summarisation prompt below goes from one line to a persona, a sentence-count rule stated three times, and six guidelines, all in one commit. If quality moves, nothing tells you which line moved it. Every line is now prompt that someone has to keep in step with the rest.
# Version 2: Improved with comprehensive guidelines
prompt_v2 = mlflow.genai.register_prompt(
name=PROMPT_NAME,
template="""You are an expert summarizer. Create a summary of the following content in *exactly* 2 sentences (no more, no less - be very careful about the number of sentences).
Guidelines:
- Include ALL core facts and key findings
- Use clear, concise language
- Maintain factual accuracy
- Cover all main points mentioned
- Write for a general audience
- Use exactly 2 sentences
Content: {{content}}
Summary:""",
commit_message="v2: Added comprehensive fact coverage with 2-sentence requirement"
)Checkpoint 2 of 7· Match them up
Each failure pattern below came from reading an agent's failing traces. Match each one to the cheapest revision that resolves it.
Tap a term, then the definition that fits it.
Each failure points to its own rung. A vague rule that already exists gets reworded. Behaviour the prompt never specifies gets one added constraint. A rule that is ignored, or an output shape that is hard to describe, gets one example. Only contradictions or buried priorities justify a restructure, along with its cost and full re-test.
“Refine system prompts to address specific failure patterns, add explicit guidelines for edge cases, include examples demonstrating correct handling”Source: docs.databricks.com
3.In a Genie space, the cheapest rung is often the metadata
A Genie space's prompt is spread over several places. There are general text instructions (plain-text notes on business rules, terminology and date formats), example SQL queries, and trusted assets: parameterized example queries and Unity Catalog SQL functions whose exact logic an author has verified, so an answer built from one comes from that verified logic. Alongside these sits a knowledge store holding table descriptions, column descriptions, synonyms, SQL expressions (structured definitions of measures, filters and dimensions that capture business logic) and prompt-matching settings (examples that help Genie match values in a question to values in the data and correct spelling slips). Databricks now calls these Genie Agents (formerly Genie spaces); the configuration surfaces are the same. The ladder still applies, but the smallest targeted edit usually belongs on the table or column that caused the failure, not in the instruction text that covers every question.
Databricks states the order of preference directly. Use SQL expressions and example SQL first, and text instructions only as a last resort when those cannot address the need, because structured definitions are more reliable and maintainable than plain text. Knowledge-store edits are also local: they apply to that Genie space only and do not change Unity Catalog metadata or other assets. Fixing a description therefore cannot affect anyone else's tables or agents.
Three failure patterns come up again and again, and each has a narrow fix:
- Wrong table. The space queries a deprecated
marketing_v1instead ofmarketing_v2because the names look alike. Edit each table's description. The default description is the Unity Catalog comment, and you can edit it to add directions that help Genie write SQL. State what the current table is for and mark the old one as historical only. - Wrong column for a business term. 'Revenue' resolves to
gross_revenuewhen users meannet_revenue. Add a column description and synonyms to the correct column. Synonyms are business terms that match user language to column names. If the term is a calculation rather than a column, define it as a SQL expression. - Value mismatch. A user asks about 'Florida', the column stores
FL, and Genie writesILIKE '%Florida%', which returns no rows. Turn on entity matching for that column. Genie then sees representative values and writesWHERE state = 'FL'. Entity matching works only on string columns and requires format assistance (the setting that gives Genie representative values for a column, so it can see data types and formatting patterns) to be on. It covers up to 120 columns with up to 1,024 distinct values each.
| Failure pattern | Targeted revision | Costlier alternatives |
|---|---|---|
| Similar table names, wrong table chosen | Edit table descriptions: purpose of the current table, historical-only note on the deprecated one | A general disambiguation instruction read on every question; several example queries that cover only the questions you foresaw; dropping the table from the warehouse |
| Business term maps to the wrong column | Column description and synonyms on the right column | A long instruction listing every related term; a dedicated warehouse; asking users for column names |
| Spoken values do not match stored codes | Enable entity matching (example values) on the column | An instruction spelling out every mapping; one example query per value; rewriting the Delta table |
| Confusing or duplicate columns pulled into answers | Hide the column in the space | Instructions telling Genie to ignore it |
Hiding a column fits the same pattern. It removes the column from the space's context, so Genie will not reference it, while the underlying data and permissions stay as they are. Every option in the right-hand column either adds text that every question pays for, adds content that goes stale as data changes, or changes systems outside the prompt to fix a problem the prompt can fix.
Checkpoint 3 of 7· Check yourself
In a Genie space, users ask about 'Florida', but the state column stores FL. Genie writes ILIKE '%Florida%' and gets no rows. Which proposed revision fixes this without adding text that goes out with every question?
Entity matching is set on the column that failed, so Genie writes WHERE state = 'FL' and no text is added to what Genie reads for every question. Each of the other options adds instructions or examples that every question pays for and that someone has to maintain, which works against keeping the prompt minimal.
“Keep prompts clear and minimal to avoid contradictory instructions, distracting information and reduce hallucinations.”Source: docs.databricks.com
Checkpoint 4 of 7· Exam question
A Genie space consistently queries the deprecated table "marketing_v1" instead of the current "marketing_v2" when users ask about marketing performance, because the agent selects tables by name similarity rather than purpose. Which revision resolves this with the least added token cost and maintenance burden?
Correct answer: C — Update each table's description to state its purpose and mark the deprecated one as historical only
- A. Incorrect. A general instruction can suppress this one failure, but it grows the space-wide instruction block, adds tokens to every request, and must be revisited whenever new similarly named tables appear, increasing maintenance burden without fixing the underlying ambiguity.
- B. Incorrect. Rewriting the whole instruction set is the highest-cost option: it increases token count substantially and risks regressing other working behavior, when the failure is isolated to one metadata gap.
- C. Correct. The failure stems from unclear table purpose, not missing instructions. Updating the table descriptions targets the root cause directly, costs almost no extra tokens per request, and requires no ongoing maintenance as new questions arrive.
- D. Incorrect. Removing the table is an infrastructure change beyond prompt revision scope, may break other legitimate uses of the deprecated table, and is a heavier-handed fix than necessary.
- E. Incorrect. Example queries help with query logic and formatting, not table selection ambiguity; adding several of them raises token cost without addressing why the wrong table gets picked.
Checkpoint 5 of 7· Exam question
Users asking about "revenue" get totals from the gross_revenue column, but they actually mean net_revenue after returns, so reported figures are consistently too high. What is the most targeted way to revise the agent's configuration to fix this?
Correct answer: A — Add field descriptions and business synonyms clarifying which column corresponds to net revenue
- A. Correct. The failure is column ambiguity, so adding field-level descriptions and synonyms directly maps the business term "revenue" to net_revenue, at a small one-time metadata cost with no per-query token overhead.
- B. Incorrect. A long general instruction enumerating terms grows with every new phrasing users invent, adding tokens to each request and ongoing upkeep, when the fix belongs at the metadata layer.
- C. Incorrect. A separate SQL warehouse changes compute routing, not column selection logic, and does nothing to resolve which column the agent picks.
- D. Incorrect. Shifting the burden to users degrades the natural-language experience the agent is meant to provide and does not fix the underlying ambiguity for anyone who doesn't comply.
- E. Incorrect. A larger model does not resolve a metadata gap; the agent lacks the information to distinguish the columns regardless of reasoning capability, and this adds cost without fixing the cause.
Checkpoint 6 of 7· Exam question
When a user asks about sales in "Australia," a Genie agent's query returns zero rows because the region column stores three-letter codes like "AUS" rather than full country names. Which revision most directly and cheaply fixes this failure?
Correct answer: A — Enable example values or a value dictionary so the agent maps spoken country names to stored codes
- A. Correct. This is exactly the value-matching failure mode that example values and value dictionaries are designed to solve: the agent learns the mapping once, with no added per-query tokens and no ongoing instruction upkeep.
- B. Incorrect. Spelling out every mapping in a general instruction bloats the prompt with data that belongs in metadata, adds tokens to every request, and must be manually updated whenever new countries appear.
- C. Incorrect. Pushing the mapping burden onto users breaks the natural-language interface and does not fix the agent's behavior for anyone who phrases the question naturally.
- D. Incorrect. Changing the underlying table schema is a data engineering change outside prompt revision scope and may break other systems that depend on the coded values.
- E. Incorrect. Enumerating every code in separate example queries is far higher maintenance than a single value dictionary and does not scale as new codes are added.
4.Keeping a revision honest and reversible
A revision is not finished when the one failing case passes. A narrow edit can still regress cases you were not looking at. Wording that steers Genie away from marketing_v1 could also steer it away from the legitimate historical questions that table exists to answer. Before you call the fix done, check both the targeted failure and everything around it.
First, make the revision reversible. Register it as a new prompt version with a commit message that names the failure it targets, and promote it behind an alias as described in Linking prompt versions to runs. If it regresses, moving the alias back to the previous version undoes the change without editing any text. The commit message also lets a future maintainer see why the rule exists before they delete it.
Second, evaluate the new version on the same dataset with the same judges as the version it replaces, using a controlled comparison with mlflow.genai.evaluate(). A judge (also called a scorer) is an automated evaluator that checks each output against one criterion, such as conciseness or safety, and labels it pass or fail. The judge for the targeted failure should flip from fail to pass while the other judges hold steady. If the targeted judge still fails, climb one rung and register another version. Don't pile a second fix into the first. For a Genie space, rerun the benchmark questions. Benchmarks are for evaluation only, and Genie does not use them as context, so they test the revision without adding to its cost.
# Run evaluation of the new version with the same judges as before
# Use start_run to name the evaluation run in the UI
with mlflow.start_run(run_name="v2"):
eval_results_v2 = mlflow.genai.evaluate(
data=eval_dataset, # same eval dataset
predict_fn=generate_sales_email_v2, # new app version
scorers=email_judges, # same judges as step 4
)Automated optimisation needs the same cost check. mlflow.genai.optimize_prompts() produces a prompt that scores better on your evaluation criteria. In Databricks' own example, it turns a one-line prompt into a five-point guideline block. That higher score comes with a much longer prompt, so compare it with the cheapest manual rung that also passes before adopting it.
Move the alias back to the previous version, or leave production on it, and treat the regression as a new failure pattern. Reword the new rule so it no longer pushes the agent toward verbose output, register that wording as another version, and rerun the same dataset and judges. Do not stack a second, compensating rule on top of the first.
Checkpoint 7 of 7· Fill the gap
Your v2 sales-email prompt adds one rule to fix missing next steps. The variable email_judges holds the set of judges that scored the original version, v1. You want this run to show two things: that the fix works, and that nothing else got worse. Which scorers value completes the sample?
# Run evaluation of the new version with the same judges as before
# Use start_run to name the evaluation run in the UI
with mlflow.start_run(run_name="v2"):
eval_results_v2 = mlflow.genai.evaluate(
data=eval_dataset, # same eval dataset
predict_fn=generate_sales_email_v2, # new app version
scorers= ? , # same judges as step 4
)Run v1 and v2 against the same dataset with the same judges. You can then see the targeted judge flip from fail to pass while the other judges hold steady. A judge set limited to the fix, or a new set of judges, would hide regressions and make the two versions impossible to compare.
Source: docs.databricks.comSources
(10)
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Keep prompts clear and minimal to avoid contradictory instructions, distracting information and reduce hallucinations.”
↩︎ What a prompt revision costs“Versioning will simplify operations and allow rollbacks and comparisons.”
↩︎ Keeping a revision honest and reversible“Keep prompts clear and minimal to avoid contradictory instructions, distracting information and reduce hallucinations.”
↩︎ Checkpoint - 2.
“Too many instructions can reduce effectiveness, especially in longer conversations, because Genie might struggle to prioritize the most important guidance.”
↩︎ What a prompt revision costs“if text instructions specify rounding decimals to two digits, then example SQL queries must also round to two digits.”
↩︎ What a prompt revision costs“Avoid vague instructions.”
↩︎ Climbing the revision ladder“Structured definitions through SQL are more reliable and maintainable than plain text guidance.”
↩︎ In a Genie space, the cheapest rung is often the metadata“Genie Agents were formerly known as Genie Spaces.”
↩︎ In a Genie space, the cheapest rung is often the metadata“Use text instructions only as a last resort when SQL expressions and examples cannot address the need.”
↩︎ Exam trap 2 - 3.
“AI/BI Genie allows users to ask questions in natural language and get instant insights from data.”
↩︎ What a prompt revision costs - 4.
“Refine system prompts to address specific failure patterns, add explicit guidelines for edge cases, include examples demonstrating correct handling, or adjust tone or style.”
↩︎ Climbing the revision ladder“Run the evaluation on the improved version using the same judges and dataset to see if you've successfully addressed the issues.”
↩︎ Keeping a revision honest and reversible“Each trace corresponds to one row in your evaluation dataset and is annotated with feedback from each judge.”
↩︎ Keeping a revision honest and reversible“When creating an improved version, focus on targeted changes based on evaluation results.”
↩︎ Key concept“When creating an improved version, focus on targeted changes based on evaluation results.”
↩︎ Exam trap 1“End with a specific, actionable next step that includes a concrete timeline”
↩︎ Prediction“Refine system prompts to address specific failure patterns, add explicit guidelines for edge cases, include examples demonstrating correct handling”
↩︎ Checkpoint - 5.
“A short sample of the output or query you want conveys intent in fewer tokens than a long description.”
↩︎ Climbing the revision ladder - 6.
“All knowledge store configurations are scoped to your Genie Agent and do not affect Unity Catalog metadata or other Databricks assets.”
↩︎ In a Genie space, the cheapest rung is often the metadata“Synonyms: Business terms and keywords that help match user language to column names.”
↩︎ In a Genie space, the cheapest rung is often the metadata“Genie might generate SQL that includes ILIKE '%Florida%', which returns no results.”
↩︎ In a Genie space, the cheapest rung is often the metadata“Entity matching only supports string columns.”
↩︎ In a Genie space, the cheapest rung is often the metadata“Hiding a column removes it from the Genie Agents's context, so Genie won't reference it when generating SQL or answering questions.”
↩︎ In a Genie space, the cheapest rung is often the metadata“Structured definitions of measures, filters, and dimensions that capture business logic.”
↩︎ In a Genie space, the cheapest rung is often the metadata“Format assistance provides representative values for all eligible columns, helping Genie understand data types and formatting patterns.”
↩︎ In a Genie space, the cheapest rung is often the metadata“Edit this description to add specific directions that help Genie author SQL for your agent.”
↩︎ Exam trap 3 - 7.
“Parameterized example queries and SQL functions whose exact logic has been verified by an agent author.”
↩︎ In a Genie space, the cheapest rung is often the metadata“Unlike instructions, benchmarks are for evaluation only.”
↩︎ Keeping a revision honest and reversible - 8.https://www.databricks.com/blog/data-dialogue-best-practices-guide-building-high-performing-genie-spacesSecondary source
“Enable Example Values or Value Dictionaries for categorical columns so Genie can perform exact matches”
↩︎ In a Genie space, the cheapest rung is often the metadata - 9.https://docs.databricks.com/aws/en/mlflow3/genai/prompt-version-mgmt/prompt-registry/evaluate-promptsOfficial docs
“Document changes: Use meaningful commit messages to track why changes were made.”
↩︎ Keeping a revision honest and reversible“Use consistent datasets: Evaluate all versions against the same data for fair comparison.”
↩︎ Exam trap 4 - 10.https://docs.databricks.com/aws/en/mlflow3/genai/prompt-version-mgmt/prompt-registry/automatically-optimize-promptsOfficial docs
“The API produces an improved prompt that performs better on your evaluation criteria.”
↩︎ Keeping a revision honest and reversible