CertSafari
    Databricks Certified Context Engineer Associate· Lessons

    Domain 2 · Lesson 10/45

    Targeted system prompt revisions: the cheapest fix that works

    Given a miscalibrated Databricks agent system prompt and its observed failure pattern, select the targeted revision strategy that resolves the failure with the least increase in token cost and maintenance burden.

    18 min read
    2.25% of exam
    10 sources
    Published 22 Oct 2026
    Docs as of 17 Oct 2026

    What you will be able to do

    • Explain why every prompt revision costs tokens on every request and adds upkeep for later editors
    • Apply the revision ladder (reword, add a constraint, add an example, restructure) to a failure pattern and stop at the first rung that resolves it
    • Pick the Genie knowledge-store edit (table description, column description and synonyms, entity matching) that fixes a failure more cheaply than a general text instruction
    • Confirm a revision on the original evaluation set and keep it reversible as a registered prompt version

    Key concept

    Prompt revision ladder — Name the observed failure pattern first. Then try the cheapest edit aimed at it: reword an existing instruction, add one constraint, add one example, and restructure only as a last resort. Stop at the first rung that makes the failure go away.

    1.What a prompt revision costs

    A miscalibrated system prompt usually fails in a pattern, not everywhere. The agent might pick the wrong table for one family of questions, skip one required element in its answers, or read one business term the wrong way. The revision should be sized to that pattern, because each word you add is paid for twice.

    The first payment is in tokens. The system prompt goes out with every request. Say you add a 400-token policy block to fix a failure that hits 3% of questions: the other 97% carry it too, on every call, for as long as the block stays in the prompt. A one-line rule aimed at the failing case costs a small fraction of that.

    The second payment is maintenance. Every rule is one more thing a later editor has to keep consistent with all the other rules and examples. Genie is the Databricks natural-language interface to data: a user asks a business question, and Genie answers by writing SQL over tables registered in Unity Catalog, guided by the table descriptions, instructions and example queries an author has curated. Genie's curation guidance gives a concrete case: if text instructions say to round decimals to two digits, the example SQL queries have to round to two digits as well. A broad rule creates many of these links between parts of the prompt. A narrow rule creates few, and you can delete it as a unit once the underlying cause is fixed.

    Length also weakens the prompt itself. Databricks recommends keeping prompts clear and minimal, because extra material brings contradictions and distraction. Genie's guidance says too many instructions reduce effectiveness, especially in long conversations, because the model struggles to prioritise. A sprawling fix can therefore break guidance that was already working, so it is not just expensive but risky.

    Checkpoint 1 of 7· Check yourself

    A Databricks agent misreads the business term "active customer" in about 3% of questions. All other questions are answered correctly. Which revision fixes the failure at the lowest token cost and maintenance burden?

    Sources123

    2.Climbing the revision ladder

    Start from the failure, not from the prompt. Read the failing traces and describe the wrong behaviour in one sentence, for example 'emails end without a concrete next step'. That sentence tells you which part of the prompt to touch. Databricks' evaluation tutorial lists the usual moves: refine prompts to address specific failure patterns, add guidelines for edge cases, include examples showing correct handling.

    Climb from the bottom rung and stop as soon as the failure goes away:

    1. Reword. An instruction for the behaviour already exists but is vague. Replace it with a precise one. The token cost barely changes. Genie's guidance gives the pattern: 'Ask clarification questions when asked about sales' becomes a rule that names exactly which missing details to ask for.
    2. Add a constraint. Nothing in the prompt covers the behaviour. Add one rule scoped to the failure, as the sales-email tutorial did for missing next steps.
    3. Add an example. The rule exists but is followed inconsistently, or the correct output is a shape that is tedious to describe in words. Add one targeted example.
    4. Restructure. The failure comes from how the prompt is organised: instructions that contradict each other, or priorities buried under less important text. Only then is a rewrite worth its cost and its re-testing.

    The choice between rung 2 and rung 3 is a real trade-off. An example usually costs more tokens than a one-line rule. It also has to be updated whenever the schema or the output format changes. It wins when the failure is about form: a short sample of the output or query you want can convey the intent in fewer tokens than a long description of it. How much a given example adds is its marginal contribution. Here the only question is whether that contribution beats a cheaper instruction edit.

    Rungs of the revision ladder, from cheapest to most expensive
    RungFailure looks likeAdded tokens per requestUpkeep
    RewordAn existing rule is vague or ambiguousAbout zeroNone new: same rule, clearer wording
    Add a constraintThe behaviour is unspecifiedOne lineOne rule to keep consistent
    Add an exampleA rule is ignored, or the output shape is hard to describeTens to hundredsMust track schema and format changes
    RestructureContradictions or buried prioritiesUnboundedEvery behaviour must be re-tested

    Compare that with a version that changes many things at once. The summarisation prompt below goes from one line to a persona, a sentence-count rule stated three times, and six guidelines, all in one commit. If quality moves, nothing tells you which line moved it. Every line is now prompt that someone has to keep in step with the rest.

    A version that adds many rules in one step: hard to tell which line did the workpython
    # Version 2: Improved with comprehensive guidelines
    prompt_v2 = mlflow.genai.register_prompt(
        name=PROMPT_NAME,
        template="""You are an expert summarizer. Create a summary of the following content in *exactly* 2 sentences (no more, no less - be very careful about the number of sentences).
    
    Guidelines:
    - Include ALL core facts and key findings
    - Use clear, concise language
    - Maintain factual accuracy
    - Cover all main points mentioned
    - Write for a general audience
    - Use exactly 2 sentences
    
    Content: {{content}}
    
    Summary:""",
        commit_message="v2: Added comprehensive fact coverage with 2-sentence requirement"
    )

    Checkpoint 2 of 7· Match them up

    Each failure pattern below came from reading an agent's failing traces. Match each one to the cheapest revision that resolves it.

    Tap a term, then the definition that fits it.

    Sources452

    3.In a Genie space, the cheapest rung is often the metadata

    A Genie space's prompt is spread over several places. There are general text instructions (plain-text notes on business rules, terminology and date formats), example SQL queries, and trusted assets: parameterized example queries and Unity Catalog SQL functions whose exact logic an author has verified, so an answer built from one comes from that verified logic. Alongside these sits a knowledge store holding table descriptions, column descriptions, synonyms, SQL expressions (structured definitions of measures, filters and dimensions that capture business logic) and prompt-matching settings (examples that help Genie match values in a question to values in the data and correct spelling slips). Databricks now calls these Genie Agents (formerly Genie spaces); the configuration surfaces are the same. The ladder still applies, but the smallest targeted edit usually belongs on the table or column that caused the failure, not in the instruction text that covers every question.

    Databricks states the order of preference directly. Use SQL expressions and example SQL first, and text instructions only as a last resort when those cannot address the need, because structured definitions are more reliable and maintainable than plain text. Knowledge-store edits are also local: they apply to that Genie space only and do not change Unity Catalog metadata or other assets. Fixing a description therefore cannot affect anyone else's tables or agents.

    Three failure patterns come up again and again, and each has a narrow fix:

    • Wrong table. The space queries a deprecated marketing_v1 instead of marketing_v2 because the names look alike. Edit each table's description. The default description is the Unity Catalog comment, and you can edit it to add directions that help Genie write SQL. State what the current table is for and mark the old one as historical only.
    • Wrong column for a business term. 'Revenue' resolves to gross_revenue when users mean net_revenue. Add a column description and synonyms to the correct column. Synonyms are business terms that match user language to column names. If the term is a calculation rather than a column, define it as a SQL expression.
    • Value mismatch. A user asks about 'Florida', the column stores FL, and Genie writes ILIKE '%Florida%', which returns no rows. Turn on entity matching for that column. Genie then sees representative values and writes WHERE state = 'FL'. Entity matching works only on string columns and requires format assistance (the setting that gives Genie representative values for a column, so it can see data types and formatting patterns) to be on. It covers up to 120 columns with up to 1,024 distinct values each.
    Failure pattern against the targeted Genie revision and the costlier alternatives it beats
    Failure patternTargeted revisionCostlier alternatives
    Similar table names, wrong table chosenEdit table descriptions: purpose of the current table, historical-only note on the deprecated oneA general disambiguation instruction read on every question; several example queries that cover only the questions you foresaw; dropping the table from the warehouse
    Business term maps to the wrong columnColumn description and synonyms on the right columnA long instruction listing every related term; a dedicated warehouse; asking users for column names
    Spoken values do not match stored codesEnable entity matching (example values) on the columnAn instruction spelling out every mapping; one example query per value; rewriting the Delta table
    Confusing or duplicate columns pulled into answersHide the column in the spaceInstructions telling Genie to ignore it

    Hiding a column fits the same pattern. It removes the column from the space's context, so Genie will not reference it, while the underlying data and permissions stay as they are. Every option in the right-hand column either adds text that every question pays for, adds content that goes stale as data changes, or changes systems outside the prompt to fix a problem the prompt can fix.

    Checkpoint 3 of 7· Check yourself

    In a Genie space, users ask about 'Florida', but the state column stores FL. Genie writes ILIKE '%Florida%' and gets no rows. Which proposed revision fixes this without adding text that goes out with every question?

    Checkpoint 4 of 7· Exam question

    A Genie space consistently queries the deprecated table "marketing_v1" instead of the current "marketing_v2" when users ask about marketing performance, because the agent selects tables by name similarity rather than purpose. Which revision resolves this with the least added token cost and maintenance burden?

    Checkpoint 5 of 7· Exam question

    Users asking about "revenue" get totals from the gross_revenue column, but they actually mean net_revenue after returns, so reported figures are consistently too high. What is the most targeted way to revise the agent's configuration to fix this?

    Checkpoint 6 of 7· Exam question

    When a user asks about sales in "Australia," a Genie agent's query returns zero rows because the region column stores three-letter codes like "AUS" rather than full country names. Which revision most directly and cheaply fixes this failure?

    Sources2678

    4.Keeping a revision honest and reversible

    A revision is not finished when the one failing case passes. A narrow edit can still regress cases you were not looking at. Wording that steers Genie away from marketing_v1 could also steer it away from the legitimate historical questions that table exists to answer. Before you call the fix done, check both the targeted failure and everything around it.

    First, make the revision reversible. Register it as a new prompt version with a commit message that names the failure it targets, and promote it behind an alias as described in Linking prompt versions to runs. If it regresses, moving the alias back to the previous version undoes the change without editing any text. The commit message also lets a future maintainer see why the rule exists before they delete it.

    Second, evaluate the new version on the same dataset with the same judges as the version it replaces, using a controlled comparison with mlflow.genai.evaluate(). A judge (also called a scorer) is an automated evaluator that checks each output against one criterion, such as conciseness or safety, and labels it pass or fail. The judge for the targeted failure should flip from fail to pass while the other judges hold steady. If the targeted judge still fails, climb one rung and register another version. Don't pile a second fix into the first. For a Genie space, rerun the benchmark questions. Benchmarks are for evaluation only, and Genie does not use them as context, so they test the revision without adding to its cost.

    Re-running the original evaluation against the revised version: same data, same judges. Here `email_judges` is the set of judges that scored the original version; 'step 4' in the comment refers to that original evaluation in the tutorial the sample comes frompython
    # Run evaluation of the new version with the same judges as before
    # Use start_run to name the evaluation run in the UI
    with mlflow.start_run(run_name="v2"):
        eval_results_v2 = mlflow.genai.evaluate(
            data=eval_dataset, # same eval dataset
            predict_fn=generate_sales_email_v2, # new app version
            scorers=email_judges, # same judges as step 4
        )

    Automated optimisation needs the same cost check. mlflow.genai.optimize_prompts() produces a prompt that scores better on your evaluation criteria. In Databricks' own example, it turns a one-line prompt into a five-point guideline block. That higher score comes with a much longer prompt, so compare it with the cheapest manual rung that also passes before adopting it.

    Checkpoint 7 of 7· Fill the gap

    Your v2 sales-email prompt adds one rule to fix missing next steps. The variable email_judges holds the set of judges that scored the original version, v1. You want this run to show two things: that the fix works, and that nothing else got worse. Which scorers value completes the sample?

    # Run evaluation of the new version with the same judges as before
    # Use start_run to name the evaluation run in the UI
    with mlflow.start_run(run_name="v2"):
        eval_results_v2 = mlflow.genai.evaluate(
            data=eval_dataset, # same eval dataset
            predict_fn=generate_sales_email_v2, # new app version
            scorers= ? , # same judges as step 4
        )

    Sources194710

    Sources

    (10)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Keep prompts clear and minimal to avoid contradictory instructions, distracting information and reduce hallucinations.”
      ↩︎ What a prompt revision costs
      “Versioning will simplify operations and allow rollbacks and comparisons.”
      ↩︎ Keeping a revision honest and reversible
      “Keep prompts clear and minimal to avoid contradictory instructions, distracting information and reduce hallucinations.”
      ↩︎ Checkpoint
    2. 2.
      “Too many instructions can reduce effectiveness, especially in longer conversations, because Genie might struggle to prioritize the most important guidance.”
      ↩︎ What a prompt revision costs
      “if text instructions specify rounding decimals to two digits, then example SQL queries must also round to two digits.”
      ↩︎ What a prompt revision costs
      “Avoid vague instructions.”
      ↩︎ Climbing the revision ladder
      “Structured definitions through SQL are more reliable and maintainable than plain text guidance.”
      ↩︎ In a Genie space, the cheapest rung is often the metadata
      “Genie Agents were formerly known as Genie Spaces.”
      ↩︎ In a Genie space, the cheapest rung is often the metadata
      “Use text instructions only as a last resort when SQL expressions and examples cannot address the need.”
      ↩︎ Exam trap 2
    3. 3.
      “AI/BI Genie allows users to ask questions in natural language and get instant insights from data.”
      ↩︎ What a prompt revision costs
    4. 4.
      “Refine system prompts to address specific failure patterns, add explicit guidelines for edge cases, include examples demonstrating correct handling, or adjust tone or style.”
      ↩︎ Climbing the revision ladder
      “Run the evaluation on the improved version using the same judges and dataset to see if you've successfully addressed the issues.”
      ↩︎ Keeping a revision honest and reversible
      “Each trace corresponds to one row in your evaluation dataset and is annotated with feedback from each judge.”
      ↩︎ Keeping a revision honest and reversible
      “When creating an improved version, focus on targeted changes based on evaluation results.”
      ↩︎ Key concept
      “When creating an improved version, focus on targeted changes based on evaluation results.”
      ↩︎ Exam trap 1
      “End with a specific, actionable next step that includes a concrete timeline”
      ↩︎ Prediction
      “Refine system prompts to address specific failure patterns, add explicit guidelines for edge cases, include examples demonstrating correct handling”
      ↩︎ Checkpoint
    5. 5.
      “A short sample of the output or query you want conveys intent in fewer tokens than a long description.”
      ↩︎ Climbing the revision ladder
    6. 6.
      “All knowledge store configurations are scoped to your Genie Agent and do not affect Unity Catalog metadata or other Databricks assets.”
      ↩︎ In a Genie space, the cheapest rung is often the metadata
      “Synonyms: Business terms and keywords that help match user language to column names.”
      ↩︎ In a Genie space, the cheapest rung is often the metadata
      “Genie might generate SQL that includes ILIKE '%Florida%', which returns no results.”
      ↩︎ In a Genie space, the cheapest rung is often the metadata
      “Entity matching only supports string columns.”
      ↩︎ In a Genie space, the cheapest rung is often the metadata
      “Hiding a column removes it from the Genie Agents's context, so Genie won't reference it when generating SQL or answering questions.”
      ↩︎ In a Genie space, the cheapest rung is often the metadata
      “Structured definitions of measures, filters, and dimensions that capture business logic.”
      ↩︎ In a Genie space, the cheapest rung is often the metadata
      “Format assistance provides representative values for all eligible columns, helping Genie understand data types and formatting patterns.”
      ↩︎ In a Genie space, the cheapest rung is often the metadata
      “Edit this description to add specific directions that help Genie author SQL for your agent.”
      ↩︎ Exam trap 3
    7. 7.
      “Parameterized example queries and SQL functions whose exact logic has been verified by an agent author.”
      ↩︎ In a Genie space, the cheapest rung is often the metadata
      “Unlike instructions, benchmarks are for evaluation only.”
      ↩︎ Keeping a revision honest and reversible
    8. 8.
      “Enable Example Values or Value Dictionaries for categorical columns so Genie can perform exact matches”
      ↩︎ In a Genie space, the cheapest rung is often the metadata
    9. 9.
      “Document changes: Use meaningful commit messages to track why changes were made.”
      ↩︎ Keeping a revision honest and reversible
      “Use consistent datasets: Evaluate all versions against the same data for fair comparison.”
      ↩︎ Exam trap 4

    Ready to test yourself?

    Practise Databricks Certified Context Engineer Associate in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.