Revising a Miscalibrated Agent System Prompt

1.Start from the failure pattern, not the prompt

On the exam, questions in this subdomain give you two things: a system prompt and a description of how the agent misbehaves. For example, it ignores a formatting rule, answers questions outside its scope, refuses things it should handle, or follows an instruction too literally. Before choosing a fix, decide what kind of failure this is. Databricks splits quality problems for retrieval-backed agents into two kinds. Retrieval quality asks whether the right context reached the model. Generation quality asks whether the model used that context well. The docs describe generation problems as ones that show up as hallucinations, inconsistent output, or answers that miss the user's question.

The difference matters because only one of these is a system prompt problem. If the trace shows the retriever returned the wrong chunks, no wording change will fix the answer. Adding instructions just makes every request more expensive and leaves the real problem in place. MLflow traces let you open a failing request and look at each step's inputs and outputs, so you can find the step where things went wrong before you touch any text. Your first job is to confirm that the prompt really is the problem.

Once the prompt is the culprit, the revision should match the symptom. The Databricks evaluate-and-improve tutorial shows one example. Its improved sales-email prompt adds a line saying to follow the user's instructions exactly, tells the model to use customer context only where it is relevant, and tells it to reference customer information only if it is directly relevant to the instructions. That is a small, symptom-specific edit for an agent that ignored instructions and padded its output. The sources do not give a fix for each symptom, so treat the following as a reasoning heuristic. For scope creep or padding, add or tighten one relevance guideline. For over-literal compliance or over-refusal, look first for the one existing instruction that is over-weighted or ambiguous and reword it, rather than adding more rules on top.

The same logic applies when the agent is a Genie agent, where the equivalent of a prompt is its configuration. Databricks says table and column descriptions and synonyms help Genie choose the right column and write more accurate SQL. It says instructions that dictate which tables or columns to use are really the job of descriptions and synonyms. If Genie confuses two similar columns, such as a gross and a net revenue field, the targeted fix is a clearer column description or a synonym, not a new text instruction. The docs also describe prompt matching, which helps Genie match values in a user's question and correct spelling. For a mismatch between how users phrase a value and how it is stored, check prompt matching and example SQL before adding rules. Databricks says to use text instructions only as a last resort, and to put hardcoded-value logic in the data layer or in an example query. A data-layer change such as removing a table is a different lever from a targeted configuration fix, and the docs reserve it for universal rules.

Sources12345

2.A ladder of revisions, cheapest first

The Databricks evaluate-and-improve tutorial lists common improvement strategies. Prompt engineering covers refining the system prompt for a specific failure, adding explicit guidelines for edge cases, and including examples. Guardrails means validation steps and post-processing in application logic. Reasoning enhancements include breaking a complex task into multiple spans or adding verification steps. The docs list these strategies but do not rank them by cost. The ordering below is this lesson's own rule of thumb: roughly, prefer changes that add the fewest tokens per request and the fewest new parts to maintain. Guardrails break the pattern, because they add code to maintain but little prompt growth. The useful skill is matching the failure to the cheapest strategy that actually fixes it.

Pay attention to the guardrail row. If a rule must hold on every response (a forbidden output, a required field), restating it in the prompt over and over gives no guarantee and costs tokens each time. A deterministic check in code can be the cheaper and more reliable fix, even though it adds code to maintain. The tutorial describes guardrails as validation in application logic and post-processing before the output reaches users.

Genie agents have their own version of this ladder. Databricks recommends prioritizing SQL expressions and example SQL over text instructions, and describes text instructions as the last resort. Column descriptions and synonyms sit at the cheap end because they are scoped to the agent and shape how Genie picks columns. A complete example query suits a hard, multi-part question. A plain-text instruction is for global context that fits nowhere else, such as a fiscal-year rule or a formatting rule. Choosing the right lever keeps the instruction set small, and Databricks warns that too many instructions can reduce effectiveness.

Sources34

3.Weighing token cost against maintenance burden

Every system prompt token is sent with every request, so a revision that adds a paragraph adds to the cost of the whole workload, not just the failing cases. Databricks treats this as something you measure. Overall latency and token consumption are metrics you compute alongside quality, so you can compare the token count of a proposed revision with the current version before you accept it. Genie agents make the same point: their text instructions and metadata are converted into tokens, and Databricks suggests streamlining descriptions, pruning redundant example queries, and simplifying instructions to cut the count.

Maintenance burden is the other cost. Every added rule is something a future editor has to understand, keep consistent with the other rules, and re-check when requirements change. Databricks' Genie guidance says to avoid conflicting instructions and to keep consistency across all instruction types, and warns that too many instructions can reduce effectiveness. Splitting work into multiple LLM calls is at the top of the ladder for a reason. The Databricks RAG cookbook says decomposition can improve results but notes that it adds latency in exchange for finer control. Databricks agent design guidance points the same way: start simple and add complexity only when you truly need it.

Sources6789410

4.Verify the revision and keep it reversible

A targeted revision only counts once you have shown it works. The MLflow Prompt Registry stores prompts as versioned artifacts in Unity Catalog, with commit messages, aliases, and rollback. Registering each revision as a new version, with a commit message explaining which failure it targets, keeps maintenance manageable. A later editor can see why each rule exists and can roll back a revision that caused a regression.

# Register a prompt template
prompt = mlflow.genai.register_prompt(
    name="docs.default.customer_support",
    template="You are a helpful assistant. Answer this question: {{question}}",
    commit_message="Initial customer support prompt"
)
print(f"Created version {prompt.version}")  # "Created version 1"

# Set a production alias
mlflow.genai.set_prompt_alias(
    name="docs.default.customer_support",
    alias="production",
    version=1
)

prompt_v1 = mlflow.genai.register_prompt(
    name=PROMPT_NAME,
    template="Summarize this text: {{content}}",
     ? ="v1: Basic summarization prompt"
)

Run the old and new versions against the same evaluation dataset, and include the failing cases that prompted the revision as well as cases that already worked. That shows two things: the target failure is fixed, and the change did not break behavior that was correct before. If the evaluation also records token counts, you can see in one comparison whether the revision paid for itself.

Sources1112

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Rewriting the whole system prompt into a comprehensive guideline list is the safest way to fix one observed failure.Why is that wrong?

   Databricks recommends targeted changes driven by evaluation results. A full rewrite adds tokens to every request, adds more for someone to maintain, and can introduce new failures in behavior that already worked.

   Covered in A ladder of revisions, cheapest first

2. 2.Any wrong answer from an agent means the system prompt needs more instructions.Why is that wrong?

   Bad responses can come from faulty retrieval even when generation is fine. Check the trace first; a prompt edit cannot supply context that was never retrieved.

   Covered in Start from the failure pattern, not the prompt

3. 3.Splitting one prompt into several chained LLM calls is a low-cost fix for any instruction-following problem.Why is that wrong?

   Decomposition can improve control, but it adds LLM calls and latency. Use it when one prompt really is carrying too much logic, not as a first response to a single misworded instruction.

   Covered in Weighing token cost against maintenance burden