Detecting Context Rot in Long-Running Databricks Agents

1.Why a long interaction degrades

An agent that answers its first questions well can still fail in turn forty, even though nobody changed the model, the prompt or the tools. The difference is how much context it now carries. On Databricks, the state of one interaction is its session: most often the conversation history, meaning the ordered transcript of messages, tool calls and results. The agent reads this at the start of each turn and appends to it as it works. The documentation describes a session as something that grows within the interaction and is complete on its own.

Databricks' AI harness article calls this failure context rot. As history accumulates, reasoning quality falls, and without a strategy to trim or summarize older context, performance often breaks down on long-running tasks. The problem sits in the harness, the system around the model that manages memory and runs tools, so the fix belongs there as well. A related harness failure makes things worse: when the harness pulls the wrong information from memory or search, the model may confidently generate incorrect answers. A long, cluttered context hurts both of the qualities this objective names, reasoning quality and retrieval accuracy. (Two terms from the exam vocabulary, "lost in the middle" and "attention budget", do not appear in the sources for this lesson, so this lesson makes no specific claims about them.)

Sources12

2.Measuring quality across the whole session

You cannot see degradation over time if every turn is graded on its own. Traditional single-turn evaluation scores each agent response independently. MLflow's conversation evaluation instead scores whole sessions, and the documentation lists the qualities that only show up at that level. The one most closely tied to context length is knowledge retention: does the agent remember information from earlier in the conversation? If retention fails only after many turns, the size of the context is the likely cause.

Session-level evaluation depends on instrumentation. A trace records one complete execution of the agent, including every LLM call, retriever lookup and tool call. A session is a tag that groups the traces of one multi-turn conversation. Set it on every trace, and MLflow groups the traces by that tag when it evaluates them.

Be clear about what a session-level judge gives you. Multi-turn assessments are stored on the first trace of each session and are marked as conversation-level, linked to the session ID. They describe the conversation as a whole, so they tell you that a conversation went badly, not which turn it went bad at. To locate the turn, you need metrics recorded on each trace, which the next section covers, and you read them in session order.

import mlflow

@mlflow.trace
def my_chatbot(question, session_id):
    mlflow. ? (
        tags={"mlflow.trace.session": session_id}
    )
    # ... your chatbot logic

# Get complete sessions (each session is a list of traces)
sessions = mlflow.search_sessions(
    locations=["<your-experiment-id>"],
    max_results=50,
)

Sources34

3.Metrics that show where quality drops

Once traces are grouped by session, you need metrics that separate the two kinds of damage the objective names. Databricks' recommended RAG metrics divide into retrieval, response, cost and latency. Retrieval metrics show whether the right chunks are still coming back as the conversation grows. Response metrics show whether the model still reasons correctly over what it was given. The token-count metrics show how large the context has become. Comparing these by position in the session tells you when the context grew and whether quality fell at the same point.

To pinpoint the degradation point, line the per-trace metrics up in session order. For each session, take the traces in chronological order and read chunk_relevance/precision, groundedness and total_input_token_count for each one. Then plot the quality metrics against the turn number, or against the input-token count. Early turns give you a baseline. The degradation point is where a quality metric turns downward while total_input_token_count keeps climbing. For example, if precision holds steady for the first stretch of turns and then falls as input tokens keep rising, the context size at that bend is the point to act on. If precision stays steady but groundedness falls, the retrieved chunks are still fine, so look at how the model handles the context it was given. The sources give no fixed threshold, so judge the drop against your own early-turn baseline, and check that it repeats across several sessions before you treat it as context rot. Then use the session-level knowledge retention result to confirm that the conversation as a whole suffered.

Some products report a limit directly, but be careful about what it measures. In a Genie Agent, text instructions and metadata are converted into tokens, and a warning appears when the agent approaches its token limit. That count reflects the agent's configuration rather than how long the conversation has run, so it shows how context filtering can cost quality but is not a measure of a growing interaction. Genie uses context filtering to keep the tokens it considers most relevant. Responses should still be generated at the warning, but quality may be reduced if important context is filtered out. Only when the limit is exceeded can you no longer send or receive messages. So the warning means quality is at risk. It does not mean the agent has stopped working.

Sources56

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Running the usual per-response evaluation on every turn is enough to catch an agent that forgets earlier information in long conversations.Why is that wrong?

   Single-turn evaluation scores each response on its own. Knowledge retention can only be judged when traces are grouped into a session and the full conversation history is evaluated.

   Covered in Measuring quality across the whole session

2. 2.A token-limit warning in a Genie Agent means the agent has stopped answering.Why is that wrong?

   At the warning, responses are still generated but may be lower quality because of context filtering. Messages are blocked only once the limit is exceeded.

   Covered in Metrics that show where quality drops