CertSafari
    Anthropic Claude Certified Associate – Foundations (CCAO-F) - December 7 onwards· Lessons

    Domain 5 · Lesson 18/21

    Diagnosing a degraded Claude configuration

    Configuration Lifecycle and Degradation

    17 min read
    3.75% of exam
    7 sources
    Published 18 Nov 2026
    Docs as of 18 Nov 2026

    What you will be able to do

    • Name the four layers of an established Claude configuration (instructions, knowledge, Skills, memory) and describe how each one goes stale
    • Check whether uploaded knowledge files are still current and replace superseded copies instead of leaving old and new versions side by side
    • Find which layer caused a drop in output quality with a reproducible test, changing one layer at a time
    • Repair instructions that have drifted or contradict each other by rewriting them, not by adding more rules
    • Roll back a regressed prompt or Skill version and correct or archive bad memories without losing their history
    • Tell the team about a configuration change and set the next review

    Key concept

    Layer attribution — A configuration that was once good can get worse because one of its layers drifted away from reality: the standing instructions, the knowledge files, a Skill or prompt version, or accumulated memory. To fix it, find which layer is responsible by changing one layer at a time against a fixed test, and leave the layers that still work alone.

    1.The four layers that can go stale

    A Claude setup that has been in use for months is several separately maintained parts, and each part ages on its own schedule. In a Project you upload documents, text, code or other files to a knowledge base, and you write project instructions that shape every reply, such as a formal tone or a specific role. Claude Code has the same split. CLAUDE.md files hold rules you write yourself. Auto memory holds notes Claude writes itself based on your corrections and preferences. Skills add reusable prompts that run when you type their name or that Claude invokes automatically. Setting these layers up is covered in Project configuration. This lesson is about what happens to them afterwards.

    Each layer, who maintains it, and how it typically drifts
    LayerWho writes itTypical way it goes stale
    Instructions (Project instructions, CLAUDE.md)You or your teamWritten for an old policy or process and never revised; rules added over time that contradict earlier ones
    Knowledge (project knowledge base, reference files)You or your teamDocuments superseded by newer ones, or old and new versions of a document both left in place
    Skills and prompt versionsYou or your teamA new version ships with a broader rule that quietly breaks a case that used to work
    Memory (auto memory, memory stores)Claude, from corrections and preferencesNotes about old preferences or past mistakes that are no longer true

    These layers can contradict each other. Suppose a team refreshes its knowledge files to a new pricing policy, but the instructions still state the old one. Every chat now gets two policies: the outdated one from the instructions, which load every time, and the current one from the files. When two instructions contradict each other, Claude may pick one arbitrarily, so drafts end up mixing the two. The layer to blame is the one still holding the outdated text. The model is not inventing the old policy, and memory is not the source either, because the outdated wording sits in the instructions and nobody edited them when the files changed.

    Knowledge freshness is its own problem, because knowledge files do not update themselves. A file uploaded to a Project is a copy: a change made to the original on your computer does not reach the Project until you upload the file again and choose Replace when asked about the existing name. A team that edits the master pricing sheet and forgets the re-upload has a Project answering confidently from last quarter's numbers, and nothing in the output flags it. Uploading the new version under a different name makes it worse, since the old and new copies then both sit in the knowledge base. When quality slips after a real-world change, compare the upload date of each knowledge file with the date its source last changed, replace the stale copies in place, and delete files nobody needs any more. Outdated material is worse than none, because Claude has no way to tell which version is current.

    Checkpoint 1 of 9· Check yourself

    A team's Project has project instructions that state the old 30-day refund policy. Last week the team replaced the knowledge files with the new 14-day policy and changed nothing else. Claude's draft replies now mix the two policies. Which layer is responsible?

    Checkpoint 2 of 9· Exam question

    A finance team shares a Claude Project that drafts client payment-terms emails. Six months ago the Project instructions were written to say "offer net-30 terms and a 2% early-payment discount." The company has since moved to net-45 with no discount, and the Project knowledge files were updated to match. Drafts now mix the two policies. The team lead must attribute the problem to the right layer. Which layer is most likely responsible?

    Sources123

    2.A diagnostic process that isolates the responsible layer

    Diagnosis needs evidence first and edits second. Typical signs of drift are Claude making the same mistake a second time, typing the same correction you typed last session, or a review catching something Claude should already have known. Turn those signs into a test before you change anything:

    1. Collect a few recent weak outputs and a few older strong ones for the same type of request, and write down exactly how they differ.
    2. Rerun one identical test request in a fresh chat and confirm that the weak behaviour reproduces. A fresh session carries no earlier conversation, so if the problem still appears, it comes from the standing configuration.
    3. Change one suspected layer at a time (instructions, knowledge, Skill version, memory) and rerun the same request after each change.
    4. Keep the change that restores quality, and revert any change that made no difference.
    5. Tell the team what changed and why, and schedule the next review.

    The Managed Agents prompt-versioning example shows step 3 in practice. A routing agent was scored against twenty labelled tickets. A new prompt version was then scored against the same set, with each test session pinned to one exact version so the only thing that differed was the prompt. Breaking the score down by category showed where the damage was.

    Scoring the same labelled tickets against two pinned prompt versions
    Teamv1 scorev2 score
    api-platform5/55/5
    auth5/55/5
    billing4/52/5 (regressed)
    dashboard5/55/5

    The new rule sent anything about API usage to the platform team. On a usage-billed product, billing tickets also talk about API usage, so the broader rule took them over. An overall score would have hidden this, but the per-category comparison showed it immediately. In Claude Code, check what actually loaded before you blame a layer: /context, /memory and /skills show what is in effect, and a Skill can be installed and still never run if its description doesn’t match how you phrase the request.

    Checkpoint 3 of 9· Check yourself

    You suspect that a new prompt version for a routing agent has made its results worse. How should you set up the test so the result shows whether the prompt caused the drop?

    Checkpoint 4 of 9· Exam question

    A consultant's Claude Project has produced slightly worse proposals for several weeks, and the consultant wants to restore its performance without breaking the parts that still work. Which sequence should the consultant follow before the team relies on the Project again?

    Sources2456

    3.Repairing instructions that have drifted

    Instructions usually degrade by accumulation. Six clear lines become thirty as colleagues add rules over a year. One person writes "keep replies under 100 words", and a later person writes "always include full background and three examples". Each rule made sense when it was added. Together they contradict each other, and since Claude may pick one arbitrarily when two conflict, output length and format vary from chat to chat. Length causes a second problem: longer instruction files take up more context and reduce adherence.

    First, rewrite the instructions as one deduplicated list. For each contradictory pair, choose one wording and delete the other. Make each rule specific enough to check: "Use 2-space indentation", not "Format code properly". Group related rules under headers, because organised sections are easier for Claude to follow than dense paragraphs. Writing good standing instructions is covered in persistent-instruction design. The point here is that once instructions have drifted, you repair them by subtracting rules. Adding more does not help.

    Claude Code has tools for this audit. The prompt-audit subcommand flags instructions written for older models in prompts, Skills and tool descriptions and proposes fixes as a diff. /doctor reports checked-in CLAUDE.md content that Claude could work out from the codebase anyway, which is a common kind of bloat.

    Checkpoint 5 of 9· Check yourself

    Over a year, colleagues have grown a project's instructions from six lines to thirty. One rule says "keep replies under 100 words" and a later one says "always include full background and three examples". Reply length now varies from chat to chat. Which repair fixes the inconsistency?

    Checkpoint 6 of 9· Exam question

    A marketing agency has a Project whose instructions began as six short lines. Over a year, different colleagues appended new rules: "keep every reply under 100 words," later "always include full background and three examples," and "never use bullet points" followed by "present options as bullets." Claude's replies are now inconsistent in length and format. What should the Project owner do first to restore consistent output?

    Sources276

    4.Rolling back Skill versions and correcting memory

    Versioned layers are the easiest to repair, because the earlier good state still exists. In Managed Agents, every agents.update creates a new immutable version, and sessions choose which version to use by ID. When the billing regression appeared, version 1 was still stored on the server, so rolling back meant passing version: 1 again. Nothing had to be redeployed. If callers pass the bare agent ID, they get whatever version is newest, so an untested change goes live on their next session. That is why production callers should pin an explicit version.

    Memory needs a different kind of care, because Claude writes it, and it can hold user preferences, project conventions, prior mistakes and domain context. A note that was once true becomes a standing instruction nobody asked for. Auto memory loads into every session, up to the first 200 lines or 25KB, so a stale note affects every session until someone removes it. Managed Agents memory stores can be edited directly through the API, which is how you build review workflows, correct bad memories, or seed a store before any session runs. To prune a stale memory without destroying it, rename it to an archive path:

    Archiving a stale memory by renaming its pathpython
    client.beta.memory_stores.memories.update(
        mem.id,
        memory_store_id=store.id,
        path="/archive/2026_q1_formatting.md",
    )

    Every mutation to a memory creates an immutable version, and those versions are not deleted when the memory itself is deleted. If a correction makes things worse, you can undo it. To roll back, retrieve the version you want and write its content back with an update. Versions are kept for 30 days after they are written, though the recent versions of a live memory are always kept. Export versions you need to keep for longer.

    Checkpoint 7 of 9· Fill the gap

    This call archives a stale memory so it stops acting as a standing instruction but is not destroyed. Which method completes it?

    client.beta.memory_stores.memories. ? (
        mem.id,
        memory_store_id=store.id,
        path="/archive/2026_q1_formatting.md",
    )

    Checkpoint 8 of 9· Check yourself

    Ten days ago, an engineer deleted a Managed Agents memory that they thought was stale. Since then, the agent's reports have lost a formatting convention that the memory held. What is the right way to recover?

    Sources542

    5.Communicating changes and scheduling the next review

    A fix nobody hears about tends to come undone. A colleague who doesn't know the 100-word rule was deliberately removed may add it back the following month. When you change a shared configuration, tell the team what changed, why, and which test confirmed it. In Claude Code, project CLAUDE.md files reach team members through source control, so the commit history records the change. Managed Agents offers no built-in approval step on agents.update, and any key in the workspace can call it. The review gate therefore sits on the pinned version number: anyone can create a new version, but changing the production pin goes through normal review.

    Drift keeps coming back, so schedule the review rather than waiting for complaints. Periodically review your instruction files, nested files and rules to remove outdated or conflicting instructions. Put the next review date where the team will see it. For higher-stakes setups, send a fraction of traffic to a new version and compare it with the current one before promoting it fully.

    Checkpoint 9 of 9· Check yourself

    Several people in your Managed Agents workspace hold API keys. The team wants prompt changes reviewed before production traffic sees them, the same way code changes are reviewed. Where should that review step sit?

    Sources25

    Sources

    (7)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “you can define project instructions for each project to further tailor Claude's responses”
      ↩︎ The four layers that can go stale
    2. 2.
      “Auto memory: notes Claude writes itself based on your corrections and preferences”
      ↩︎ The four layers that can go stale
      “if two instructions contradict each other, Claude may pick one arbitrarily”
      ↩︎ The four layers that can go stale
      “Claude makes the same mistake a second time”
      ↩︎ A diagnostic process that isolates the responsible layer
      “Longer files consume more context and reduce adherence.”
      ↩︎ Repairing instructions that have drifted
      “Organized sections are easier for Claude to follow than dense paragraphs.”
      ↩︎ Repairing instructions that have drifted
      “Every session (first 200 lines or 25KB)”
      ↩︎ Rolling back Skill versions and correcting memory
      “Review your CLAUDE.md files, nested CLAUDE.md files in subdirectories, and .claude/rules/ periodically to remove outdated or conflicting instructions.”
      ↩︎ Communicating changes and scheduling the next review
      “if two instructions contradict each other, Claude may pick one arbitrarily”
      ↩︎ Exam trap 1
      “Review your CLAUDE.md files, nested CLAUDE.md files in subdirectories, and .claude/rules/ periodically to remove outdated or conflicting instructions.”
      ↩︎ Exam trap 3
      “Review your CLAUDE.md files, nested CLAUDE.md files in subdirectories, and .claude/rules/ periodically to remove outdated or conflicting instructions.”
      ↩︎ Checkpoint
    3. 3.
      “uploads are copies, so a change you make on your computer afterward doesn’t reach the project until you upload the file again”
      ↩︎ The four layers that can go stale
    4. 4.
      “Each Managed Agents session starts with a fresh context by default.”
      ↩︎ A diagnostic process that isolates the responsible layer
      “Memory stores let the agent carry information across sessions: user preferences, project conventions, prior mistakes, and domain context.”
      ↩︎ Rolling back Skill versions and correcting memory
      “Use this for building review workflows, correcting bad memories, or seeding stores before any session runs.”
      ↩︎ Rolling back Skill versions and correcting memory
      “to roll back, retrieve the version you want and write its content back with client.beta.memory_stores.memories.update()”
      ↩︎ Rolling back Skill versions and correcting memory
      “Versions are retained for 30 days after they are written”
      ↩︎ Rolling back Skill versions and correcting memory
      “are not deleted when the memory itself is deleted”
      ↩︎ Exam trap 4
      “Versions belong to the store (not the individual memory) and are not deleted when the memory itself is deleted”
      ↩︎ Checkpoint
    5. 5.
      “pins to an exact version, which is what we want for a controlled comparison”
      ↩︎ A diagnostic process that isolates the responsible layer
      “Every agents.update produces a new immutable version, and sessions choose which version to use by ID.”
      ↩︎ Rolling back Skill versions and correcting memory
      “Billing regressed. Version 1 is still sitting on the server.”
      ↩︎ Rolling back Skill versions and correcting memory
      “Any key in the workspace can call it.”
      ↩︎ Communicating changes and scheduling the next review
      “Treat the pinned version number as the gate for changing prompts.”
      ↩︎ Communicating changes and scheduling the next review
      “route a fraction of traffic to a new version and compare”
      ↩︎ Communicating changes and scheduling the next review
      “pins to an exact version, which is what we want for a controlled comparison”
      ↩︎ Key concept
      “pins to an exact version, which is what we want for a controlled comparison”
      ↩︎ Exam trap 2
      “pins to an exact version, which is what we want for a controlled comparison.”
      ↩︎ Checkpoint
      “Treat the pinned version number as the gate for changing prompts. Creating versions is exploratory; updating the production pin is what goes through review.”
      ↩︎ Checkpoint
    6. 6.
      “Use /context, /doctor, /hooks, and /mcp to see what actually loaded.”
      ↩︎ A diagnostic process that isolates the responsible layer
      “or its description doesn’t match how you phrase the request”
      ↩︎ A diagnostic process that isolates the responsible layer
      “checked-in CLAUDE.md content Claude can derive from the codebase, with proposed fixes”
      ↩︎ Repairing instructions that have drifted
    7. 7.
      “Flag instructions written for older models in your prompts, skills, and tool descriptions and propose fixes as a diff”
      ↩︎ Repairing instructions that have drifted

    Ready to test yourself?

    Practise Anthropic Claude Certified Associate – Foundations (CCAO-F) - December 7 onwards in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.