What you will be able to do
- Place Claude-assisted work in the five stages of the development lifecycle
- Start eval-driven development with a small test set, add LLM-as-judge grading, and keep human review
- Pin production callers to an explicit prompt version and roll back by re-pinning
- Explain why stateful agents need gradual rollouts such as rainbow deployments
1.A lifecycle frame for Claude-powered systems
A Claude system goes through the same stages as any other software: you work out what to build, design it, build it, deploy it, and then support and scale it. Anthropic's support article on Claude Code uses this five-stage frame, and it describes Claude Code as a tool meant to help at every stage. The frame is useful beyond Claude Code because it shows where testing, deployment and monitoring belong. Those later stages are where LLM systems most often surprise their teams.
| Stage | Example tasks |
|---|---|
| 1. Discover | Explore codebase and history; search documentation |
| 2. Design | Plan project; develop tech specs; define architecture |
| 3. Build | Implement code; write and execute tests; create commits and PRs |
| 4. Deploy | Automate CI/CD; configure environments; manage deployments |
| 5. Support and scale | Debug errors; large-scale refactor; monitor usage and performance |
Checkpoint 1 of 6· Check yourself
In this lifecycle frame, which stage covers configuring environments and managing deployments?
The Deploy stage covers automating CI/CD, configuring environments and managing deployments. Build covers implementation, tests and PRs.
“It is designed to assist at every stage of the software development lifecycle”Source: support.claude.com
Sources1
2.Eval-driven iteration: start small, then scale the grading
In the Build stage, write and run tests. For an LLM system, that means evals you run every time you change a prompt or swap a model. Anthropic's research-agent team found that early changes have dramatic effects: a single prompt tweak might lift success rates from 30% to 80%. Differences that large show up in a handful of cases. So start evaluating straight away with a small set built from real usage patterns, and don't wait until you have hundreds of cases. The team says plainly that delaying evals for that reason is a mistake they often see.
As the eval set grows, grading becomes the bottleneck. Free-form outputs rarely have one correct answer, so the team used an LLM judge that scored each output against a rubric. The rubric covered factual accuracy, citation accuracy, completeness, source quality and tool efficiency. They tried several judges, one per criterion, but a single call with a single prompt that output 0.0–1.0 scores plus a pass/fail grade was the most consistent and agreed best with human judgement. Automation still doesn't replace people. Human testers noticed that early agents preferred SEO-optimised content farms to authoritative sources, and the evals had not caught it.
Checkpoint 2 of 6· Exam question
A fintech company runs automated compliance workflows against Claude Opus 4.5 and must guarantee that model behavior never changes between deployments without an explicit, reviewed migration step. When specifying the model in their Messages API calls, which practice should they adopt?
Correct answer: B — Specify the exact dated snapshot ID, and update it only during a deliberate, scheduled migration you control.
- A. Incorrect -- a convenience alias can silently resolve to a newer dated snapshot over time, which is exactly the unreviewed behavior change the compliance workflow needs to avoid.
- B. Correct -- pinning the exact dated snapshot ID keeps behavior fixed until the team deliberately migrates to a new snapshot on their own schedule.
- C. Incorrect -- there is no unpinned 'latest' pointer in the Messages API model field, and even if there were, it would introduce the same uncontrolled drift the team is trying to prevent.
- D. Incorrect -- the model parameter is required on Messages API requests; there is no default model selection to fall back on.
Checkpoint 3 of 6· Check yourself
A team's LLM-judge scores look good, but they plan to drop manual testing to save time. What does Anthropic's experience suggest?
Human testers caught a source-selection bias that the automated evals missed. Anthropic calls manual testing essential even alongside automated evaluation.
“Even in a world of automated evaluations, manual testing remains essential.”Source: www.anthropic.com
3.Prompts as versioned, pinned configuration
The API already makes callers pin what they depend on: every request must send an anthropic-version header. Prompts need the same treatment. In a typical codebase a prompt is a string in the source, so changing or reverting it takes a PR, a CI run and a deploy. Claude Managed Agents stores prompts on the server instead. Each agents.update creates a new immutable version under the same agent ID, and each session chooses which version it runs.
| What the caller passes | Which prompt version runs | Production suitability |
|---|---|---|
| Bare agent ID (agent=AGENT_ID) | Whatever the latest version is | Risky: a new version reaches callers on their next session, without review |
| {"type": "agent", "id": ..., "version": ...} | Exactly the pinned version | Recommended: new versions stay unused until the pin is changed |
The cookbook walks through a full cycle. It creates a ticket-triage agent, which comes back as version 1, and scores it on 20 labelled tickets. It then ships a v2 prompt that sends API-usage tickets to the platform team. Scored again with the pin set to v2, billing accuracy falls from 4/5 to 2/5, because billing tickets on a usage-billed product also mention API usage. Version 1 is still on the server, so rolling back means callers pass version: 1 again, and no rebuild is needed.
Checkpoint 4 of 6· Put it in order
Put the cookbook's prompt-change cycle in order
- 1.Update the system prompt, producing version 2
- 2.Roll back by pinning sessions to version 1
- 3.Score version 2 and detect the billing regression
- 4.Score version 1 against the labelled test set
- 5.Create the agent and receive version 1
Each update creates a new version. Pinned evaluations make the regression visible, and re-pinning the old version is the rollback.
“create v1, evaluate against a labelled test set, ship v2, detect a regression, roll back by pinning sessions to version 1.”Source: platform.claude.com
4.Change control and gradual rollout
Server-side versioning has a cost: any key in the workspace can call agents.update, and there is no built-in approval workflow. The review step comes back if production callers always pin an explicit version and that pinned number is put under change control. Anyone can create new versions, and they receive no traffic until promoted. Promotion is a config change that goes through normal review. Because of this, the cookbook says the version pin can work like a feature flag. For higher-stakes agents, send a fraction of traffic to the new version and compare results before you promote it fully.
Checkpoint 5 of 6· Check yourself
Under the cookbook's pattern, which action needs to go through your normal review process?
Creating versions is cheap and exploratory, because unpinned versions receive no production traffic. The production pin is the gate.
“Treat the pinned version number as the gate for changing prompts.”Source: platform.claude.com
Agents add a deployment problem that stateless services don't have. They are long-running, stateful combinations of prompts, tools and execution logic, so when you deploy an update, some agents are in the middle of a task. Anthropic's research system doesn't move every agent to the new version at once. It uses rainbow deployments: old and new versions run side by side while traffic shifts gradually. This is one reason Anthropic says the gap between a prototype and a production agent is often wider than teams expect. In agentic systems, errors compound, and one failed step can send an agent down an entirely different path.
Checkpoint 6 of 6· Check yourself
Why does Anthropic use rainbow deployments for its multi-agent research system?
Updating every agent at once could break sessions that are already running. Rainbow deployments keep both versions live and move traffic over gradually.
“we use rainbow deployments to avoid disrupting running agents, by gradually shifting traffic from old to new versions while keeping both running simultaneously.”Source: www.anthropic.com
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Evals are only useful once you have hundreds of test cases, so building them can wait.Why is that wrong?
Early changes have large effects, so a small set of about 20 real-usage queries already shows their impact. Start evaluating right away.
Covered in Eval-driven iteration: start small, then scale the grading
2.Production callers can safely pass the bare agent ID, because new prompt versions only take effect after review.Why is that wrong?
A bare ID resolves to the latest version, so any update reaches callers on their next session. Pin an explicit version and put that pin under change control.
Covered in Prompts as versioned, pinned configuration
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“It is designed to assist at every stage of the software development lifecycle”
↩︎ A lifecycle frame for Claude-powered systems“Write and execute tests”
↩︎ Eval-driven iteration: start small, then scale the grading - 2.
“Test your applications with newer models well before the retirement date of your current model.”
↩︎ Eval-driven iteration: start small, then scale the grading“Applications relying on Anthropic models may need occasional updates to keep working.”
↩︎ Change control and gradual rollout - 3.
“A prompt tweak might boost success rates from 30% to 80%.”
↩︎ Eval-driven iteration: start small, then scale the grading“a single LLM call with a single prompt outputting scores from 0.0-1.0 and a pass-fail grade was the most consistent”
↩︎ Eval-driven iteration: start small, then scale the grading“the gap between prototype and production is often wider than anticipated.”
↩︎ Change control and gradual rollout“AI developer teams delay creating evals because they believe that only large evals with hundreds of test cases are useful.”
↩︎ Exam trap 1“We started with a set of about 20 queries representing real usage patterns.”
↩︎ Prediction“Even in a world of automated evaluations, manual testing remains essential.”
↩︎ Checkpoint“we use rainbow deployments to avoid disrupting running agents, by gradually shifting traffic from old to new versions while keeping both running simultaneously.”
↩︎ Checkpoint - 4.https://platform.claude.com/docs/en/api/versioningOfficial docs
“When making API requests, you must send an anthropic-version request header.”
↩︎ Prompts as versioned, pinned configuration - 5.https://platform.claude.com/cookbook/managed-agents-cma-prompt-versioning-and-rollbackSecondary source
“Every agents.update produces a new immutable version, and sessions choose which version to use by ID.”
↩︎ Prompts as versioned, pinned configuration“Passing a plain string (agent=AGENT_ID) gets you whatever the latest version is.”
↩︎ Prompts as versioned, pinned configuration“production callers always pin to an explicit version, and that pinned number is the thing under change control.”
↩︎ Change control and gradual rollout“You can effectively use this versioning as a feature flag.”
↩︎ Change control and gradual rollout“Have production callers pin to an explicit version, not the bare agent ID.”
↩︎ Exam trap 2“create v1, evaluate against a labelled test set, ship v2, detect a regression, roll back by pinning sessions to version 1.”
↩︎ Checkpoint“Treat the pinned version number as the gate for changing prompts.”
↩︎ Checkpoint