CertSafari

    Free Anthropic Claude Certified Developer – Professional (CCDV-P) Sample Questions

    35 free sample questions from our bank of 441+, covering every exam domain, with answers and detailed explanations. Updated November 2026.

    1: Advanced Agent Architecture and Orchestration

    1.2: Durable Execution and State Management

    1.Driftwood Software runs a document-intake service for law firms, and its production pipeline includes a stateless, one-shot redaction agent written with the TypeScript Agent SDK. Each invocation receives a single contract, masks personal data, and returns the result, with no follow-up turns and no need to resume later. During a compliance review, the auditors inspected a worker after a batch finished and found conversation transcript files in the session directory on its disk, including prompts and tool results containing client text. The team already restricts the agent to read-only tools and starts every run with no earlier history, yet the files still appear. The review now requires that the run leave no conversation transcript on the worker's disk. Which approach meets this requirement?

    1. A.Set `persistSession` to `false`, so the run's history is held in memory and no file is written for it
    2. B.Set `forkSession` to `true`, so the run continues on a copy and the original session is removed from disk
    3. C.Limit `allowedTools` to read-only tools, so the run's tool calls and results are kept out of the transcript
    4. D.Set `continue` to `false`, so the run starts without prior history and is not saved to the session directory
    Show answer & explanation

    Correct answer: A — Set `persistSession` to `false`, so the run's history is held in memory and no file is written for it

    • A. With `persistSession` set to `false` the SDK keeps the session in memory only and writes no transcript file, which suits a stateless one-shot agent that must leave nothing on the worker's disk.
    • B. `forkSession` applies when resuming and creates a new persisted session with copied history. It does not delete the original, so transcript files remain on disk.
    • C. `allowedTools` limits what the agent may do, not what is recorded. Prompts, replies and tool results are still written to the transcript on disk.
    • D. `continue` only controls whether the most recent earlier session is picked up. The new run is still persisted and its transcript is written to disk.

    1.2: Durable Execution and State Management

    2.Fennel Travel runs a production trip-planning assistant on the Claude API that uses the memory tool so agents remember each traveler's preferences across separate sessions. Their data-protection officer is preparing a vendor risk review and must record where the memory files physically reside and which party sets how long they are kept and which travelers' files can be read by others. Engineers confirm that the application's request logs show Claude emitting file view, create and delete commands, and that the memory files still exist days later when a traveler returns in a brand-new conversation that shares no message history. No code execution tool is enabled in the deployment. Which statement correctly describes where the memory tool's data lives and who controls it?

    1. A.Your handler stores the memory files in a backend you choose, and your own code decides their retention and isolation
    2. B.Anthropic hosts the memory files in a store for your workspace, and its own policy decides their retention and isolation
    3. C.The API embeds the memory files in the message history, and your resent history decides their retention and isolation
    4. D.A code execution container holds the memory files, and the container's expiry and sandbox decide their retention and isolation
    Show answer & explanation

    Correct answer: A — Your handler stores the memory files in a backend you choose, and your own code decides their retention and isolation

    • A. The memory tool is client-side: Claude only requests file operations, and the application's handler executes them against a database, filesystem or other storage the developer picks. Retention and tenant isolation therefore depend on that handler.
    • B. Anthropic does not host a workspace-level memory store with its own retention policy. The memory tool writes nothing on Anthropic's servers; the developer's handler holds the files and sets how long they last and who can read them.
    • C. Memory files are not placed in the message history. They live in storage the handler manages, which is why they can persist across conversations that share no history and are not lost when a message list stops being resent.
    • D. The memory tool is unrelated to the code execution sandbox. Files are not kept in an Anthropic-managed container with its own expiry; the handler writes them to storage under the developer's control, so lifetime and isolation are the developer's to define.

    1.3: Architecture Selection at Scale

    3.Russet Insurance runs a production claims-support agent that exposes 40 tools in one flat list: policy lookup, claims intake, payments, and fraud screening account for roughly ten each. Over the past quarter its tool-selection error rate has risen to 12% as tools were added. The agent often picks a neighbouring tool with a similar name, such as a payment-status check instead of a claim-status check. Trace analysis shows the tools cluster into four distinct domains, and most tasks touch only one of them. The team has already confirmed that each tool description states its purpose and inputs. Wrong calls that reach payment tools have caused duplicate refunds, and finance requires per-call cost not to rise. Which change best addresses this?

    1. A.Split the 40 tools among four specialist agents by area, and add a router that sends each task to the matching agent.
    2. B.Move the agent to a larger model and keep the 40 tools in one shared list, relying on stronger reasoning to pick well.
    3. C.Rewrite the 40 tool descriptions with sharper boundaries, and add worked examples for each to the system prompt.
    4. D.Enable parallel tool calls so the agent can try several candidate tools at once, and keep the output that looks right.
    Show answer & explanation

    Correct answer: A — Split the 40 tools among four specialist agents by area, and add a router that sends each task to the matching agent.

    • A. Most tasks use tools from just one of four groups, so a router plus specialist agents gives each decision a short, focused tool list. Fewer competing choices means fewer selection errors, and the 12% rate can be re-measured afterwards.
    • B. A larger model still faces the same flat list of 40 tools, with the same overlap between neighbouring tools. The trace analysis shows most tasks need only one group, so the fix is to narrow the choice. It also raises cost on every call.
    • C. Better descriptions and examples still leave all 40 tools in front of the model on every task, although most tasks touch only one of the four domains. The choice stays as broad as before, and the prompt grows to hold the extra text.
    • D. Parallel calls execute several tools at once, so wrong choices still run and can cause side effects and extra cost. Judging an output by whether it looks right is unreliable, and the agent is no better at picking the right tool in the first place.

    1.3: Architecture Selection at Scale

    4.Yarrow Payments runs a content team that produces long-form merchant guides and compliance explainers through a drafting pipeline in production. Each piece is several paragraphs of prose with no single correct answer, and editors currently review every draft by hand. Leadership proposes adding an evaluator-optimizer loop, where a second model call critiques each draft and the generator then revises it. The model-call budget is fixed, and every extra revision round is billed against it. The editors have a written style and accuracy checklist, but nobody has compared a model's grading with theirs, and nobody has measured whether revised drafts beat first drafts. Which TWO conditions justify adopting that pattern? Select TWO.(Select 2)

    1. A.The team can state clear evaluation criteria that a model grader applies consistently, matching human reviewers.
    2. B.Measured quality improves across revision rounds on a held-out set, enough to offset the extra model calls.
    3. C.The output is a short factual answer with one known value, which a second model call can verify before release.
    4. D.The evaluator reuses the generator's prompt and context, which keeps both roles working from the same instructions.
    5. E.The pipeline runs as an overnight batch, so extra revision rounds add no delay that the business would notice.
    Show answer & explanation

    Correct answers: A, B — The team can state clear evaluation criteria that a model grader applies consistently, matching human reviewers.; Measured quality improves across revision rounds on a held-out set, enough to offset the extra model calls.

    • A. Clear, consistent criteria are what make the evaluator's feedback actionable and trustworthy. Grader agreement with humans is the check.
    • B. The pattern is only worth its added calls when iteration demonstrably raises quality. A held-out measurement shows that.
    • C. An output with one known correct value has nothing to iteratively improve. A direct comparison against that value settles it, so a generate-evaluate-revise loop adds model calls without raising quality.
    • D. An evaluator that shares the generator's prompt and context inherits its assumptions and blind spots, so it tends to approve what the generator produced. This is a design choice that weakens the loop, not a condition that justifies adopting it. A separate, rubric-driven evaluator is the usual design.
    • E. Tolerance for delay only removes one objection to the pattern. It does not show that revision rounds improve the drafts, and each extra round still costs model calls. A reason to adopt the loop needs evidence of a quality gain.

    1.1: Multi-Agent Orchestration

    5.Dune Telecom runs a research orchestrator in production that compiles nightly network-reliability briefings for its regional operations teams, unattended between 01:00 and 05:00. After switching to a model that delegates far more eagerly, the nightly run spawns subagents that spawn their own subagents. Spend per run tripled, and one run produced 60 concurrent workers. A system-prompt line already tells the orchestrator to delegate sparingly, and that same wording held on the previous model but failed on this one. The team must keep the new model because its briefings scored higher in quality review. Finance requires that no single run can exceed its agreed footprint, whatever the model decides to do, and nobody is awake to intervene. Which change bounds this most directly?

    1. A.Move the delegation guidance to the top of the system prompt with exact caps on subagent count and nesting levels, so the model sees the thresholds before it plans.
    2. B.Move the orchestrator to a smaller model tier with a lower per-token price, so each spawned subagent and each synthesis step costs less across the nightly run.
    3. C.Set caps on nesting depth, concurrent subagents and total query budget in the run configuration, so the runtime refuses spawns beyond those thresholds.
    4. D.Log Agent tool calls to a dashboard and page the on-call engineer when active subagents pass ten, so the run can be stopped before it grows further.
    Show answer & explanation

    Correct answer: C — Set caps on nesting depth, concurrent subagents and total query budget in the run configuration, so the runtime refuses spawns beyond those thresholds.

    • A. Incorrect. A more prominent prompt with numeric caps is still guidance the model may weigh against its own eagerness to delegate. The existing sparing-delegation line already failed to hold, so nothing here makes the limits binding.
    • B. Incorrect. A cheaper model lowers the price of each worker but puts no limit on how many are spawned or how deep they nest. A model that delegates eagerly can still fan out to dozens of workers, and quality drops on every task.
    • C. Correct. Depth, concurrency and budget limits set in the run configuration are enforced by the runtime itself. The tree of subagents stays bounded whatever the model chooses to delegate, which is the most direct control over the fan-out.
    • D. Incorrect. Monitoring and paging only detect the fan-out after workers are already running and spending, and they depend on a person responding in time during an unattended nightly run. Nothing in the system itself caps the number of subagents.

    1.1: Multi-Agent Orchestration

    6.Driftwood Health runs a claims-audit system in production in which an orchestrator dispatches eight subagents, each reviewing a different provider network. Every subagent returns a 12,000-token narrative of everything it examined, though traces show each report contains only a dozen or so actual findings buried in exploration notes. By the sixth return the orchestrator starts contradicting earlier findings and is near its context limit. Its output has never come close to its token ceiling. The compliance team requires that all eight sets of findings be reconciled in one final report and that underlying evidence remain retrievable for auditors. Which change best addresses this?

    1. A.Have subagents return a short structured summary of key findings with brief evidence, and write the bulky detail to files the orchestrator opens when needed.
    2. B.Have the orchestrator read only the opening paragraph of each worker report and skip the rest, keeping the earliest findings at the top of its window.
    3. C.Run the subagents in batches of two and reset the orchestrator's conversation after each batch, carrying forward only the user's request and the task plan.
    4. D.Raise the orchestrator's maximum output tokens, and have its final synthesis step restate each worker's findings in full before reconciling them in a single pass.
    Show answer & explanation

    Correct answer: A — Have subagents return a short structured summary of key findings with brief evidence, and write the bulky detail to files the orchestrator opens when needed.

    • A. Correct. Only each subagent's final message enters the orchestrator's context, so a compact report format keeps eight returns small. Detail written to files stays reachable when the orchestrator needs to check evidence.
    • B. Incorrect. Telling the model to skip text does not keep that text out of its context: every full report still arrives and uses up the remaining space. Findings that sit after the first paragraph are also lost, which makes inconsistent conclusions more likely.
    • C. Incorrect. Resetting the conversation between batches discards the findings from earlier batches. The orchestrator then cannot reconcile them with later results, so contradictions get worse. Only the request and plan survive, not the results.
    • D. Incorrect. The orchestrator is running out of room because eight long reports accumulate as input. A larger output allowance does nothing about that, and restating each report in full adds even more text to the same context.

    1.4: Context and Memory Architecture

    7.Umber Energy runs a coding agent that maintains its grid-forecasting codebase, on a model with a 1M-token window. Around turn 150 the agent starts ignoring earlier instructions and repeating edits it already made, although the session has used only 40% of the window. Most of the history is stale tool output: full test logs and file dumps from steps finished long ago. Latency and spend are acceptable, and the team does not want to switch models or split the work across multiple requests, because each step depends on the full task state. A teammate proposes preloading the entire documentation set at session start so the agent has more to draw on. Any given step needs only a few documentation pages. What is the best response to this proposal?

    1. A.Preload the documentation in a cached prompt prefix and enable prompt caching, keeping the large static block cheap and fast to reuse across turns.
    2. B.Prune stale tool outputs from the history and load documentation pages on demand through search or file reads, keeping the working context lean.
    3. C.Switch to a model with a larger context window and preload the documentation there, giving the history and the docs room to coexist across turns.
    4. D.Split the history across two parallel requests and fetch documentation pages on demand in each, merging the two partial answers afterward.
    Show answer & explanation

    Correct answer: B — Prune stale tool outputs from the history and load documentation pages on demand through search or file reads, keeping the working context lean.

    • A. Prompt caching reduces the cost and latency of repeating a large prefix, but the cached tokens still occupy the context. The agent already degrades with 40% of the window used, so a large preloaded block makes the problem worse.
    • B. Correct. Quality drops as tokens accumulate well before the window is full, so the fix is to curate the context. Dropping stale tool results and fetching documentation only when a step needs it keeps attention on the current task.
    • C. The agent degrades with well over half the window still free, so capacity is not the bottleneck. A larger window with the full documentation preloaded just lets even more tokens build up, and recall degrades the same way.
    • D. On-demand documentation is sound, but splitting the history is not. Each request sees only half of it, so neither holds the full task state, and the merged answer is patched together from partial views instead of a curated context.

    2: Production Integration, Reliability, and Knowledge Grounding

    2.1: Reliability Engineering for Claude Services

    8.Fjord Labs runs a nightly reporting job in production that asks Claude for a single 60k-token output through a non-streaming call, issued from behind the corporate NAT. About 30% of calls die after roughly 5 minutes with connection resets and no response, and each retry repeats the full cost. Raising the client timeout did not change the failure point, and the provider's status page shows no incidents. The same request run from a laptop outside the corporate network completes. The network team says the NAT's policy toward silent connections is fixed and will not change. Downstream consumers need the complete report at full length as one message, and the existing pipeline parses that message. What is the best fix?

    1. A.Set the client timeout to 30 minutes and cap retries at 2, with exponential backoff between each attempt.
    2. B.Lower max_tokens to 8k and instruct the model to write more concisely, with the nightly report cut down in length.
    3. C.Enable streaming on the request and read the final message from the SDK's stream helper, keeping max_tokens at 60k.
    4. D.Route the request to a second region and retry there on reset, keeping the same non-streaming call and 60k limit.
    Show answer & explanation

    Correct answer: C — Enable streaming on the request and read the final message from the SDK's stream helper, keeping max_tokens at 60k.

    • A. The resets come from the corporate NAT dropping a connection that stays silent for about 5 minutes, which happens regardless of the client's timeout. A longer timeout and fewer retries still leave each long non-streaming call exposed, so about 30% of runs would keep failing.
    • B. A shorter call might finish before the NAT drops it, but only by cutting the nightly report from 60k tokens to a fraction of its size. That removes the deliverable the job exists to produce and leaves the long-call problem unsolved.
    • C. Streaming sends data continuously, so the NAT never sees a silent connection to drop. The SDK's stream helper returns the same complete final message the job already expects, so the 60k output and the rest of the pipeline stay unchanged.
    • D. The connection is dropped by the caller's own corporate NAT after about 5 silent minutes, so the provider's region is not the cause. A second region would still receive the same silent non-streaming call and fail the same way, while retries repeat the full cost.

    2.2: Scale and Throughput Management

    9.Birch Telecom runs a customer-support assistant in production. A single FIFO queue feeds Claude for both live agent-assist requests and a nightly bulk reprocessing job over archived call transcripts, on the same model. During bulk runs, live requests wait about ten minutes. Queue metrics show items dequeue strictly in arrival order, so each live request sits behind thousands of batch items enqueued earlier. In a previous trial, raising the worker count left the wait unchanged and increased 429 responses. Shortening batch timeouts only sent abandoned items back into the same queue, with tokens spent twice. The bulk job has no deadline before morning but must still complete, while support agents need answers within seconds. Which change is best?

    1. A.Give interactive requests their own queue with reserved throughput, and cap batch concurrency at the remaining capacity.
    2. B.Add more workers to the existing queue, and retry with backoff on 429 responses to keep the backlog draining.
    3. C.Shorten client timeouts on batch calls, and re-queue the abandoned ones at the tail so interactive calls get slots sooner.
    4. D.Add client-side retries for interactive requests that time out, and spread them with jitter to avoid hitting the queue together.
    Show answer & explanation

    Correct answer: A — Give interactive requests their own queue with reserved throughput, and cap batch concurrency at the remaining capacity.

    • A. Correct. A separate path with reserved throughput means interactive requests no longer sit behind queued batch items, and capping batch concurrency keeps the batch job from consuming the capacity held back for them.
    • B. Extra workers on the same single queue still process items in arrival order, so interactive requests stay behind the batch backlog. More concurrent calls also raise 429 pressure on the shared model limit, making delays worse.
    • C. Shorter timeouts discard work and tokens already spent, and the re-queued batch items return to the same queue. Interactive requests still have no priority and still wait behind batch traffic.
    • D. Retries with jitter only change when requests re-enter the single queue. They still land behind the existing batch backlog, so waits stay long and the extra attempts add load.

    2.2: Scale and Throughput Management

    10.Cedar Travel runs a production fleet that uses Claude to summarize supplier contracts and rewrite itinerary notes for its booking platform. Many teams share one organization's limits, so spare capacity fluctuates through the day. The fleet runs with static concurrency of 64. It sees 429s whenever neighboring teams run big jobs, and the failed itineraries delay customer confirmations. In quiet periods, such as early mornings, dashboards show the fleet using only a fraction of the throughput the organization could give it while the contract backlog grows. Every response, including each 429, carries headers describing the remaining request and token room, plus a retry-after value. Engineers also note that the wording of the 429 message body has changed between API releases, and the neighbors' peak jobs keep getting larger each quarter. Which design is best?

    1. A.Adjust concurrency at runtime, cutting it on 429s and `retry-after` and raising it while rate-limit headers show room left.
    2. B.Pin concurrency at the lowest level that stayed error-free during neighbors' busiest jobs, then leave it fixed for the fleet.
    3. C.Raise concurrency from 64 to 128 and keep it there, so more requests are in flight while neighboring teams' jobs are quiet.
    4. D.Read the quota figure from the 429 error message at startup, then fix concurrency at that number for the whole run.
    Show answer & explanation

    Correct answer: A — Adjust concurrency at runtime, cutting it on 429s and `retry-after` and raising it while rate-limit headers show room left.

    • A. Correct. Rate-limit headers and `retry-after` report the current room left in the shared limits. Cutting concurrency on 429s and raising it when room appears lets the fleet follow the fluctuation instead of guessing a level.
    • B. A level sized to the neighbors' busiest period leaves the fleet idle whenever they are quiet. It stays fixed, so it cannot use spare capacity as it appears and gives no protection if the neighbors' peak grows.
    • C. A larger fixed pool sends even more requests exactly when neighbors run big jobs, so 429s become more frequent. Nothing makes the fleet back off, and the extra concurrency is only useful in quiet periods.
    • D. A number read once at startup is a snapshot of a limit that other teams' usage keeps moving during the day, so the fixed value goes stale. Error message text is also not a stable interface to build on.

    2.4: Retrieval and Knowledge Grounding at Scale

    11.Cedar Labs runs a support assistant for its logistics-software customers. For every query, the retrieval layer returns 40 chunks, and the application concatenates all of them into the prompt ahead of the user's question. Reviewers find that answers often omit facts that sit in the middle of the list, even though the retriever's relevance scores show those chunks among the best matches. In a test, the same fact was used reliably when its chunk was moved to the start or end of the list. Per-query token spend is also well above the finance team's target, and the budget is fixed. An earlier prompt instruction to address every passage in sequence lengthened the answers but changed neither the omissions nor the bill. Which two changes best address both problems? Select two.(Select 2)

    1. A.Rerank the 40 candidates and send only the top few passages, so the prompt carries the best-scoring material.
    2. B.Raise retrieval to 80 passages and switch to a model with a larger context window, so more relevant material is in the prompt.
    3. C.Add a system prompt instruction to work through the passages in order and reference each before answering, so none is skipped.
    4. D.Put the top-ranked passages first and last, with the question after the documents, so the best evidence is used more reliably.
    5. E.Re-chunk the corpus into 50-token passages and keep retrieving 40 per query, so the whole prompt gets shorter.
    Show answer & explanation

    Correct answers: A, D — Rerank the 40 candidates and send only the top few passages, so the prompt carries the best-scoring material.; Put the top-ranked passages first and last, with the question after the documents, so the best evidence is used more reliably.

    • A. Correct: reranking and sending only the top few passages shrinks the prompt, which lowers cost, and leaves less marginal text to bury the relevant facts.
    • B. Retrieving 80 passages makes the prompt longer and more expensive, and a larger context window does not stop facts in the middle of a long list from being overlooked. Both symptoms get worse.
    • C. An instruction to reference each passage does not shorten the prompt, so cost stays high. It also pushes the model toward padded answers instead of focusing on the strongest evidence.
    • D. Correct: models use information at the start and end of a long context more reliably than information in the middle, so placing the best passages there and the question after the documents reduces omissions.
    • E. Smaller passages cut some tokens, but they fragment evidence across more pieces. Retrieval count and ordering are unchanged, so relevant facts still end up in the middle and are still missed.

    2.3: Production MCP and Change Management

    12.Fennel Energy runs a customer support agent that handles billing disputes and outage reports for its residential utility customers, calling internal account and ticketing tools. The team has drafted a revised system prompt and will roll it out progressively. Last quarter, a prompt change that was ramped on a set calendar went unreviewed until full traffic, and a regression in tool-call formatting reached most customers before anyone noticed. Staff-only testing has also proven unrepresentative, since staff rarely phrase requests the way customers do. Fennel's incident review also found that users who were moved between prompt versions mid-conversation received contradictory instructions, and that the old and new groups were too uneven to compare. Which TWO controls most reduce risk during the progressive rollout? Select TWO.(Select 2)

    1. A.Ramp traffic from 5% to 25% to 50% to 100% on a fixed weekly calendar, reviewing dashboards once the final step completes.
    2. B.Gate each ramp step on tool-error rate, schema-validation failures and sampled task success, with the pipeline rolling back on a breach.
    3. C.Run the new prompt for internal staff for four weeks, then switch the full user base in one step after a clean review of the results.
    4. D.Deploy the new prompt to the full user base in the lowest-traffic window, keeping the previous version ready to restore by hand.
    5. E.Bucket users by a deterministic hash of user or session ID, holding each conversation on one prompt version and keeping cohorts comparable.
    Show answer & explanation

    Correct answers: B, E — Gate each ramp step on tool-error rate, schema-validation failures and sampled task success, with the pipeline rolling back on a breach.; Bucket users by a deterministic hash of user or session ID, holding each conversation on one prompt version and keeping cohorts comparable.

    • A. The ramp is graduated, but the steps advance on the calendar and dashboards are only reviewed after reaching full traffic. A regression at 5% or 25% keeps expanding until the end, so the exposure is not limited by measured quality.
    • B. Correct: each step must pass guard metrics (tool errors, schema-validation failures, sampled task success) before traffic grows, and a breach triggers a rollback without waiting for someone to notice. This caps the number of users a bad prompt can reach.
    • C. Staff traffic differs from real customer traffic in topics, phrasing and volume, so a clean staff period says little about customer behaviour. The final switch moves the whole user base at once, so there is no graduated exposure or early rollback signal.
    • D. A low-traffic window reduces how many users are online, but the new prompt still reaches the full user base at once and the next peak exposes everyone who comes back. Restoring by hand responds only after users have been affected, so the exposure is not staged.
    • E. Correct: a deterministic hash of the user or session ID assigns the same prompt version for the whole conversation, avoiding mid-conversation switches. It also keeps the old and new cohorts stable, so their metrics can be compared cleanly at each ramp step.

    2.3: Production MCP and Change Management

    13.Indigo Bank runs a customer-facing assistant that calls Claude through the Messages API in production. During development, engineers used Claude Code, where an internal account-lookup MCP server worked correctly through the project config file. The server is a local stdio binary. After the team registered it through the Messages API MCP connector, every production request fails. The connector logs show no connection to the server at all, even though the binary is installed on the application hosts. Once the connection problem is understood, the server's tools are also not visible to Claude, and the security team requires that credentials no longer come from developer-local files. Which TWO changes bring the production setup to parity? Select TWO.(Select 2)

    1. A.Copy the command and args entries from the project config into the request's mcp_servers definition, with the binary installed on the app host.
    2. B.Pass the server's authorization token in its mcp_servers entry and enable the needed tools with a toolset entry in the request's tools array.
    3. C.Deploy the server behind a public HTTPS URL with a remote transport such as Streamable HTTP, and point its connector entry at that URL.
    4. D.Raise the request's `max_tokens` and add a longer client timeout so the server's tool list can finish loading before generation starts.
    5. E.Wrap the stdio binary in a launcher shell script and reference it in the connector entry so the request starts it on the provider's side.
    Show answer & explanation

    Correct answers: B, C — Pass the server's authorization token in its mcp_servers entry and enable the needed tools with a toolset entry in the request's tools array.; Deploy the server behind a public HTTPS URL with a remote transport such as Streamable HTTP, and point its connector entry at that URL.

    • A. The connector's server definition takes a remote URL, not a command to spawn. A binary sitting on the application host is never launched by the API, so the same failure recurs.
    • B. Correct: the connector does not read the project config, so credentials must be supplied in the server entry and the tools Claude Code exposed locally must be enabled explicitly through a toolset in the request.
    • C. Correct: the connector calls the server from Anthropic's side, so the server must be reachable over the network through a remote transport instead of running as a local stdio process.
    • D. Output token limits and client timeouts do not affect whether the connector can reach or authenticate to a server. The failure comes from the stdio transport and missing connector configuration.
    • E. The connector never executes customer scripts or binaries on the provider's infrastructure. It only connects to a server that is already running at a remote URL, so a launcher script has nothing to start.

    2.4: Retrieval and Knowledge Grounding at Scale

    14.Yarrow Analytics runs a retrieval-augmented support assistant over its internal knowledge base of product documentation, and answer accuracy has stalled. The team ran an offline evaluation against labeled questions: the correct chunk appears somewhere in the top-100 retrieved results 96% of the time, but in the top-5 chunks actually passed to the model only 52% of the time. Answer accuracy tracks that 52% figure almost exactly. Inspection of failed cases shows the right chunk usually sits somewhere between rank 6 and rank 40, below passages that share vocabulary with the query but do not answer it. The current chunking is sound, and the assistant must keep its present response latency and per-request cost. Which change best targets the bottleneck?

    1. A.Swap in a larger embedding model and re-index the corpus, widening the candidate pool that first-stage retrieval returns.
    2. B.Increase the chunk size and re-index the corpus, so each passage carries more surrounding text and scores higher in similarity search.
    3. C.Add a cross-encoder reranker after retrieval, rescoring the candidates jointly with the query and sending its best five onward.
    4. D.Switch to a model with a larger context window and pass the top 100 chunks directly, keeping the retriever's current order.
    Show answer & explanation

    Correct answer: C — Add a cross-encoder reranker after retrieval, rescoring the candidates jointly with the query and sending its best five onward.

    • A. The correct chunk is already among the 100 retrieved results 96% of the time, so the candidate pool is not the weak point. The loss happens when candidates are ordered and cut to five, and widening the pool does not address that.
    • B. Larger chunks tend to dilute the embedding signal and change what gets retrieved. The correct chunk is already retrieved 96% of the time, and the gap is in how candidates are ranked, so re-chunking does not target it.
    • C. Correct. Recall at 100 is high, so the correct chunk is usually retrieved but ranked too low. A cross-encoder scores each query-passage pair together and can move the right chunk into the five the model sees.
    • D. Sending 100 chunks raises cost and latency and buries the relevant text among distractors. It avoids the ranking problem instead of fixing it, and the retriever's order stays as poor as before.

    3: Observability and Production Monitoring

    3.1: Telemetry and Instrumentation

    15.Juniper Pharma runs a production streaming chat assistant that lets pharmacists ask questions about internal drug-handling procedures. Support tickets say answers feel slow, and several pharmacists report staring at a blank reply area before any text shows up. Yet the team's dashboards show a healthy p50 total request duration of 4 s, the gateway is lightly loaded, error rates are flat, and the same median holds for short and long answers alike. The front end renders each streamed chunk as it arrives. The team can add one field to the existing per-request telemetry events without changing the architecture. Which instrumentation addition best locates the perceived delay?

    1. A.Record time to first token for each request, along with total request duration and output token count.
    2. B.Record concurrent gateway connections for each request, along with total request duration and error rate.
    3. C.Record response length in characters for each request, along with total request duration and prompt template name.
    4. D.Record client-side render time of the final message for each request, along with total request duration and browser type.
    Show answer & explanation

    Correct answer: A — Record time to first token for each request, along with total request duration and output token count.

    • A. On a streamed response, the wait before the first token is what users feel as sluggishness, and a 4 s median for the whole request hides it. Measuring it separately shows how long users stare at an empty reply.
    • B. Concurrent connections show how loaded the gateway is, but not which phase of a request the user waits in. Even if load were high, the delay before output starts would remain unmeasured.
    • C. Response length helps explain why some requests take longer overall, but it is only known once generation finishes. It cannot reveal how long users waited before any text appeared.
    • D. Rendering the final message happens after the stream has ended, so it measures a phase that comes after the wait users complain about. It cannot show how long they waited for output to begin.

    3.2: Production Monitoring and Alerting

    16.Basalt Labs runs a customer-support summarization service on the Claude API, and its finance team keeps a dashboard that estimates spend by multiplying the sum of input tokens and output tokens by list price. The dashboard shows $4.1k per day, while the organization's usage and cost reports show $5.6k. The service relies heavily on prompt caching, with a long shared system prompt reused across most requests. Finance notes that the gap is widest on days with high cache hit rates and shrinks on days after a prompt change. The usage reports list cache creation tokens and cache read tokens as separate lines, each charged at a different per-token rate from the other and from standard input. Finance needs the dashboard to be reliable for daily chargeback to internal teams. What is the best fix?

    1. A.Apply a 1.37 multiplier to the token total, recalibrated against the usage reports at the end of each calendar month
    2. B.Price uncached input, cache writes, cache reads and output at their own rates, reconciling against the usage reports daily
    3. C.Leave cache creation and read tokens out of the total, reconciling against the usage reports once each week
    4. D.Price cache writes and reads at the standard input rate, and output at the list output rate, checking daily against reports
    Show answer & explanation

    Correct answer: B — Price uncached input, cache writes, cache reads and output at their own rates, reconciling against the usage reports daily

    • A. A single multiplier hides the pricing error instead of fixing it. The size of the gap moves with cache hit rate and traffic mix, so a factor recalibrated monthly would be wrong on most days in between.
    • B. Cache writes, cache reads, uncached input and output are billed at different rates. Pricing each category separately and checking against the organization's usage and cost reports makes the dashboard match what is actually charged.
    • C. Cache creation and read tokens are billed, and with heavy caching they are a large share of usage. Leaving them out moves the dashboard further from the reports, and a weekly check only shows the gap after the fact.
    • D. Cache writes are billed at a premium over the standard input rate and cache reads at a steep discount. Pricing both at the standard rate misstates cost in each direction, so the dashboard still diverges from the reports.

    3.2: Production Monitoring and Alerting

    17.Granite Payments runs a dispute-resolution assistant built on Claude for cardholders in several countries. Customers routinely paste account numbers, addresses and transaction details into their messages, so the prompts contain customer PII. Compliance has ruled that conversation content may not be stored in the company's shared observability index or metrics backend, whatever the encryption, access controls or retention period; a previous audit flagged a pilot that did exactly that. The platform team still needs cost, latency and truncation visibility across all traffic, and the quality team needs some real conversations to judge whether answers are accurate and on-policy. Many engineers outside the approved group use the shared dashboards. Which two designs satisfy both the storage restriction and the monitoring needs? Select TWO.(Select 2)

    1. A.Send full prompts and responses to the shared metrics backend, with role-based access limiting who can open the dashboards
    2. B.Replace user ids with salted hashes, then log full prompt text to the shared index with a 30-day retention window
    3. C.Emit token counts, model id, `stop_reason`, latency and prompt version per request, and keep prompt and response text out of telemetry
    4. D.Encrypt prompt and response logs with a customer-managed key, then store them in the shared index for the dashboards
    5. E.Sample a small share of conversations into a dedicated store, redacting before write, with short retention and access logs
    Show answer & explanation

    Correct answers: C, E — Emit token counts, model id, `stop_reason`, latency and prompt version per request, and keep prompt and response text out of telemetry; Sample a small share of conversations into a dedicated store, redacting before write, with short retention and access logs

    • A. Role-based dashboard access limits who can view the data, but the full prompts, including PII, would still be stored in a backend that compliance does not allow to hold content.
    • B. The PII is in the prompt text, not only in the user id, so hashing ids leaves personal data in the shared index. A 30-day retention window does not change where the content is stored.
    • C. Cost, latency and stop-reason monitoring work from per-request metadata, so no content needs to leave the service and the storage restriction is respected.
    • D. Encryption at rest protects the bytes, but the prompts and responses are still stored in a shared location that compliance does not permit for this content.
    • E. Quality review needs some real conversations. A small redacted sample in a dedicated, access-logged, short-retention store provides them while keeping content out of general telemetry.

    3.3: Trace Analysis at Scale

    18.Kestrel Software runs a fleet of customer-support agents in production, built on Claude with tool access. Trace logs record every turn, tool call with its arguments, result size and stop reason. Agents average 7 turns per run, but a few percent run past 40 turns. Engineers who read a handful of long traces noticed that the opening turns usually look like distinct, sensible calls, and the trouble appears later. The weekly mean has stayed near 7 even while tickets about stuck agents climbed, because the typical short runs dominate it. The platform team cannot change any production settings this quarter, so the work must use the existing traces. Product wants to find loops in the traces and learn how widespread they are. Select TWO analyses that most directly quantify looping.(Select 2)

    1. A.Count repeated tool calls with the same name and normalized arguments per trace, then plot the spread of repeat counts across runs.
    2. B.Raise the turn cap from 50 to 100, then watch whether the mean turns per run moves once long runs have room to finish.
    3. C.Split runs at the 95th percentile of turn count, then compare the long and typical groups on tool mix, result sizes and stop reason.
    4. D.Review the first five turns of each run for repeated tool calls, then extrapolate that repeat rate to the whole fleet.
    5. E.Track the fleet-wide mean turns per run by week, then flag weeks above 8 turns and read the size of each rise as loop prevalence.
    Show answer & explanation

    Correct answers: A, C — Count repeated tool calls with the same name and normalized arguments per trace, then plot the spread of repeat counts across runs.; Split runs at the 95th percentile of turn count, then compare the long and typical groups on tool mix, result sizes and stop reason.

    • A. Correct: a tool called repeatedly with the same normalized arguments inside one trace is the direct signature of a run that is stuck, and the distribution of repeat counts shows how widespread that behavior is.
    • B. Incorrect: a higher cap is a production change rather than a trace analysis, and the mean turns per run reflects task length as much as repetition. Looping runs would simply continue to the new cap without being identified or counted.
    • C. Correct: contrasting the tail above the 95th percentile with typical runs on tool mix, result sizes and stop reason shows what separates the long runs, which turns a handful of outliers into a measurable pattern.
    • D. Incorrect: runs that go past 40 turns are mostly still making distinct calls in their first five turns, so the sample misses the repetition and the extrapolated rate understates it.
    • E. Incorrect: a fleet-wide mean is dominated by the typical short runs, so it cannot show which runs repeat calls or how many there are. A rise could equally come from legitimately longer tasks.

    4: Advanced Debugging and Incident Response

    4.1: Complex Failure Diagnosis

    19.Alder Labs runs a claims-triage assistant in production. Last week one release bundled three changes: a rewritten system prompt, an update to the model alias the service points at, and a modified schema for the tool that looks up policy records. Since then, quality has dropped. The team keeps a fixed eval set of labelled claims that it has scored on every earlier build, and the harness can run any build against it on demand. Production traffic this month differs from last month's, with a seasonal mix of claim types, and the team does not trust that period-to-period comparisons are like for like. A rollback has already been proposed, but the engineers also need to know which change to fix going forward, and they want to avoid reopening the investigation later. What is the most efficient way to find the cause?

    1. A.Roll back the whole release, re-run the eval set on the previous build to confirm scores recover, and record the rollback as the root cause.
    2. B.Score a sample of production traces from before and after the release with an LLM judge, then match the failure patterns to one of the three changes.
    3. C.Revert only the prompt rewrite, re-run the eval set on that build, and if scores recover, treat the prompt as the cause and close the investigation.
    4. D.Re-run the eval set on builds that revert each of the three changes separately, keeping the other two in place, then compare the scores per build.
    Show answer & explanation

    Correct answer: D — Re-run the eval set on builds that revert each of the three changes separately, keeping the other two in place, then compare the scores per build.

    • A. Incorrect: a rollback confirms that something in the release caused the drop, but the prompt, model alias and tool schema all went out together. Recovery on the old build says nothing about which of the three is responsible, so the real cause stays unknown.
    • B. Incorrect: production traffic differs between the two periods, and an LLM judge's scores on it cannot separate three changes that shipped together. Matching failure patterns to a change is inference, not a controlled comparison, and the fixed eval set that would give one is left unused.
    • C. Incorrect: this tests one of the three suspects and stops. Scores recovering after that single revert does not rule out the model alias update or the tool schema change, and any interaction between the changes is never examined.
    • D. Correct: running the fixed eval set once per build, with a single change reverted each time and the other two held in place, isolates each change's effect on the score. Comparing the results attributes the regression to a specific change with measured evidence, at the cost of only three runs.

    4.2: Incident Response and Learning Loops

    20.Birch Learning runs a production summarization service that calls a model through a floating alias. Overnight, summaries changed style and the downstream JSON parser, which feeds the course-catalog pipeline, began rejecting most responses. Version control shows no code or prompt deploy for three weeks. Response metadata now reports a different resolved model identifier than the one logged yesterday. The dated identifier that served the service until then is still available, and it passed the team's stored evaluation set the last time that set was run. Nobody evaluated the new model before it began serving traffic, and the catalog team needs valid output restored before the morning ingest. Which response is the best mitigation and lesson?

    1. A.Point the service at the last dated snapshot that produced valid output, then require an eval run before promoting a new model version.
    2. B.Add a repair-and-retry layer in front of the parser, then keep tracking the alias and read the provider's release notes before each move.
    3. C.Redeploy the previous code and prompt release, then add a nightly regression run against the live alias to flag future drift.
    4. D.Set temperature to 0 on the summarizer calls, then re-run the parser tests against the alias each month to catch format drift.
    Show answer & explanation

    Correct answer: A — Point the service at the last dated snapshot that produced valid output, then require an eval run before promoting a new model version.

    • A. Correct: the failure came from the alias resolving to a different model, so pinning to the last working dated snapshot restores the old behaviour immediately. Requiring evals before any model version change keeps a silent upgrade from reaching production again.
    • B. Incorrect: a repair layer masks the symptom at one consumer, but the service still follows a moving alias and keeps running untested models. Release notes do not help when the alias moves overnight, and the old behaviour is never restored.
    • C. Incorrect: no code or prompt deploy happened, so redeploying the earlier release changes nothing; the model behind the alias is what changed. A nightly regression run detects drift only after production has already seen it.
    • D. Incorrect: temperature controls sampling randomness within a model and cannot recreate the behaviour of an earlier model version. At 0 the service would produce stable output from the new model, and a monthly test finds drift long after it has caused failures.

    5: Evaluation Engineering and Quality Assurance

    5.1: Evaluation System Design

    21.Pebble Robotics runs a RAG assistant that answers employees' internal policy questions, such as expense limits and leave rules, from a vector index of about 3,000 policy documents. Its eval suite of 400 labeled questions shows wrong answers on 14% of items. Reviewing a sample of failures, the team finds some answers contradict the policy and others are fluent but cite nothing the policy says. Nobody has yet inspected which passages were returned for the failing questions. The generator already runs on the largest model the budget allows, and the team plans to rewrite the generation prompt. The eval set includes a labeled list of the policy passages that contain each correct answer. Which step should come first to localise the failure?

    1. A.Increase `max_tokens` and enable extended thinking so the model has room to reconcile conflicting passages before it answers each question.
    2. B.Score the retriever alone against labeled relevant passages, using recall@k and precision@k, to see whether the evidence was ever retrieved.
    3. C.Switch the generator to a larger model and rerun the suite, since a stronger model is more likely to ignore irrelevant retrieved passages.
    4. D.Add an instruction telling the model to answer only from provided passages and to quote them, then compare accuracy before and after the change.
    Show answer & explanation

    Correct answer: B — Score the retriever alone against labeled relevant passages, using recall@k and precision@k, to see whether the evidence was ever retrieved.

    • A. This assumes the model has the right passages and fails to reason over them, which has not been shown. Extra reasoning budget cannot recover evidence that was never retrieved.
    • B. Retrieval and generation fail for different reasons and need different fixes. Measuring retrieval against labeled relevant passages shows whether the answer was reachable before any prompt is rewritten.
    • C. A model swap changes cost and latency and confounds the diagnosis. If the right passage is missing, a larger model will still answer wrongly or hallucinate.
    • D. This is a generation-side change made before knowing whether retrieval is the cause. A before-and-after comparison cannot separate the two failure stages.

    5.1: Evaluation System Design

    22.Indigo Freight runs a customer support bot that answers shipment-delay and customs-hold tickets. To improve the prompt, an engineer copied the best-performing few-shot examples directly from the team's golden set, the same cases that CI scores on every prompt change. Over the next release, the golden-set pass rate rose from 81% to 96%. Meanwhile, the weekly review of live tickets by human agents shows resolution quality is flat, and the cases the bot fails in production look unlike anything in the golden set. The team lead wants a change that makes the CI score a trustworthy signal of real resolution quality again, not just a gate that is easier or harder to pass. What is the best fix to restore the suite's validity?

    1. A.Draw the few-shot examples from a separate pool, and run the CI suite on a reserved split that prompt authors never see, rotating it over time.
    2. B.Raise the golden-set pass threshold from 90% to 98%, and require two consecutive passing runs before a prompt version can be shipped.
    3. C.Regenerate the suite with Claude as a larger synthetic set written from the current prompt, and rewrite its cases each release cycle.
    4. D.Add an LLM judge that scores helpfulness on each response, calibrate it against human-rated tickets, and gate releases on its score.
    Show answer & explanation

    Correct answer: A — Draw the few-shot examples from a separate pool, and run the CI suite on a reserved split that prompt authors never see, rotating it over time.

    • A. Keeping the prompt's example pool disjoint from the scored cases removes the leakage, so the score measures generalisation again. Rotating the reserved split limits drift as authors iterate against it.
    • B. The few-shot examples are still copied from the scored cases, so the prompt keeps matching them from memory. A higher bar and repeated runs make the gate harder to pass without making the score reflect real resolution quality.
    • C. Cases written from the current prompt inherit its embedded examples, wording and assumptions, so the scored data still depends on the prompt. Fresh text and extra volume do not make the set independent.
    • D. The prompt has still seen the examples it is scored on, so overlap inflates results whatever grades them. Calibrating the judge improves grading accuracy but leaves the contaminated cases in the suite.

    5.2: LLM-as-Judge Engineering

    23.Umber Games runs an LLM judge that passes or fails player-support replies before they reach customers. It was validated at kappa 0.78 against human labels at launch and has run for six months, during which the product added several intents, such as subscription disputes and parental-control requests, that the launch set never contained. The judge's weekly mean score looks flat, and an earlier trial of a second LLM judge showed it agreeing with the first almost always, even on replies humans disputed. Support leads still report more borderline replies being wrongly passed. Reviewers can label only a small batch each week, and the team expects to revise the judge prompt or switch model versions occasionally. Which two practices best detect silent degradation? Select TWO.(Select 2)

    1. A.Sample production items on a schedule, stratified by intent with near-threshold cases oversampled, and get fresh human labels to track agreement.
    2. B.Keep a frozen golden set, rerun the judge on it after each prompt or model change, and alert when its agreement with the labels shifts.
    3. C.Relabel the original launch set each quarter, rerun the judge against the refreshed labels, and track the agreement trend across quarters.
    4. D.Add a second LLM judge, have it re-score a weekly sample of the pipeline's verdicts, and track how often the two judges agree as the quality signal.
    5. E.Track the judge's weekly mean score by intent, alert when it drifts from the launch baseline, and treat a stable mean as steady agreement with humans.
    Show answer & explanation

    Correct answers: A, B — Sample production items on a schedule, stratified by intent with near-threshold cases oversampled, and get fresh human labels to track agreement.; Keep a frozen golden set, rerun the judge on it after each prompt or model change, and alert when its agreement with the labels shifts.

    • A. Fresh human labels on a sample of current traffic, including the new intents, directly measure how well the judge agrees with people today. Stratifying by intent and oversampling near-threshold items concentrates effort where verdicts are most likely to flip.
    • B. A frozen labelled set keeps the data and the reference labels constant. A change in agreement after a prompt or model change can therefore be attributed to the judge itself rather than to shifting inputs.
    • C. The launch set contains only the intents that existed at launch, so rescoring it cannot show degradation on the newer intents in current traffic. Relabelling it each quarter is also costly and adds nothing about the live distribution.
    • D. Agreement between two LLM judges measures how similar the models are, not how well either matches human labels. Models can share the same biases and agree while both drift away from human ground truth.
    • E. A stable weekly mean can hide offsetting errors, and it can hide new intents being scored wrongly. Score levels carry no information about per-item agreement with human reviewers, so a steady mean does not show agreement is unchanged.

    5.3: Online Quality Measurement

    24.Nettle Housing runs a customer-support agent for tenants that can issue rent refunds and other account credits through write tools against its production billing API. A team wants to compare a candidate model for this agent on live traffic with zero user impact. Leadership has stated that no tenant may be served by the candidate or have their account touched by it, and that this holds throughout the evaluation. The comparison must reflect this week's real tenant requests, because the mix of maintenance-fee disputes changed recently. In earlier offline tests the candidate sometimes ignored restrictions written in its system prompt, and last month's archived conversations no longer resemble current inbound requests. Which design satisfies the requirement?

    1. A.Mirror live requests to the candidate in a shadow deployment, replace refund and other write tools with recording stubs, then compare offline.
    2. B.Route a small share of live users to the candidate with refunds capped at a low amount, then compare outcomes against the incumbent offline.
    3. C.Replay last month's logged conversations against the candidate, point refund tools at the production API, then compare outcomes offline.
    4. D.Mirror live requests to the candidate in a shadow deployment, keep real tools but forbid refunds in its prompt, then compare offline.
    Show answer & explanation

    Correct answer: A — Mirror live requests to the candidate in a shadow deployment, replace refund and other write tools with recording stubs, then compare offline.

    • A. Mirrored traffic gives realistic current inputs, and recording stubs for write tools make side effects impossible by construction, so users are never affected.
    • B. This is a live canary: the sampled users are served by the unvalidated model and can receive real refunds, even if capped, so impact is not zero.
    • C. Pointing refund tools at the production API means the replay can issue real refunds against customer accounts, and month-old logs are not current live traffic.
    • D. A prompt rule is an instruction, not an enforced control. With real tools attached, one ignored instruction can trigger a real refund, so zero impact is not guaranteed.

    5.3: Online Quality Measurement

    25.Vireo Labs runs an email-drafting feature inside its customer-success platform. Account managers type a short instruction, such as a renewal nudge or an apology for a delayed shipment, and the model returns a full draft. Reviewers who read sampled production drafts report that several quite different emails are acceptable for the same instruction, so no single reference answer exists for any request. They also note that the worst drafts often read smoothly but get the customer's order details wrong, use a tone that does not suit the situation, or skip part of the instruction. The team wants a production quality signal that tracks these reviewer judgments over time on sampled live traffic. Which online quality metric is best matched to this task type?

    1. A.ROUGE-L overlap of sampled drafts against a reviewer-written reference email, with the threshold tuned on reviewer-rated drafts
    2. B.Embedding similarity of sampled drafts against a stored golden email for the template, with the score checked against reviewer ratings
    3. C.LLM judge scoring sampled drafts on a rubric for tone, faithfulness and instruction following, with the judge calibrated against human ratings
    4. D.Mean perplexity of sampled drafts under the serving model, used as the readability score and checked against reviewer ratings
    Show answer & explanation

    Correct answer: C — LLM judge scoring sampled drafts on a rubric for tone, faithfulness and instruction following, with the judge calibrated against human ratings

    • A. Many different emails are valid for one request, so lexical overlap with a single reference scores good drafts low when they are worded differently. Tuning the threshold on reviewer-rated drafts does not remove that single-reference dependence.
    • B. Embedding similarity to one golden email still treats that email as the target. Valid drafts with different content or tone drift away from it and are penalized, and checking against reviewer ratings does not fix the metric's single-reference design.
    • C. Open-ended drafts have no reference answer to compare against, so they need reference-free grading across several quality dimensions. Calibrating the judge against human ratings keeps its scores trustworthy.
    • D. Perplexity measures how predictable the text is to the serving model. A fluent draft that is inaccurate, off-tone or ignores instructions can still score well, so it is a poor proxy for quality even when compared with reviewer ratings.

    6: Cost and Latency Optimization at Scale

    6.2: Latency Engineering

    26.Driftwood Analytics runs a document-processing agent in production for its insurance-claims clients. The workflow is fixed: classify the document, extract fields, validate against a schema, then format output. The agent performs this as six model turns with tool calls between them, taking 18 s end to end. Traces show the step order is identical on every document, and four of the six turns do nothing but pass a tool result forward with no decision to make. Each turn spends most of its time re-reading the growing conversation and generating a short, predictable message. The team already turned on streaming, and users see text sooner, but total completion time did not change. Raising reasoning depth on a trial turn only made that turn slower. What change would best cut latency?

    1. A.Enable streaming on each of the six model turns so partial output reaches the user sooner and the workflow finishes below 18 s
    2. B.Add a prompt-caching breakpoint after each model turn so later turns reuse the earlier conversation as a cached prefix
    3. C.Move the routine steps into application code or one composite tool so the model is called only for steps needing judgement
    4. D.Set the effort parameter to high on each turn so the model plans the whole workflow upfront and needs fewer tool calls
    Show answer & explanation

    Correct answer: C — Move the routine steps into application code or one composite tool so the model is called only for steps needing judgement

    • A. Streaming only makes output appear earlier. The six model turns and the tool calls between them still run one after another, so the workflow's completion time stays near 18 s.
    • B. Caching saves reprocessing of the repeated prefix, but all six round trips and their generation time still occur. Only removing model turns for fixed steps cuts the main cost.
    • C. Because the step order is fixed, code or a composite tool can run those steps without a model round trip. That removes repeated input processing and generation across the six turns, so the model is used only where judgement matters.
    • D. Higher effort makes each turn spend more time reasoning. The fixed steps still run as separate turns and tool calls, so total latency rises or stays flat instead of dropping.

    6.1: Cost Architecture at Scale

    27.Driftwood Pharma runs a production reasoning agent that answers questions from clinical operations staff. Its extended-thinking budget is set high on every request, and the median output is 6,000 tokens, including thinking billed as output. A cost review found that a large share of traffic is simple lookups, such as confirming a batch's storage condition, which need a one-line answer yet still produce long reasoning traces. Interactive users expect responses within seconds, so the team will not move this traffic to asynchronous processing. The same agent also handles multi-step protocol-deviation analyses, where reviewers have rated answers highly and say that shorter reasoning would hurt accuracy. Which change most reduces fleet cost without cutting quality on hard tasks?

    1. A.Route requests by difficulty, giving lookups a small `budget_tokens` and keeping a large one for complex reasoning.
    2. B.Lower `max_tokens` to 1,000 on the reasoning agent, sized to the final answer that a typical request produces.
    3. C.Add a system prompt instruction to think briefly and avoid long reasoning chains, with the thinking budget left as is.
    4. D.Move the agent's requests to the Message Batches API, where thinking tokens are billed at the discounted batch rate.
    Show answer & explanation

    Correct answer: A — Route requests by difficulty, giving lookups a small `budget_tokens` and keeping a large one for complex reasoning.

    • A. Correct. Thinking tokens are billed as output, so a small `budget_tokens` on lookups removes most of their spend, while complex reasoning keeps its large budget and its quality.
    • B. `max_tokens` covers thinking plus the final answer, so a 1,000-token cap truncates hard tasks mid-reasoning. It lowers cost by cutting quality exactly where the large budget is needed.
    • C. A prompt instruction is a soft hint, not an enforced limit on thinking spend, and it also pushes hard tasks toward shallow reasoning. The configured budget stays in place, so cost is not reliably bounded.
    • D. Batch pricing lowers the per-token rate but leaves the 6,000-token median untouched, so lookups still burn thinking tokens, and results arrive asynchronously. Routing by difficulty removes far more spend.

    7: Runtime Guardrails and Production Security

    7.1: Runtime Guardrail Systems

    28.Willow Utilities runs an internal agent in production that reads wiki pages, drafts a weekly summary, and can post to one Slack channel seen by 40 employees. Any employee can edit the wiki, and some pages contain pasted customer account details or text of unknown origin. A weekly run makes about 30 read calls and a single Slack post, which all 40 people see the moment it is sent. Security policy requires that controls on tool use be enforced outside the model, not left to instructions. The proposal adds an input classifier, approval on every call, a sandbox, and a PII redactor. In the pilot, reviewers called it unusable because they had to approve dozens of routine prompts per run. Which design fits the risk profile?

    1. A.Keep the input classifier and PII redactor, let the agent post summaries to Slack directly, and review the channel after posting.
    2. B.Keep the four proposed layers, run the input classifier asynchronously after each model response, and shorten the approval prompts.
    3. C.Auto-allow read-only tools inside a network-restricted sandbox, redact PII from drafts, and require approval for the Slack post alone.
    4. D.Replace the layers with one system prompt covering PII, channel rules and tool limits, and test it on twenty adversarial examples.
    Show answer & explanation

    Correct answer: C — Auto-allow read-only tools inside a network-restricted sandbox, redact PII from drafts, and require approval for the Slack post alone.

    • A. The Slack post is the agent's one outward action and 40 employees see it immediately. Reviewing the channel afterward only finds a bad summary once it is visible, so that action needs a gate before it executes.
    • B. A classifier that runs after the model has responded cannot stop a malicious page from influencing the agent. Keeping the other layers also leaves approvals in place on routine read-only steps, so the friction reviewers complained about remains.
    • C. Correct. Read-only wiki access is low risk and runs without prompts inside a restricted sandbox. PII redaction covers the drafts, and human approval is reserved for the one action that reaches the 40-person channel, so safety and usability are both met.
    • D. A system prompt is guidance the model may not follow, not an enforced control. Twenty examples cannot show that PII or channel rules hold under adversarial wiki content, so the post goes out with no technical safeguard.

    7.2: Threat Defense and Adversarial Testing for Agentic Systems

    29.Birch Payments runs an internal chat assistant that summarizes contracts and finance memos for its staff, and the web UI renders the model's replies as markdown. During a review, the security team found that a poisoned vendor document had made the model emit an image link pointing to an attacker-controlled host, with a base64 copy of the summary in the query string. The employee's browser fetched the image automatically while rendering the reply, and the request appears in the browser's network log. The assistant had no tool call in that session, and its service already runs with no web-fetch tools. The team has seen that injected instructions in documents can still steer the model's wording, and expects the attacker to vary how the data is encoded. Which control most directly closes this exfiltration path?

    1. A.Add a system-prompt rule telling the model to reply in plain text and never emit markdown image syntax, and repeat it after each document.
    2. B.Set a Content-Security-Policy img-src limited to first-party hosts and make the markdown renderer drop image URLs pointing to other domains.
    3. C.Add a response filter that blocks rendering of replies with long base64-style query strings in links and logs each one for the security team.
    4. D.Remove the web-fetch and browsing tools from the assistant and block outbound network calls from the summarization service to external hosts.
    Show answer & explanation

    Correct answer: B — Set a Content-Security-Policy img-src limited to first-party hosts and make the markdown renderer drop image URLs pointing to other domains.

    • A. Incorrect: a system-prompt rule is a request to the model, and the poisoned document already shows that injected text can steer its output. If an image link still appears, the browser loads it, so nothing enforces the rule at the point of the leak.
    • B. Correct: the data leaves when the user's browser loads the image URL, so limiting which hosts the page and renderer may load from stops the request whatever the model writes. The control is enforced outside the model and does not depend on the payload's format.
    • C. Incorrect: a pattern filter only catches the encoding it looks for. The attacker can switch to hex, split the data across several parameters or put it in the URL path, and the request still reaches the attacker host. It filters content instead of restricting where the browser may load from.
    • D. Incorrect: the request was made by the user's browser while rendering markdown, with no tool call involved. Removing fetch tools and restricting the service's outbound traffic does nothing about requests that originate on the client.

    7.2: Threat Defense and Adversarial Testing for Agentic Systems

    30.Hollow Telecom runs its network-provisioning platform from a monorepo that engineers across several teams clone daily. A repository's project-level MCP config launches a local stdio server through `npx`, and that server inherits the developer's full shell environment, including cloud credentials. In a recent exercise, the red team planted a look-alike package in a config. It started on a laptop as soon as the repository was cloned and opened, before any pull request existed, and it read the credentials. Hollow already has an onboarding guide listing vetted servers, but it is advisory only, and developers routinely add servers to their own settings by hand. The security team wants controls that still hold when someone skips the guide or a vetted server is later compromised. Select TWO changes that best reduce the risk.(Select 2)

    1. A.Define MCP servers only in each developer's user-level config instead of project files, and share the vetted list in the onboarding guide.
    2. B.Forward each server's process and network logs to the central SIEM, and alert the security team on anomalous outbound connections.
    3. C.Have code reviewers inspect each repository's MCP config before merging, and block changes that add unfamiliar servers or packages.
    4. D.Require developer approval before project-scoped servers start, and enforce an organization allowlist of permitted servers in managed settings.
    5. E.Start approved servers with an explicit minimal environment holding one narrowly scoped, short-lived token, not the developer's shell variables.
    Show answer & explanation

    Correct answers: D, E — Require developer approval before project-scoped servers start, and enforce an organization allowlist of permitted servers in managed settings.; Start approved servers with an explicit minimal environment holding one narrowly scoped, short-lived token, not the developer's shell variables.

    • A. Incorrect: moving definitions to user-level config only changes where they live. Nothing enforces which servers may run, and developers can still add a malicious one by hand. Any server that does start still inherits the full shell environment, including cloud credentials.
    • B. Incorrect: log forwarding and alerting are detective controls. By the time an anomaly is flagged, the server process has already read the inherited cloud credentials and may have sent them out. Neither the launch nor the credential exposure is prevented.
    • C. Incorrect: review at merge time comes after the repository has been cloned, and the config can already launch the npx server on a developer's machine. It also depends on people spotting a malicious package by eye, which is not a reliable control.
    • D. Correct: per-developer approval plus an organization allowlist in managed settings means a cloned repository cannot start an arbitrary server just by shipping a config. Only vetted servers run.
    • E. Correct: launching a server with a minimal environment and a narrowly scoped, short-lived token limits what a compromised or malicious server can reach. It no longer receives the developer's cloud credentials from the shell.

    7.3: Identity, Secrets, and Audit at Scale

    31.Marlow Pharma runs a clinical-trial document summarization service in production that calls the Claude API through one shared key used by three backend services. An engineer discovers that this production key was committed to a public repository 40 minutes ago, and the repository has already been forked several times. The Console usage page now shows requests from IP addresses that none of Marlow's services use, and their volume is climbing. The compliance team wants a full incident record of where the requests came from, and the security group wants leak prevention added across every repository. The three services are customer-facing and cannot stay degraded for long. The Console retains historical usage records for the key. What should the team do first?

    1. A.Rewrite the repository history with a force-push to remove the commit, confirm no copies remain in forks, then rotate the credential afterward.
    2. B.Revoke the exposed credential in the Console, ship a replacement to the dependent services, then audit usage logs and clean the repository history.
    3. C.Export the Console usage report and trace the unfamiliar IP addresses to document the incident in detail, then revoke the credential afterward.
    4. D.Roll out a secret-scanning pre-commit hook to all of the team's repositories, then rotate the credential during the next scheduled maintenance window.
    Show answer & explanation

    Correct answer: B — Revoke the exposed credential in the Console, ship a replacement to the dependent services, then audit usage logs and clean the repository history.

    • A. Rewriting history cannot undo the exposure. The credential has been public for 40 minutes and may already be scraped, and forks or clones keep their copies. Until it is rotated, whoever holds it can keep making requests.
    • B. Revoking the credential cuts off the unfamiliar requests at once. A replacement restores service for the legitimate callers. Usage review and history cleanup can follow safely because the leaked value no longer works.
    • C. Tracing the traffic and writing up the incident first leaves the credential working while the unauthorized requests continue. The usage records remain available after revocation, so nothing is lost by revoking first.
    • D. A pre-commit hook prevents future leaks but does nothing about a credential that is already public and being used. Waiting for a scheduled window keeps the abuse going when the credential should be rotated immediately.

    8: Agent Fleet Operations for Software Delivery

    8.1: Parallel Agent Orchestration for Engineering Work

    32.Heron Insurance is adding claims-portal endpoints to its production policy-administration service, and a lead engineer fans out six agents, each assigned one endpoint. Every endpoint must return failures through a new shared `ApiError` type. Each agent works in its own git branch and checkout, and none can see the others' work until its branch is merged. In an earlier trial run, every agent added its own variant of the type to the same types file. Merging the branches produced conflicting definitions that disagreed on shape, such as error code fields and detail payloads. The endpoint handlers on each branch were written against that branch's own variant. Heron wants all six endpoints merged this week with no reworked handlers. What is the most effective strategy?

    1. A.Add a step to each agent's prompt to search the types file for an existing ApiError before writing code, then reuse that definition and extend it where needed.
    2. B.Merge the six branches in completion order with the `-X theirs` strategy option, then run the type checker on the merged result and fix whatever it reports.
    3. C.Have each agent define its ApiError in a new file inside its own endpoint folder, then merge the six definitions into one type after the branches land.
    4. D.Commit the ApiError type to main in its own small change, then create the six agent branches from that commit and tell the agents to import it, not redefine it.
    Show answer & explanation

    Correct answer: D — Commit the ApiError type to main in its own small change, then create the six agent branches from that commit and tell the agents to import it, not redefine it.

    • A. Incorrect: the six agents run in parallel on separate branches, and each sees only its own checkout. An instruction to search for an existing ApiError finds nothing, so every agent still writes its own version.
    • B. Incorrect: `-X theirs` keeps the incoming side of each conflicting hunk, so earlier agents' definitions are dropped. Their endpoints were written against a shape that no longer exists, so the type checker reports rework across several branches while the shape disagreement stays unresolved.
    • C. Incorrect: separate files give six different ApiError shapes, and the endpoints are built on them before anyone reconciles them. The consolidation step repeats the same shape disagreement later, after more code depends on each variant.
    • D. Correct: a type every endpoint needs is a sequencing dependency, not parallel work. Landing it first and branching from that commit gives all six agents one definition to import, so their branches no longer overlap in the types file.

    8.2: Verification and AI Code Review Operations

    33.Pebble Retail runs its online storefront and order-fulfillment platform, where coding agents now open about 120 pull requests a day. Human reviewers can clear roughly 60, so the median review wait has reached three days and agent-authored changes pile up in the queue. Engineering leadership wants to add an AI code reviewer to ease the backlog. The team has never run it on its own repositories, so nobody knows how often it flags clean changes or misses real defects. Authors are already frustrated with delays, and the payments code owners require that any control affecting merges be backed by measured evidence. What is the best first rollout step?

    1. A.Set the AI reviewer as a required status check on the whole repository from day one, with humans looking only at the PRs it flags for escalation.
    2. B.Run the AI reviewer on the full PR stream in shadow mode, with findings hidden from authors and checked against human review outcomes.
    3. C.Enable the AI reviewer as a blocking check on agent-authored PRs only, with its verdict standing in for human sign-off on those changes.
    4. D.Tune the reviewer prompt on last quarter's merged PRs until it matches the human approvals, then let it auto-approve low-risk changes.
    Show answer & explanation

    Correct answer: B — Run the AI reviewer on the full PR stream in shadow mode, with findings hidden from authors and checked against human review outcomes.

    • A. A required check on day one gives an unmeasured reviewer control over merges. Its false positives would block authors, and its misses would pass unseen because humans only see what it flags.
    • B. Shadow mode runs the reviewer on real PR traffic without affecting merges or authors. The team can measure it against actual human decisions before it gets any authority.
    • C. Replacing human sign-off on agent PRs removes oversight where the volume is highest. Nothing yet shows the reviewer catches what humans catch, so a blocking verdict there is premature.
    • D. Matching past approvals teaches the reviewer to repeat human misses and overfits to historical PRs. Letting it auto-approve afterwards skips any check against live traffic.

    8.3: Unattended and Event-Driven Agent Operations

    34.Gull Health runs a claims-processing platform and wants coding agents to work unattended overnight, triaging failing integration tests and opening fix pull requests with no engineer watching. The security lead points out that the shared long-lived build host has accumulated cached tokens, leftover working directories and broad service credentials from earlier jobs, and that these agents read repository code and can call the deploy API. Platform engineers are weighing a move to ephemeral isolated runners. They already accept that the model's fixes may differ from night to night whatever the environment, and their API usage sits under an organization-wide quota with a queue in front. Permission rules for the agent's actions will stay in place either way. A team is deciding whether to run unattended coding agents in ephemeral isolated runners rather than on the shared build host. What is the main operational reason?

    1. A.A fresh runner starts from a clean checkout and holds only narrowly scoped credentials, which limits what one faulty or hijacked run can reach.
    2. B.A fresh runner starts with identical files and dependencies, which makes the agent's generated fixes come out the same from one night to the next.
    3. C.A fresh runner brings its own API rate limit allowance, which lets the fleet scale out to many parallel runs without a concurrency cap or queue.
    4. D.A disposable runner is torn down after the run, which makes permission rules and credential scoping unnecessary for the agent's actions.
    Show answer & explanation

    Correct answer: A — A fresh runner starts from a clean checkout and holds only narrowly scoped credentials, which limits what one faulty or hijacked run can reach.

    • A. A clean checkout and narrowly scoped credentials restrict what an unattended run can read or change, and nothing carries over between runs. That containment is the main operational reason to prefer ephemeral runners over a shared long-lived host.
    • B. Identical files and dependencies remove environment drift, but the model's generation stays nondeterministic. The same task can produce a different fix on the next night's run, so a fresh runner does not make results repeat.
    • C. Rate limits are enforced per account or workspace, not per runner, so new runners share the same quota. A large fleet still needs concurrency limits or queueing to avoid throttling errors.
    • D. Teardown only discards local state. Pushed commits, API calls and leaked credentials persist after the runner is gone, so permission rules and credential scoping still bound what an unattended run can do while it is alive.

    8.4: Organization-Wide Coding Agent Configuration

    35.Quarry Games runs a live-service multiplayer platform across roughly 120 repositories, and its engineers use a coding agent daily. Teams share skills by copying folders between repositories. One skill with a `Bash(*)` entry in its allowed tools reached 40 repos before anyone noticed. The post-incident review found that nothing inspected the skill before it spread, and the security team only learned of it from an engineer's chat message. Many repos belong to studios that have not adopted any central review tooling, and engineers can currently add any plugin source through their own user settings. Leadership wants central distribution with a quality bar enforced technically before release, not left to guidance or after-the-fact reports, and wants a single vetted origin for skills. Which TWO changes fit? Select TWO.(Select 2)

    1. A.Let teams push straight to the internal marketplace's main branch, and add a rollback command plus an alert channel for skills that engineers report as misbehaving.
    2. B.Host an internal plugin marketplace for skills, and limit the permitted marketplaces in managed settings to that reviewed internal source, blocking outside ones.
    3. C.Add a CODEOWNERS rule in each repo so the security team reviews changes to allowed-tools in SKILL.md, and publish a guide on scoping tool grants for skill authors.
    4. D.Gate release tags on CI that lints frontmatter and allowed-tools scope and runs the skill on sample tasks, and require a named owner's approval before publishing.
    5. E.Approve a well-known public marketplace for the org, and keep a wiki list of vetted skill names that engineers are asked to check before they install one.
    Show answer & explanation

    Correct answers: B, D — Host an internal plugin marketplace for skills, and limit the permitted marketplaces in managed settings to that reviewed internal source, blocking outside ones.; Gate release tags on CI that lints frontmatter and allowed-tools scope and runs the skill on sample tasks, and require a named owner's approval before publishing.

    • A. Direct pushes to the marketplace's main branch mean nothing is checked before a skill is distributed. A skill with a broad `Bash(*)` grant would spread to many repos before anyone reported it, which is the failure already seen. A rollback command and alert channel react after the fact and set no quality bar.
    • B. An internal marketplace gives one central place to distribute skills. Limiting permitted marketplaces in managed settings makes it the only source plugins and skills can be installed from, so engineers cannot add an outside marketplace from their own settings to avoid review.
    • C. A CODEOWNERS rule only covers repositories that adopt it, and only changes made inside them. A skill folder copied into another repo still arrives without that review, so distribution stays decentralised. The scoping guide is advisory, so nothing technical stops an overly broad tool grant from being published.
    • D. A release pipeline enforces the bar before anything is published. Linting allowed-tools scope would have flagged the broad Bash grant, sample-task runs catch behavioural problems, and the named owner's approval gives accountability for each release.
    • E. A public marketplace puts unreviewed third-party skills within reach of every engineer, and the org has no say over their tool permissions. A wiki list is advisory: nothing technical restricts what can be installed, and it does not provide central distribution or an org-defined quality bar.

    Want the full experience?

    These are just samples. Practice the full Anthropic Claude Certified Developer – Professional (CCDV-P) question bank in quiz mode — free, no signup, with domain practice and exam simulation.