CertSafari
    Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards· Lessons

    Domain 4 · Lesson 22/39

    Deriving evaluations from business processes

    Deriving Evaluations from Business Processes

    16 min read
    2.43% of exam
    4 sources
    Published 18 Nov 2026
    Docs as of 4 Nov 2026

    What you will be able to do

    • Pick the part of a business process that a release gate, prompt change or model change has to be evaluated on, and reject parts that are easy to score but don't measure the outcome
    • Turn a business consequence into success criteria with a threshold for each dimension, set by how much each kind of error costs
    • Keep an eval set matched to current production traffic as the product changes
    • Combine code, model and human graders with production methods so the mix covers the deployed distribution

    Key concept

    Outcome-anchored evaluation — An evaluation is valid for a decision only if it scores the part of the process that determines the business outcome, against a known correct result taken from that process. Format, latency and how good the output looks are real dimensions, but on their own they don't show that the outcome happened.

    1.Choosing the part of the process an eval must cover

    Every Claude step inside a business process produces several things you could score: latency, token counts, whether the output parses, how clear an explanation reads. Almost all of them can be measured, and that makes it easy to gate a release on one of them. The real question is narrower. Which observable result tells you the business got what it needed? An eval is valid for a decision only if a better score on it means that result improved. So the order is fixed: define success first, then design the evaluation that measures performance against it.

    What a system says and what a process produces are different things. A flight-booking agent can report that the flight is booked. The outcome is whether a reservation exists in the database. Grade the outcome, not the report. For a ticket-triage step, the outcome is whether the ticket reached the team that resolves it. For an invoice-extraction step, the outcome is whether the amount paid matches the invoice.

    The process usually holds its own golden answers already, and these are the reference you compare model output against. A ticket that went straight to a team and was resolved there without reassignment has a confirmed correct queue. A reassigned ticket started with a disputed label. Replaying the confirmed cases through the new prompt and comparing each assignment with the team that resolved it tests exactly the outcome the business pays for. The other candidates measure something else. A JSON-and-valid-queue check confirms the output is well-formed, not that it is right. Latency and token cost are legitimate criteria, since response time and budget belong in most criteria sets, but they say nothing about whether routing improved. Expert ratings of the rationale grade a by-product nobody downstream uses.

    The decision shapes how the suite is used, not what counts as success. For a prompt change or a model change, run the same task bank on both configurations and compare. Teams that already have evals can assess a new model and upgrade in days, as covered in model version migration. A release gate is a regression question: does the system still handle what it used to? Regression evals should pass at close to 100%, so a drop means something broke. Capability evals start at a low pass rate. They are a hill to climb, not a gate.

    Checkpoint 1 of 7· Check yourself

    A team is deciding whether to move a meeting-scheduling agent to a new model. At the end of each trial the agent tells the user something like "Your meeting is booked for Tuesday at 10:00." Which grader gives a valid basis for the migration decision?

    Checkpoint 2 of 7· Exam question

    A support organization uses Claude through the Messages API to triage inbound tickets into one of 14 queues. A prompt rewrite is proposed, and the architect owns the release gate. The business outcome is that tickets reach the team that resolves them without reassignment. Which portion of the process should the gate evaluate?

    Sources123

    2.Deriving the success criterion from the business consequence

    A success criterion should be specific, measurable, achievable and relevant, and relevance is where the business process comes in. How much an error costs depends on where the output goes next. Citation accuracy can be critical in a medical app and minor in a casual chatbot. The same holds inside one workflow. When outputs feed different downstream actions, each action's cost of error sets its own criterion.

    Most use cases need several success criteria at once. A strong criteria set lists the dimensions separately with a threshold for each: an F1 floor, a non-toxicity rate, a latency percentile, and a severity condition saying that 90% of errors should cause inconvenience rather than egregious harm. The severity condition matters most here. It admits that errors differ in cost and limits the expensive kind directly, instead of letting an average absorb them.

    One multidimensional criteria set: each dimension carries its own threshold
    DimensionThreshold
    Task qualityF1 score of at least 0.85
    Safety99.5% non-toxic outputs
    Error severity90% of errors would cause inconvenience, not egregious error
    Latency95% response time < 200ms

    Take invoice extraction ahead of payment scheduling. A wrong total pays the wrong amount. A wrong payee pays the wrong party. A wrong due date only shifts a reminder. The criterion therefore needs a separate threshold for each field, tightest on total and payee and loosest on due date. One combined accuracy figure lets correct due dates pull the average up while totals fail. The grader follows from the same reasoning. A payee or an amount has exactly one correct value, which is the clear-cut, categorical kind of answer that exact match handles well. A judge scoring "closeness" gives partial credit to a total with one wrong digit, even though that error is fully expensive. Mismatches on the costly fields can then be routed to human review checkpoints.

    Exact-match grading: the output either equals the verified value or it failspython
    def evaluate_exact_match(model_output, correct_answer):
        return model_output.strip().lower() == correct_answer.lower()

    Baselines tell you whether a target is achievable, not whether it is enough. Targets should rest on industry benchmarks, prior experiments, research or expert knowledge, and "5% better than the current baseline" is a reasonable achievability check. A release bar set at whatever the current prompt already scores turns today's failure rate into the definition of success. The required level comes from the cost of the error. The baseline only shows how far away that level is.

    Checkpoint 3 of 7· Fill the gap

    The payee field in an invoice-extraction step has exactly one correct value, and a near miss pays the wrong party. Which operator completes this grader so that only an output equal to the verified value passes?

    def evaluate_exact_match(model_output, correct_answer):
        return model_output.strip().lower()  ?  correct_answer.lower()

    Checkpoint 4 of 7· Exam question

    A finance team uses Claude to extract payee, invoice total and due date from supplier invoices before payments are scheduled. A wrong total triggers an incorrect payment, while a wrong due date only causes a minor reminder error. You must derive the success criterion for the evaluation from the business outcome. Choose two.(Select 2)

    0 of 2 selected

    Sources1

    3.Keeping the eval matched to the deployed distribution

    An eval predicts production quality only while its cases look like production traffic. Design evals that mirror the real-world task distribution, and keep the mix of questions and difficulty close to what users actually send. An eval that doesn't match real usage gives false confidence, and that is worse than having no eval, because it keeps passing.

    Drift happens when the business changes and the eval set doesn't. Suppose an order assistant's eval set was written six months ago and every release passes at 97%. The company then launches a subscription product. Subscription conversations become a growing share of traffic, and the eval contains almost none of them. The score stays high because it measures last year's traffic. Production monitoring catches this kind of drift after launch, and the complaints are that signal.

    The fix is to bring the eval set back in line with current traffic. Source realistic tasks from the failures you see. Sample recent production conversations, including the new flows, and add them with verified labels. Then reweight the combined set so each intent's share matches its current traffic share. Report the new intent as its own score, the same way a support suite reports refunds, cancellations and escalations separately, so its weakness can't hide inside a healthy overall number. Keep the original cases. Those intents still carry traffic, and work the system already handles well belongs in the regression suite. Raising the threshold or swapping the grader leaves the stale distribution in place, so a stricter gate still measures the wrong traffic. Finding what an existing suite misses is covered in evaluation coverage gaps.

    A labelled eval set that mixes routine cases with tagged edge casespython
    tweets = [
        {"text": "This movie was a total waste of time. 👎", "sentiment": "negative"},
        {"text": "The new album is 🔥! Been on repeat all day.", "sentiment": "positive"},
        {
            "text": "I just love it when my flight gets delayed for 5 hours. #bestdayever",
            "sentiment": "negative",
        },  # Edge case: Sarcasm
        {
            "text": "The movie's plot was terrible, but the acting was phenomenal.",
            "sentiment": "mixed",
        },  # Edge case: Mixed sentiment
        # ... 996 more tweets
    ]

    Edge cases belong in the distribution in proportion to how often they occur. Typical ones are irrelevant or missing input, overly long input, poor or harmful user input, and ambiguous cases where even humans would disagree. When privacy rules block testing on real inputs, Claude can generate synthetic test cases from the prompt template's variables. Those cases still have to resemble what production sends.

    Checkpoint 5 of 7· Check yourself

    Since a returns portal launched, every release of a ticket-triage step has still passed its eval at 96%, but complaints about misrouted returns tickets keep rising. Returns tickets now carry a growing share of traffic. What should the team change before trusting the next gate?

    Checkpoint 6 of 7· Exam question

    Six months ago a team built an eval set for its Claude order-assistant, and every release has passed it with 97 percent. Support now reports rising complaints after the company launched a subscription product, which accounts for a growing share of conversations. Which actions best restore the evaluation's link to the deployed distribution? Choose two.(Select 2)

    0 of 2 selected

    Sources1324

    4.Selecting the test-methodology mix

    Choose a grader for each dimension of the criterion, and automate wherever the task allows. Code-based checks are fast, cheap and reproducible, but they break on valid variations of the expected answer. Model-based graders handle open-ended output but are non-deterministic and need calibration against human graders. Human grading is the gold standard but is slow and expensive. Grading cost comes back on every run, so a large eval graded automatically usually beats a small hand-graded one. Questions can often be restructured, for example as multiple choice, so that code can grade them. Rubric and bias design for model graders is covered in LLM-as-judge design.

    Grader types and what each can cover
    GraderTypical methodsUse it forWeakness
    Code-basedString match, binary tests, outcome verification, tool call verificationFields and outcomes with one correct answerBrittle to valid variations
    Model-basedRubric-based scoring, pairwise comparison, reference-based evaluationTone, open-ended and freeform outputNon-deterministic; needs human calibration
    HumanSME review, spot-check sampling, inter-annotator agreementCalibrating model graders; subjective outputsExpensive and slow

    No single offline suite can cover the whole deployed distribution, so the mix includes production methods too. Automated evals run in CI on every change and model upgrade. Production monitoring detects drift and real-world failures nobody anticipated. A/B tests validate significant changes against actual user outcomes once there is enough traffic. Weekly transcript reading and user-feedback triage catch what the other methods miss. Systematic human studies are reserved for calibrating model graders and for judging subjective outputs. Each layer, like a slice in the Swiss cheese model, catches failures that slipped through another.

    Checkpoint 7 of 7· Match them up

    No single offline suite covers the deployed distribution of a ticket-triage step. Match each layer of the methodology mix to the job it does.

    Tap a term, then the definition that fits it.

    Sources132

    Sources

    (4)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Building a successful LLM-based application starts with clearly defining your success criteria and then designing evaluations to measure performance against them.”
      ↩︎ Choosing the part of the process an eval must cover
      “What is the acceptable response time for the model?”
      ↩︎ Choosing the part of the process an eval must cover
      “Strong citation accuracy might be critical for medical apps but less so for casual chatbots.”
      ↩︎ Deriving the success criterion from the business consequence
      “Most use cases need multidimensional evaluation along several success criteria.”
      ↩︎ Deriving the success criterion from the business consequence
      “90% of errors would cause inconvenience, not egregious error”
      ↩︎ Deriving the success criterion from the business consequence
      “It’s a simple, unambiguous metric that’s perfect for tasks with clear-cut, categorical answers”
      ↩︎ Deriving the success criterion from the business consequence
      “Base your targets on industry benchmarks, prior experiments, AI research, or expert knowledge.”
      ↩︎ Deriving the success criterion from the business consequence
      “Be task-specific: Design evals that mirror your real-world task distribution. Don’t forget to factor in edge cases!”
      ↩︎ Keeping the eval matched to the deployed distribution
      “More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals.”
      ↩︎ Selecting the test-methodology mix
      “Automate when possible: Structure questions to allow for automated grading (for example, multiple-choice, string match, code-graded, LLM-graded).”
      ↩︎ Selecting the test-methodology mix
      “Most use cases need multidimensional evaluation along several success criteria.”
      ↩︎ Exam trap 2
      “Be task-specific: Design evals that mirror your real-world task distribution. Don’t forget to factor in edge cases!”
      ↩︎ Exam trap 3
    2. 2.
      “the outcome is whether a reservation exists in the environment’s SQL database”
      ↩︎ Choosing the part of the process an eval must cover
      “Regression evals ask, “Does the agent still handle all the tasks it used to?” and should have a nearly 100% pass rate.”
      ↩︎ Choosing the part of the process an eval must cover
      “They should start at a low pass rate, targeting tasks the agent struggles with and giving teams a hill to climb.”
      ↩︎ Choosing the part of the process an eval must cover
      “Can create false confidence if it doesn’t match real usage patterns”
      ↩︎ Keeping the eval matched to the deployed distribution
      “Production monitoring kicks in post-launch to detect distribution drift and unanticipated real-world failures.”
      ↩︎ Keeping the eval matched to the deployed distribution
      “Source realistic tasks from the failures you see.”
      ↩︎ Keeping the eval matched to the deployed distribution
      “a customer support eval suite might test refunds, cancellations, and escalations.”
      ↩︎ Keeping the eval matched to the deployed distribution
      “Like the Swiss Cheese Model from safety engineering, no single evaluation layer catches every issue.”
      ↩︎ Selecting the test-methodology mix
      “Reactive; problems reach users before you know about themSignals can be noisyRequires investment in instrumentationLacks ground truth for grading”
      ↩︎ Selecting the test-methodology mix
      “the outcome is whether a reservation exists in the environment’s SQL database”
      ↩︎ Key concept
      “the outcome is whether a reservation exists in the environment’s SQL database”
      ↩︎ Exam trap 1
      “capability evals with high pass rates can “graduate” to become a regression suite that is run continuously to catch any drift”
      ↩︎ Exam trap 4
      “the outcome is whether a reservation exists in the environment’s SQL database.”
      ↩︎ Checkpoint
      “Production monitoring kicks in post-launch to detect distribution drift and unanticipated real-world failures. A/B testing validates significant changes once you have sufficient traffic.”
      ↩︎ Checkpoint
    3. 3.
      “A "golden answer" to which we compare the model output.”
      ↩︎ Choosing the part of the process an eval must cover
      “try to have the distribution in your eval represent ~ the real life distribution of questions and question difficulties.”
      ↩︎ Keeping the eval matched to the deployed distribution
      “Grading on the other hand is a cost you will incur every time you re-run your eval, in perpetuity”
      ↩︎ Selecting the test-methodology mix
      “try to have the distribution in your eval represent ~ the real life distribution of questions and question difficulties”
      ↩︎ Checkpoint
    4. 4.
      “maybe you aren't allowed to test on the ones you do have for privacy reasons. No worries -- Claude can make them up!”
      ↩︎ Keeping the eval matched to the deployed distribution

    Ready to test yourself?

    Practise Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.