Human Review Workflows and Confidence Calibration for Extraction

1.Why 97% overall can still mean a broken pipeline

Structured-extraction systems usually report one headline number, such as "97% of fields extracted correctly." That number is a weighted average, and the weights come from volume. Say typed invoices make up most of your traffic and come out almost perfect. Then a small share of handwritten forms, or one hard field such as a dosage, a policy number or a date inside a long table, can be wrong far more often and barely move the overall figure. The average is accurate arithmetic, but it answers the wrong question. The question behind any decision to cut human review is "where is this pipeline safe to trust?", and an overall number can't answer that.

Anthropic's own evaluation guidance keeps pushing toward breaking results down by segment. The support-ticket routing guide doesn't stop at one routing accuracy. It asks teams to check whether accuracy holds across segments, for example to measure accuracy "across different languages, aiming for no more than a 5–10% drop in accuracy for non-primary languages." Swap "languages" for "document types" or "fields" and you have the discipline this objective tests. A team building a clinical-abstraction system on Claude put it the same way: granular evaluation lets you find the cause of a failure "rather than staring at an aggregate score wondering what went wrong."

Sources12

2.Validate every segment before you automate it

Validating before automating means setting a bar per segment, then showing that each segment you plan to automate clears it. In practice: take the labeled evaluation set you already have and group it by document type (auto, home and health claims, say). Within each type, group it by field. Compute accuracy for each cell of that grid. Every cell clears the bar or it doesn't, and you only reduce review for the cells that do. If a segment has too few labeled examples to give a stable number, that means you need more labeled data for it. It isn't a pass.

The edge-case row matters most here. The guide says to "create a test set of edge cases and measure the routing accuracy" as its own number. Handwritten forms, poor scans and unusual layouts are the edge cases of document extraction. If they are left mixed into the main set, their failures disappear into the average again.

Sources1

3.Field-level confidence, calibrated against labeled data

Once you know which segments are weak, you need a way to point reviewers at the specific extractions that are likely wrong. The standard approach has the model return a confidence score for each field, not one score per document. A document-level score can't tell a reviewer that the invoice total is solid but the due date is a guess. Anthropic's prompting guidance recommends the same shape for other structured outputs: ask for a confidence level on every item so a later stage can act on it. The exact wording is "include your confidence level and an estimated severity so a downstream filter can rank them."

The catch is that a model's self-reported confidence is only a raw signal. A 0.9 from the model doesn't mean the field is right nine times in ten. To make it meaningful, you calibrate it. Run the pipeline over a labeled validation set, where every field has a golden answer to check against. Group the extractions by stated confidence, and measure how often each group is actually correct, broken down by field and document type. Then set each routing threshold at the confidence level where the measured error rate for that field is low enough to accept. You set the number from measured data. You don't pick it because it sounds reasonable.

Sources34

4.Spending limited reviewer time where it counts

Reviewer capacity is always limited, so the review queue works as a triage system. Two kinds of extraction belong at the front. The first is fields whose calibrated confidence falls below their threshold. The second is extractions whose source is ambiguous or contradicts itself: two different totals on one invoice, a date that conflicts with another field, a document that fits none of the known types. The model may report high confidence and still be wrong in those cases, because the source doesn't support a single answer. Ambiguity and contradiction should therefore be separate routing signals, not something folded into one score.

Anthropic's prompting guidance suggests separating the two jobs. The model reports everything, with its confidence attached, and a separate stage decides what to act on, because "moving confidence filtering out of the finding step often helps." For extraction, that means the model doesn't quietly drop or quietly guess an uncertain field. It returns the field with its confidence and any conflict it noticed, and the routing layer, driven by calibrated thresholds, decides whether a human sees it. Then watch the queue itself. The content-moderation cookbook advises "Route needs_review to humans and watch its rate". If one field keeps showing up in review, look at its instructions or its segment.

Sources35

5.Keep sampling what you have automated

Automating the high-confidence extractions doesn't end the measuring. If nobody ever looks at auto-accepted fields, you can't see their error rate, and you won't notice when a new vendor template, a new form version or a change in scan quality introduces an error pattern your validation set never contained. So keep sending a random sample of high-confidence, auto-accepted extractions to human review. The sample exists for measurement, not for catching individual errors. The monitoring guidance in Anthropic's writing on evals describes this post-launch job: "Production monitoring kicks in post-launch to detect distribution drift and unanticipated real-world failures."

The sample should be stratified, not a simple random draw from the whole stream. A simple random sample reproduces your traffic mix, so a document type that is 2% of volume gets 2% of the sample, too few examples to estimate its error rate. Stratified sampling draws a set number from each document type (and, where it matters, each field or confidence level), so every segment gets enough reviewed examples. Anthropic's own benchmark reporting uses the same idea to keep a small sample representative: one result was "measured on a 50-task subset stratified across all themes." The reviewed examples do two jobs. They give each segment a running error rate for its auto-accepted fields, which tells you whether a threshold still holds. They also surface novel errors, which you can add to the labeled validation set before recalibrating.

Sources67

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.A high overall accuracy figure such as 97% is enough evidence to reduce human review across the whole pipeline.Why is that wrong?

   An overall figure weighted by volume can hide a document type or field that performs far worse. Break accuracy down by document type and field, and reduce review only for segments that clear the bar themselves.

   Covered in Why 97% overall can still mean a broken pipeline

2. 2.The model's self-reported confidence can be used directly as a review threshold, for example sending everything below 0.85 to review.Why is that wrong?

   Raw confidence has to be checked against a labeled validation set, per field, before a threshold means anything. If it isn't, high-confidence fields can still be wrong often and reviewers stop trusting the score.

   Covered in Field-level confidence, calibrated against labeled data

3. 3.Once high-confidence extractions are auto-accepted, human review only needs to see the low-confidence ones.Why is that wrong?

   Without a stratified random sample of auto-accepted, high-confidence extractions, their error rate can't be seen, and new error patterns from drift or new document formats go unnoticed.

   Covered in Keep sampling what you have automated