Human Review Routing and Confidence Calibration for Extraction

1.Why 97% overall can still hide a failing segment

An aggregate accuracy figure is a weighted average. The documents you see most often dominate it, and the fields that appear on every document dominate it too. Suppose a pipeline gets typed forms almost perfectly right and has trouble with handwritten ones. If typed forms make up most of the volume, it can still report a strong overall number. The same happens across fields. A pipeline might extract names and dates very well while getting dosage amounts or currency totals wrong far more often, and those rarer, harder fields are often the ones that cause the most damage when they are wrong.

Anthropic's evaluation guidance asks you to define success in terms that include the hard cases, not only the typical ones. Its classification cookbook builds a confusion matrix for this reason: it shows which specific categories the model confuses, which a single accuracy number cannot.

This leads to the rule the exam tests most often. Before you reduce human review of any segment, measure accuracy for that segment. Use your labeled set to compute accuracy by document type, by field, and by the combination of the two, such as 'dosage field on handwritten prescriptions'. You can automate a segment only once it meets your bar on its own. If one cell falls short, the right response is targeted. Keep review on that document type or field, or improve its prompt and schema, and still reduce review on the segments that pass. A failing segment does not mean you should keep everything under review, and a good aggregate does not mean everything is safe to automate.

Sources12

2.Field-level confidence, calibrated on a labeled validation set

To decide which extractions a human should look at, you need a signal for each extraction. More precisely, you need one for each field, because a single document can have a reliable invoice number and a doubtful total. A common design asks the model to return a confidence score next to every extracted field. Structured outputs keep that shape dependable: once the schema requires a confidence property for each field, every response contains one and your routing code can read it.

A score being present does not make it accurate. Scores the model reports about itself are not automatically probabilities. In Anthropic's own structured-extraction cookbook, a classification example returns category scores that are not even consistent with one another. The fragment below gives Technology 0.7 and Business 0.5 for the same text, along with other scores, and the total is well over 1. These numbers are useful for ranking items. They are not yet error rates.

    {
      "name": "Technology",
      "score": 0.7
    },
    {
      "name": "Entertainment",
      "score": 0.1
    },
    {
      "name": "Business",
      "score": 0.5
    }
  ]
}

Calibration is how you turn those scores into something you can rely on. Run the pipeline on a labeled validation set that is held out from anything you used to tune the prompts. Group the fields by the score the model gave them, and for each group measure how often the extraction was actually correct. Then choose the review threshold from the measured accuracy, not from the raw score. You might find that 'confidence ≥ 0.9' means 99% correct for dates but only 85% correct for line-item totals. In that case each field needs its own threshold. Repeat the calibration whenever the prompt, the model, or the mix of documents changes. A threshold that was never calibrated, or was calibrated against a different document mix, is the usual reason reviewers find high-scoring fields that are wrong and stop trusting the score.

Sources314

3.Routing the right items to limited reviewers

Reviewer time is limited, so the routing rule decides where it goes. Two kinds of signal should send an extraction to a person.

Low calibrated confidence. The field's score falls below the threshold you set for that field in the previous step.

Problems in the source itself. Some inputs are hard regardless of how good the model is: a document that contradicts itself, two documents that disagree, a field that is illegible or missing. Carta Healthcare's clinical abstraction work is a good example. Different sources in one record can support different answers, and choosing between them takes judgment. For these cases, prompt the model to flag the conflict explicitly instead of quietly choosing one value. Anthropic's hallucination guidance recommends giving Claude explicit permission to say it doesn't know. That gives you an honest 'unsure' or 'conflicting sources' signal you can route on.

Once the right items reach the queue, put them in order. Send the cases most likely to be wrong and the most costly ones (high value, regulated, or legally significant) first, and let clean high-confidence items pass. Anthropic's guidance for financial-services agents describes routing complex cases to specialists 'based on clear criteria'. The content-moderation cookbook adds a monitoring habit worth borrowing: track how often items are routed to review. If one field suddenly starts generating many 'needs review' results, its definition probably needs work.

Sources5678

4.Stratified sampling of what you no longer review

Once high-confidence extractions are accepted automatically, nobody looks at them. That creates a blind spot. A calibration that held last quarter can drift when a new vendor's layout appears, a form is revised, or the model is upgraded. The pipeline will keep reporting high confidence on outputs that are now wrong. The fix is to keep auditing a small random sample of the accepted extractions, check them against human judgment, and measure the error rate.

The sample should be stratified: split the auto-accepted output into document type × field cells and sample from each cell. A plain random sample has the same problem as the aggregate metric. The rare, risky cells (handwritten forms, an uncommon field) would almost never be picked, so you could not measure their error rates. Sampling every stratum gives each segment its own ongoing error estimate. The sampled items also act as an early warning, because they show you new types of error that your labeled set never contained.

Write the findings back into the system. Add confirmed new errors to the labeled validation set, recalibrate the affected thresholds, and if a segment's sampled error rate rises above your bar, put it back under full review. Cainex, as described in Anthropic's startup guide, follows the same pattern: it tests candidate changes against a golden set plus random samples before anything ships. Anthropic's evals article makes the general point that production monitoring catches issues synthetic evals miss.

Sources1910

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.A high overall accuracy (for example 97%) is enough evidence to reduce human review across the whole pipeline.Why is that wrong?

   The aggregate is dominated by common document types and fields. Accuracy has to be checked for each document type and field, and review can only be reduced on the segments that meet the bar on their own.

   Covered in Why 97% overall can still hide a failing segment

2. 2.A model-reported confidence of 0.9 means the field is correct 90% of the time, so it can be used as a threshold directly.Why is that wrong?

   A confidence the model states about itself is not calibrated by default. Thresholds have to be set by comparing scores with ground truth on a labeled validation set, often separately for each field.

   Covered in Field-level confidence, calibrated on a labeled validation set

3. 3.Once high-confidence extractions are auto-accepted, they need no further human checking.Why is that wrong?

   Calibration drifts and new error patterns appear. Ongoing stratified random sampling of auto-accepted output is how you keep measuring their error rate.

   Covered in Stratified sampling of what you no longer review