Confidence scores in document extraction: what they mean and how to use them

What does a confidence score mean?

A confidence score is the system’s own estimate of how likely a value is to be correct. Microsoft’s documentation puts it concretely: a field confidence of 0.95 indicates the prediction is likely correct 19 times out of 20. Amazon Textract uses a 0 to 100 scale for the same idea. What the score covers varies more than the number suggests. Microsoft separates document type confidence, field confidence, word confidence and, for tables, cell, row and table scores. Its field score for custom models reflects the position of the value, while the word score reflects transcription, and it advises combining them. So a field can be found in the right place and still be misread. Before relying on any score, find out what it measures, whether it is reported on every field, and which step of the process produced it. Two vendors’ 0.9s are not the same thing.

Can you trust confidence scores?

Trust a score once you have checked it against your own documents, and not before. The property you want is calibration: of all the values scored around 0.9, about 90% should be right. That cannot be assumed. Guo et al. found that modern neural networks, unlike those of a decade earlier, are poorly calibrated. Xiong et al. found that LLMs tend to be overconfident when they state their own confidence, which matters if a parser simply asks a model for a score. To test calibration, take a few hundred labelled values, group them into score bands and compare each band’s stated confidence with its observed accuracy. A flat result, where high and low scores are right equally often, means the score cannot do the job for that field. Microsoft’s own guidance concedes the point: for some fields, confidence and correctness are not closely enough related to automate at all.

How do you set a confidence threshold for each field?

Set thresholds per field, because a single cutoff behaves differently on every field. Microsoft’s Foundry team shows this on receipts, where one 0.70 cutoff gave very different auto-approval and missed-error rates from field to field. The decision is a trade. Research on selective classification calls it risk against coverage: abstain more often and the answers you do give are safer. Google’s Document AI describes the same thing as precision against recall, and defaults to the threshold that maximises F1. That default treats both kinds of error as equal, which is rarely true of a payment amount. Amazon’s published examples are a starting range: perhaps 50% for archival notes, 90% or higher for financial decisions. Treat those as prompts, not answers. The method below replaces intuition with evidence, so “0.8 feels safe” never becomes policy.

1. Label real documents for each type, including poor scans, and hold some back.
2. For each field, decide the target: the share of errors you must catch, or the error rate you accept among auto-accepted values.
3. Find the lowest threshold that meets the target, so that the fewest values go to review.
4. If no threshold meets it, keep that field in review permanently.
5. Validate on the held-back documents, then record the threshold and why it was chosen.

How should human review work?

Human review works best as a targeted queue: a person sees the doubtful value, the source page beside it, and nothing else that needs attention. Review should correct the field, not retype the document. It also has limits, so do not treat a threshold as a guarantee. In a case Microsoft reports, covering more than 22,000 field values from over 1,000 production documents, targeted review selected 41% fewer values while catching more than 80% of known errors. That is a useful saving, and it also means some errors passed unflagged, so values above the threshold need checking too. Amazon’s Augmented AI service, for example, was built around both low-confidence predictions and random samples. Sample the auto-accepted stream, track the errors you find, and tighten any field that slips. Corrections are also labelled data: keep them, because they are what your next calibration check will run on.

Should extraction return null or a best guess?

Return null. A missing value stops at the next check, where a person can deal with it. A guessed value looks the same as a real one and travels on into your ledger. Models lean towards guessing for structural reasons: Kalai et al. argue that training and evaluation reward guessing over admitting uncertainty, and research on degraded invoices and ID cards found multimodal models inventing content where the image could not support an answer. Empty results need a policy of their own, too. Microsoft’s Foundry team found that confidence on empty results did not reliably tell “no value exists” from “value missed”, so base that policy on how often blank is the right answer for the field. A PO number that is blank on most invoices from a supplier can pass; a blank total never should. Ask a vendor to show you what happens when a requested field is not on the page.

When should you revisit thresholds?

Thresholds expire. Microsoft says calibration applies only to the documents, configuration and field schema it was measured on, and should be repeated when any of those change, when the underlying model changes, or when monitoring shows drift. In practice that means rechecking after a vendor’s model update, after you add fields, after you onboard a new document type, and when a sender changes their layout. A change in review volume with no change in incoming work is a warning sign in either direction.

How does docRefinery handle confidence?

In docRefinery, every field carries a confidence score by default, checked against a threshold you set, per field and per document type. The score can be lowered, never talked up. A field that is not on the page comes back null, not as a plausible guess, and reviewers see the source next to the value, with a page and box wherever it can be placed. Scores use the same format on every document type, so one rule in your code can route them. A reviewer’s correction can become a suggested fix, tested on that layout’s own documents before anyone applies it, and Insights shows flagged fields and auto-approval trends when you revisit thresholds. See the platform page for the review tools, the invoice field reference for an example, and the overview of extraction approaches if you are still choosing a method. For how scoring differs between tools, see the comparison page. The invoice extraction guide covers the checks that should sit beside any score.