AI data extraction: accuracy, confidence and the review threshold

Vendors quote an accuracy figure and it is almost always a single number, which is the least useful form the information could take. Accuracy differs by field, by document quality and by source, and the fields that matter most are frequently the ones extracted worst.

What determines whether an extraction system is worth running is not the headline figure. It is where you set the threshold between automatic and reviewed, how you arrived at that threshold, and what a wrong value costs downstream when nobody checks it.

What we'll cover

• Accuracy is a distribution
• What a wrong field costs
• Setting the threshold
• Building it with CodeWords
• Validating without a person
• Measuring it honestly
• The fields that are hardest
• Keeping it accurate over time
• Where the extracted data lands
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

Accuracy is a distribution

Per field, not overall. A system that is excellent at dates and poor at line items has one average and two very different realities.

Per source. One supplier's documents extract cleanly and another's do not, and the average conceals which.

Per document quality. A clean native file and a photograph are different populations.

The critical fields are often the hardest — totals, line items, reference numbers — which is why the overall figure flatters.

Ask for the breakdown rather than the headline, and if a vendor cannot provide it that is itself informative.

Measure it on your own documents, since a benchmark on somebody else's corpus predicts very little.

What a wrong field costs

It varies enormously by field, which is what should drive the threshold.

A wrong amount propagates into a payment, a report or a reconciliation, and is expensive.

A wrong date may shift something into the wrong period, which surfaces at month end.

A wrong reference breaks a link to a record and produces an orphan somebody investigates.

A wrong description is frequently harmless.

Set the threshold per field accordingly, rather than applying one number to everything, which is the most common design error here.

Setting the threshold

Empirically, from labelled data, not by choosing a round number that sounds prudent.

Look at what was wrong below each level, which tells you what a given threshold actually buys.

Accept a review rate. A system with no review queue either has a trivial task or is producing wrong records silently.

Different thresholds per field and per source, since the underlying accuracy differs.

Tighten for consequence. Anything feeding a payment or a regulatory record deserves a stricter threshold than something feeding a dashboard.

Revisit it, since the threshold that was right for last year's document mix may not be for this year's.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: what to extract, the validation rules, and what goes to review.
3. Authorize the connections — where documents arrive, and the system receiving the extracted data.
4. Describe the exceptions: a low-confidence field, a failed validation, a total that does not reconcile, an unreadable input.
5. Run it against documents you have already processed by hand and compare field by field.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Validating without a person

The cheapest accuracy improvement available, and the one most often skipped.

Check the arithmetic. Line items summing to the total catches a large share of extraction errors for free.

Check against a known list — supplier names against your supplier records, product codes against your catalogue.

Check plausibility: a date in the future, an amount an order of magnitude from normal, a quantity that cannot be right.

Check cross-field consistency, such as a tax amount against a rate and a subtotal.

Escalate a failed validation rather than correcting it, since a rule that silently fixes things hides the underlying problem.

Validation catches what confidence misses, because a model can be confident and wrong in ways arithmetic exposes immediately.

Measuring it honestly

Build a labelled set from real documents, which is tedious and unavoidable.

Field-level accuracy, per source, tracked over time.

Measure what got through, not just what was extracted — the errors that passed review are the ones that matter.

Sample the automatic path periodically, since that is where silent errors live and nothing else looks there.

Count corrections in review as the ongoing signal, which is free data you are already generating.

Publish the accuracy alongside the output, because a figure presented as exact will be trusted until somebody checks.

The fields that are hardest

Tables and line items, consistently, particularly across page breaks.

Anything handwritten, including signatures and annotations on printed forms.

Nested or conditional fields, where the meaning depends on another field.

Multi-currency and unusual number formats, where a decimal separator convention changes the value by a factor.

Dates without a clear format, which are ambiguous between conventions and silently wrong rather than obviously so.

Checkboxes and marks, which are surprisingly unreliable and frequently important.

Keeping it accurate over time

Document formats change, and a supplier changing accounting system changes everything about their invoices.

Monitor accuracy per source, since degradation is usually localised rather than general.

Watch the review rate as the leading indicator, because it rises before accuracy visibly falls.

Re-evaluate after any model or provider change, which can alter behaviour with nothing changing on your side.

Add corrected items to the labelled set, which is how it stays representative.

Treat it as an ongoing measurement, not a one-off validation before go-live.

Where the extracted data lands

Extraction is usually evaluated on its own and then feeds a system with its own opinions, which is where a second class of error appears.

The destination validates too, and rejections there arrive after extraction has reported success.

Type and format mismatches are common — a date the destination will not accept, a currency field expecting a specific shape.

Required fields the document did not contain force a choice between a placeholder and a failure, and placeholders are how bad data enters.

Write atomically where you can, since a partially created record is worse than none.

Handle the rejection path, because a document that extracted perfectly and failed to write is currently invisible in most implementations.

Reconcile counts between documents processed and records created, which is the check that catches everything the two systems disagree about.

Making it survive

Never write an unvalidated critical field, and define which fields are critical explicitly.

Keep the source document linked to every extracted record, permanently, since checking is otherwise impossible.

Record the confidence with the value, so a later investigation can distinguish a confident error from an uncertain one.

Make extraction idempotent on the document, since the same file arrives twice.

Report the outcome: documents processed, fields extracted, validation failures, review rate, and corrections made.

Limits worth knowing about

No system reaches perfect accuracy, and one claiming to is being measured on a narrow corpus.

Confidence is imperfectly calibrated, so the threshold needs validating rather than trusting.

Poor source quality sets a ceiling that no amount of processing removes.

Labelled data is expensive to produce and is the input everything here depends on.

Automation does not remove the reviewer, it changes what they look at, and an implementation sold on removing them entirely is overpromising.

What to build first

The validation rules, before tuning anything about extraction. Arithmetic checks, list lookups and plausibility rules catch a substantial share of errors, they cost almost nothing, and they catch precisely the failures that confidence scores miss — a model being confident and wrong in a way a sum exposes immediately.

Two habits make the difference. Set the threshold per field rather than globally, since a wrong amount and a wrong description have completely different costs and one number cannot serve both. And sample the automatic path periodically, because the errors that passed review are the ones nobody is looking at and the only ones that reach your records.

Frequently asked questions

Why is a single accuracy figure misleading?

Because accuracy varies by field, by source and by document quality, and the critical fields — totals, line items, reference numbers — are frequently the hardest. The average conceals exactly what you need to know.

How should the review threshold be set?

Empirically from labelled data, per field and per source, by looking at what was actually wrong below each level. Tighter for anything feeding a payment or a regulatory record than for something feeding a dashboard.

What is the cheapest accuracy improvement?

Deterministic validation: arithmetic checks that line items sum to the total, lookups against known supplier or product lists, plausibility ranges, and cross-field consistency. It catches errors that confidence scores miss entirely.

Should a failed validation be corrected automatically?

No. Escalate it. A rule that silently fixes things hides the underlying problem and means nobody ever learns that a source has changed.

Which fields are hardest to extract?

Tables and line items, especially across page breaks; handwriting; nested or conditional fields; multi-currency and unusual number formats; ambiguous dates; and checkboxes, which are surprisingly unreliable and frequently important.

How do I know accuracy has not degraded?

Monitor per source rather than overall, since degradation is usually localised — a supplier changing accounting system changes all their documents. Watch the review rate, which rises before accuracy visibly falls.

What should be sampled?

The automatic path. Errors that passed review are the only ones reaching your records, and nothing else looks at them.

Can extraction remove the reviewer entirely?

No. It changes what they look at. Any implementation sold on eliminating the review queue is describing a narrow corpus rather than real intake.

Should I keep the original document?

Yes, linked to every extracted record for the retention period. Without it, checking a disputed value is impossible and the extracted data cannot be re-derived if a rule turns out to have been wrong.

Why record confidence alongside the value?

So a later investigation can distinguish a confident error from an uncertain one. They point at different causes, and without the score both look identical in the record.

What happens after extraction succeeds?

The destination system validates too, and rejections there arrive after extraction has reported success. Reconciling documents processed against records created is the check that catches what the two systems disagree about.

What should happen when a required field is missing from the document?

Fail visibly rather than writing a placeholder. Placeholders are the main route by which known-bad data enters a system and then becomes indistinguishable from real values.

Related reading

• AI document processing: classify before you extract
• AI workflow automation: when the model is needed
• Workflow automation for finance teams