AI data extraction: accuracy, confidence and the review threshold

A single accuracy figure from an AI data extraction vendor tells you little. Accuracy differs by field, by document quality and by source, and the fields that matter most (totals, line items, reference numbers) are often the ones extracted worst.

What makes an extraction system worth running is the review threshold: where you set the line between automatic and reviewed, how you chose that line, and what a wrong value costs downstream when nobody checks it.

What we'll cover

• Accuracy is a distribution
• What a wrong field costs
• Setting the threshold
• Building it with CodeWords
• Validating without a person
• Measuring it honestly
• The fields that are hardest
• Keeping it accurate over time
• Where the extracted data lands
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

Key takeaways

• AI data extraction accuracy varies by field, by source and by document quality, so one headline figure hides the fields that fail.
• Set the review threshold per field, using labelled data from your own documents, and tighten it for fields that feed payments or regulatory records.
• Validation rules such as arithmetic checks and list lookups catch errors that confidence scores miss.
• Sample the automatic path periodically, because errors that pass without review are the ones that reach your records.
• Automation changes what a reviewer looks at; it does not remove the reviewer.

Accuracy is a distribution

Extraction accuracy is a distribution across fields, sources and document types, not a single number. An average hides which fields fail and which suppliers send documents that extract badly.

Measure per field. A system that is excellent at dates and poor at line items has one average and two very different realities.

Measure per source. One supplier's documents extract cleanly and another's do not, and the average conceals which.

Measure per document quality. A clean native file and a photograph are different populations.

Expect the critical fields to be the hardest. Totals, line items and reference numbers are hard to extract, which is why the overall figure flatters.

Ask the vendor for the breakdown, not the headline. If a vendor cannot provide one, that tells you something.

Test on your own documents. A benchmark on someone else's corpus predicts little about your intake.

What a wrong field costs

A wrong field costs different amounts depending on the field, and that cost should set the review threshold. A wrong amount is expensive, and a wrong description is often harmless.

• A wrong amount flows into a payment, a report or a reconciliation.
• A wrong date can move an item into the wrong period, which surfaces at month end.
• A wrong reference breaks a link to a record and creates an orphan that someone investigates.
• A wrong description is often harmless.

Set the threshold per field. Applying one number to every field is the most common design error in extraction.

Setting the threshold

Set the review threshold empirically, from labelled data, not by choosing a round number that sounds prudent. Then check what was wrong below each level, because that shows what a given threshold buys you.

Accept a review rate. A system with no review queue either has a trivial task or produces wrong records silently.

Use different thresholds per field and per source, because the underlying accuracy differs.

Tighten for consequence. Fields that feed a payment or a regulatory record need a stricter threshold than fields that feed a dashboard.

Revisit the threshold. The level that was right for last year's document mix can be wrong for this year's.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations (CodeWords pricing page, 2026), and you make each connection once and reuse it.

1. Open CodeWords and start a new automation.
2. Describe to Cody, the automation builder, what should happen in plain language: what to extract, the validation rules, and what goes to review.
3. Authorize the connections: where documents arrive, and the system that receives the extracted data.
4. Describe the exceptions: a low-confidence field, a failed validation, a total that does not reconcile, an unreadable input.
5. Run it against documents you have already processed by hand and compare field by field.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows (CodeWords pricing page, 2026). Details are on the pricing page.

Validating without a person

Rule-based validation is the cheapest accuracy improvement, and teams often skip it. Each check below runs without a person and catches errors that a model's confidence score does not.

Check the arithmetic. Line items that sum to the total catch many extraction errors at no cost.

Check against a known list. Compare supplier names with your supplier records and product codes with your catalogue.

Check plausibility: a date in the future, an amount an order of magnitude from normal, a quantity that cannot be right.

Check cross-field consistency, such as a tax amount against a rate and a subtotal.

Escalate a failed validation instead of correcting it. A rule that silently fixes errors hides the underlying problem.

Use validation alongside confidence. A model can be confident and wrong in ways that arithmetic exposes immediately.

Measuring it honestly

Measure extraction accuracy with a labelled set built from your real documents, tracked per field and per source over time. Labelling is tedious and unavoidable.

Measure what got through, not only what was extracted. The errors that passed review are the ones that matter.

Sample the automatic path periodically. Silent errors live there, and nothing else looks there.

Count corrections in review as the ongoing signal. You already generate this data.

Publish the accuracy alongside the output. A figure presented as exact is trusted until someone checks it.

The fields that are hardest

The hardest fields to extract are tables, handwriting, conditional fields, unusual number formats, ambiguous dates and checkboxes. Review these fields more closely than plain text fields.

• Tables and line items, particularly across page breaks.
• Handwriting, including signatures and annotations on printed forms.
• Nested or conditional fields, where the meaning depends on another field.
• Multi-currency and unusual number formats, where a decimal separator convention changes the value by a factor.
• Dates without a clear format, which are ambiguous between conventions and silently wrong instead of obviously wrong.
• Checkboxes and marks, which are unreliable and often important.

Keeping it accurate over time

Keep extraction accurate by monitoring per source, watching the review rate and re-evaluating after any change. Treat accuracy as an ongoing measurement, not a one-off check before go-live.

Expect document formats to change. A supplier that changes accounting system changes everything about its invoices.

Monitor accuracy per source. Degradation is usually localised to one source, not general.

Watch the review rate. It rises before accuracy visibly falls.

Re-evaluate after any model or provider change. Behaviour can change even if nothing changed on your side.

Add corrected items to the labelled set. This keeps the set representative.

Where the extracted data lands

The destination system adds a second class of error, because it validates data too. Extraction can report success and the destination can still reject the record.

Expect destination rejections after extraction has reported success.

Watch for type and format mismatches, such as a date the destination will not accept or a currency field that expects a specific shape.

Decide what to do with required fields the document did not contain. You must choose between a placeholder and a failure, and placeholders are how bad data enters.

Write atomically where you can. A partially created record is worse than none.

Handle the rejection path. A document that extracted perfectly and failed to write is invisible in many implementations.

Reconcile counts between documents processed and records created. This check catches anything the two systems disagree about.

Making it survive

Make extraction reliable by validating critical fields, keeping source documents, recording confidence and reporting outcomes.

• Never write an unvalidated critical field, and define explicitly which fields are critical.
• Keep the source document linked to every extracted record, so checking stays possible.
• Record the confidence with the value, so a later investigation can tell a confident error from an uncertain one.
• Make extraction idempotent (safe to run twice) on the document, because the same file can arrive twice.
• Report the outcome: documents processed, fields extracted, validation failures, review rate and corrections made.

Limits worth knowing about

No system reaches perfect accuracy. A vendor that claims it has measured accuracy on a narrow corpus.

Confidence scores are imperfectly calibrated, so validate the threshold instead of trusting it.

Poor source quality sets a ceiling that no processing removes.

Labelled data is expensive to produce, and everything above depends on it.

Automation does not remove the reviewer. It changes what the reviewer looks at, and an implementation sold on removing review entirely overpromises.

What to build first

Build the validation rules first, before tuning extraction. Arithmetic checks, list lookups and plausibility rules cost almost nothing, and they catch the failures that confidence scores miss: a model that is confident and wrong in a way a sum exposes immediately.

Two habits make the difference. Set the threshold per field instead of globally, because a wrong amount and a wrong description have different costs and one number cannot serve both. And sample the automatic path periodically, because errors that pass without review are the ones that reach your records.

Frequently asked questions

Why is a single accuracy figure misleading?

A single figure hides variation. Accuracy differs by field, by source and by document quality, and the critical fields (totals, line items, reference numbers) are often the hardest. The average conceals the failures you most need to see.

How should the review threshold be set?

Set it from labelled data, per field and per source, by checking what was wrong below each level. Use a tighter threshold for fields that feed a payment or a regulatory record than for fields that feed a dashboard.

What is the cheapest accuracy improvement?

Rule-based validation is the cheapest. Check that line items sum to the total, look up values in known supplier or product lists, apply plausibility ranges, and test cross-field consistency. These checks catch errors that confidence scores miss.

Should a failed validation be corrected automatically?

No. Escalate a failed validation to a person. A rule that silently fixes errors hides the underlying problem, and nobody learns that a source has changed its format.

Which fields are hardest to extract?

Tables and line items (especially across page breaks), handwriting, nested or conditional fields, multi-currency and unusual number formats, ambiguous dates and checkboxes are hardest. Checkboxes are unreliable but often important, so review them closely.

How do I know accuracy has not degraded?

Monitor accuracy per source, because degradation is usually localised. A supplier that changes accounting system changes all of its documents. Also watch the review rate, which rises before accuracy visibly falls.

What should be sampled?

Sample the automatic path, the documents that skip review. Errors there are the ones that reach your records, and nothing else checks them.

Can extraction remove the reviewer entirely?

No. Extraction changes what the reviewer looks at, but a reviewer is still needed. An implementation sold on eliminating the review queue describes a narrow corpus, not real intake.

Should I keep the original document?

Yes, linked to every extracted record for the retention period. Without the original, checking a disputed value is impossible, and you cannot re-derive the data if a rule turns out to have been wrong.

Why record confidence alongside the value?

Recording confidence lets a later investigation tell a confident error from an uncertain one. The two point to different causes, and without the score they look identical in the record.

What happens after extraction succeeds?

The destination system validates the data too, and it can reject a record after extraction reports success. Reconcile documents processed against records created to catch anything the two systems disagree about.

What should happen when a required field is missing from the document?

Fail visibly instead of writing a placeholder. Placeholders are the main route by which known-bad data enters a system and becomes indistinguishable from real values.

Related reading

• AI document processing: classify before you extract
• AI workflow automation: when the model is needed
• Workflow automation for finance teams