Why "99% OCR accuracy" tells you nothing useful
An accuracy percentage is a claim about documents you have never seen. Here is a check you can run on the document in front of you.
Survey the document extraction market and you will find the same number everywhere: 99%. Sometimes 99.8%. Occasionally 99.9%. The numbers are not comparable, not audited, and not measured on your documents.
Three reasons the number is meaningless
- Accuracy of what? Character-level accuracy, field-level accuracy and document-level accuracy differ by an order of magnitude. 99% character accuracy on a 12-digit IBAN means roughly one IBAN in eight is wrong.
- Measured on whose documents? Vendor benchmarks run on vendor test sets. Your Italian corporate statements with three-line SEPA descriptions are not in it.
- Averaged over what? A vendor processing mostly clean digital PDFs will report a higher number than one processing scans, without being better at either.
The most honest figure published in this market is not 99% — one established vendor reports 92.5% average across real customer use cases, and another concedes that raw OCR alone reaches about 85% before human review. Those numbers are lower and considerably more informative.
What to measure instead
For financial documents there is an arithmetic identity you can test on every single document, without a labelled dataset and without trusting anybody:
Opening balance + credits − debits = closing balance
Both endpoints are printed on the statement by the bank. If extracted rows reproduce the printed closing balance, the row set is complete and the amounts are right. If they do not, you have a specific, quantified discrepancy to investigate.
This is a control, not an estimate. It is either satisfied or not, per document, and it does not care what any vendor claims.
What reconciliation does not catch
Being honest about the limits matters more than overselling the check:
- Two compensating errors — a debit and a credit both misread by the same amount — will still balance. Rare, but possible.
- A misread description or counterparty does not affect the arithmetic at all.
- A statement with no printed opening or closing balance cannot be reconciled, and the honest answer there is "unknown", not "reconciled".
Which is why per-row confidence still matters alongside it. Reconciliation proves the totals; confidence flags the individual rows worth a human glance. Together they cover most of what an accuracy percentage pretends to.
What to ask a vendor
Replace "what is your accuracy" with three better questions: Do you return a reconciliation result as a machine-readable field? Do you flag individual low-confidence rows rather than silently dropping them? And can I test it on my own worst documents before committing?
A vendor comfortable with all three is telling you something real. A vendor who answers only with a percentage is telling you about their marketing.
Frequently asked questions
Is OCR accuracy completely irrelevant?
No — it matters for scanned documents, where character recognition is the bottleneck. It is just not a useful basis for comparing vendors, because nobody measures it the same way.
What is a good reconciliation rate in practice?
On machine-readable statements, near total. On scanned documents it depends heavily on scan quality. The useful metric is what proportion of your own documents reconcile, which you can measure yourself in a week.
What should happen when a statement does not reconcile?
It should never be posted automatically. The right behaviour is to surface the discrepancy and the flagged rows for a human, which is far cheaper than finding the error at year end.
Does human-in-the-loop review fix this?
It raises accuracy and it also means people read your documents. That is a real privacy and cost trade-off, and worth an explicit decision rather than a default.
Try it on your own statements
Free tools with no account, or a workspace with an API and free documents to evaluate it.