The most dangerous extraction error is the one that looks right. Modern document AI tools return clean, complete, schema-conformant output almost every time. Every field is present, every type is correct, and the record loads into the enterprise resource planning (ERP) system without complaint. Stakeholders see well-formed data and read it as accurate data. Our evaluation shows those are different properties, and the gap between them is wide. Across five extraction approaches tested on the same corpus, the four that produced structurally perfect output on every document still spread from 0.798 to 0.908 in field-level accuracy; the fifth failed structure visibly and scored 0.680 (Aicadium Phase 1 evaluation, 2026). This post explains what that gap is and why standard evaluation practice misses it. It also shows how to measure the gap using an evaluation set that contains none of your confidential data.
When an AI system extracts data from a document, its output is usually checked in two different senses. The first is structural. Does the output match the agreed schema, meaning the expected fields, types and format? Schemas are typically expressed in JavaScript Object Notation (JSON), a standard machine-readable data format. The second is factual. Does each value match what the document actually says? Automated systems, and most procurement processes, test the first sense because it is easy to test. The second sense requires ground truth, meaning a verified record of the correct answer for every field. Most organisations do not have that record for their own documents. The question is urgent now because tools built on large language models produce structurally perfect output almost universally, which makes them look interchangeable and finished. Decisions about which tool to buy, and whether human review can be reduced, are being made on that appearance. The commercial stake is direct. A wrong value inside a well-formed record flows into payment runs, certification decisions and financial reports without tripping any alarm. The error surfaces later, as a dispute, an audit finding or a compliance exposure. This post is written for the systems and transformation leaders who select and integrate these tools. It is written equally for the finance and quality owners who inherit the consequences when a confident number is wrong.
Schema adherence and field accuracy are different tests
The gap between well-formed and correct is not a technicality; it is two different tests, and only one of them is usually run. Setting them side by side makes the difference concrete.
Schema adherence asks whether the output has the right shape. Are all required fields present, are dates formatted as dates, are amounts numeric, does the record validate. This test is cheap, automatic and objective, which is why pipelines, demos and acceptance criteria lean on it.
Field accuracy asks whether each value is true to the source document. Is the invoice total the number printed on the invoice, is the line description the one on the page, is the tax the tax. This test requires knowing the correct answer independently, which means someone, or some process, must have established ground truth first.
The two tests fail in opposite ways. A schema failure is loud: the record does not load, someone investigates. An accuracy failure is silent: the record loads perfectly and the wrong number begins its journey through downstream systems.
Silent failures are only a theoretical concern until they are measured. So we measured them, on a corpus built specifically to make exact measurement possible.
Five extractors, one corpus: an 11-point spread behind perfect structure
The test put five extraction approaches up against the same documents, with a known answer for every field.
The corpus held 39 synthetic business documents: 21 invoices across five use cases and 18 quotes across four. Each document was rendered from validated structured data, so the correct answer for every field was known by construction. The approaches spanned direct model extraction and parser-based pipelines that convert documents to an intermediate format first.
On structure, the results split four to one. Four of the five approaches achieved perfect schema adherence, a score of 1.000: their output always had the right shape. The fifth validated on roughly one record in five, exactly the kind of failure schema checks exist to catch.
On accuracy, the four structurally perfect approaches spread from 0.798 to 0.908 in mean field score. The strongest performer still fell short of ground truth on the equivalent of roughly one field in eleven. One approach answered every document-level question correctly, 39 of 39, while scoring 0.826 on individual fields: broad reasoning about a document outran precise grounding in it.
Read those two results together. The one loud failure was caught; schema validation flagged the weakest tool immediately, and it was also the least accurate, at 0.680. But among the four survivors, whose output looked identical and finished, accuracy still differed by 11 points, and none reached a level where checking becomes optional. Schema adherence filtered out the worst approach and said nothing about the rest.
A spread that wide, hidden behind identical-looking output, distorts decisions. It is worth being precise about which decisions.
How the schema-accuracy gap misleads procurement and review planning
The first casualty is tool selection; the second is review staffing.
In procurement, demonstrations show structure, not truth. A vendor demo processes a handful of documents and displays clean, complete records. Without ground truth for those documents, the audience cannot distinguish a 0.80 tool from a 0.91 tool, because both produce output that looks the same on screen. Selection then drifts toward interface polish, price and claimed benchmarks measured on someone else’s documents.
In planning, well-formed output invites the assumption that review can be light. If the records validate and load, sampling a few per batch feels sufficient. But accuracy failures are silent and unevenly distributed, so light sampling systematically underestimates the error rate the organisation is absorbing.
There is also an engineering cost. When a wrong value surfaces downstream, the team that integrated the tool inherits the incident. Without stage-level evidence, the diagnosis defaults to the integration is broken. Errors can originate in document conversion, in schema interpretation or in the model’s reading; localising the failure requires evaluation infrastructure most deployments never built.
All three problems have the same root: no ground truth on your own document mix. That is the fixable part.
Ground truth without a corpus of real documents
Getting that record sounds like a confidentiality dead end. It is not.
Real invoices, test reports and contracts usually cannot be shared with vendors or annotation teams, and hand-labelling them at useful volume is unaffordable. The alternative is to synthesise the corpus. Author the documents as structured data first, in the same JSON shape the extraction should produce. Validate the invariants that make a document internally coherent: totals equal the sum of lines, quantities multiply against rates, references resolve.
Only then render each record into a realistic document, in both clean digital and image-only variants. The result is a set of documents whose correct answers are known by construction rather than by after-the-fact labelling. Ground truth is authored, not re-extracted, so the evaluation never depends on the tools it is meant to judge.
With such a corpus, the two tests separate cleanly. Schema adherence and field accuracy get measured independently, per approach and per field type, on documents that reproduce the structure of your formats without containing your data.
The evaluation habit matters more than any single result, because tools will keep changing. What should not change is the standard: output is correct when it matches ground truth, not when it merely validates.
Closing
Schema adherence and field accuracy are different properties, and the evidence shows how far they diverge. The only failure that announced itself was the one schema checks caught; among the approaches whose output looked identical, accuracy still spread by 11 points. Every decision made on the appearance of correctness, from tool selection to review staffing, absorbs that spread as invisible risk. The remedy is not distrust of the tools; it is measurement on your own document mix, made practical by synthetic ground truth. The economics that make this measurement urgent are set out in the opening post of this series, and the design standard it supports in the closing one, The citation is the deliverable.
