Document AI is not one problem but five: digitising the page, classifying the document, extracting the fields, matching what you extracted to your own master data, and interpreting content where the answer is not text at all. Naming which one you have decides the tooling, the accuracy you can reasonably expect and where a person belongs in the process.
Getting that wrong is expensive in a specific way. Teams buy a tool that solves step one, discover the difficulty was in step four, and conclude that document AI does not work. In our own delivery, document processing is the single largest category of build work, and the projects that stall almost never stall on reading the page.
This article sets out the five problems, how to tell them apart, what accuracy is realistic for each, and why matching deserves far more scoping attention than it usually gets.
Why does naming the problem matter?
Because each of the five has a different failure mode, a different cost profile and a different answer to the question of whether a specialist tool or a general model fits better. Treating them as one thing means buying for the wrong constraint.
It also changes what you measure. Accuracy on digitising is a character-level question. Accuracy on classification is a confusion matrix. Accuracy on extraction is per field, not per document, because a document with nine correct fields and one wrong one is not ninety percent right in any way that helps the person receiving it. Accuracy on matching is measured against your own reference data, which means it depends on the quality of that data as much as on the model.
RAND’s study on the root causes of failure for AI projects, based on interviews with sixty-five experienced practitioners, puts inadequate data and misunderstood problem definition among the leading causes. Document work is where both show up most literally. The data is the documents, and the problem definition is which of these five steps you are actually solving.
What are the five document AI problems?
Digitising means getting text off the page. This is the step people picture when they say OCR, and for clean digital documents it is largely a solved problem. It becomes genuinely hard again with scans of poor quality, handwriting, photographs taken at an angle, and forms where the layout carries meaning. If your input is a phone photo of a maintenance docket rather than a system-generated PDF, this step is not free.
Classifying means deciding what kind of document you are holding. Inbound post that mixes invoices, complaints and contracts needs sorting before anything else can happen, and archiving work is largely classification plus metadata. This step is generally reliable when the categories are few and clearly distinct, and gets difficult when categories overlap or when the distinction depends on content buried deep in the document.
Extracting means pulling specific values out: invoice number, date, line items, rates, payment terms. This is what most people mean by document AI. On structured, repetitive documents it works well. The difficulty scales with variation, so a single supplier’s invoices are straightforward and two hundred suppliers’ invoices are a different project.
Matching means mapping what you extracted onto your own master data. The supplier name on the document has to become a supplier record in your system. The described service has to become an item in your catalogue. The line has to land on a general ledger account. This is the step that quietly consumes projects, and the next section is about why.
Interpreting covers cases where the answer is not written anywhere on the page. Counting bends and T-joints in a technical drawing to estimate insulation, or reading a floor plan, means reasoning about geometry rather than reading text. This is the hardest of the five and the one where feasibility genuinely has to be tested rather than assumed, particularly when every client draws differently and there is no shared convention to learn.
Why does matching cause more failures than reading?
Because reading has a correct answer and matching often does not.
An extraction step can be verified against the document itself: either the rate on the page is one hundred and twenty euros or it is not. A matching step has to bridge two vocabularies, and the target vocabulary belongs to your organisation. In a contract intake project we scoped recently, the hardest part was not reading supplier contracts, which arrive as PDFs in several languages and convert reasonably well. The hard part was that the service name usually is not stated in the contract at all, and the supplier refers to itself differently from how the receiving ERP does. Nothing on the page tells you which catalogue entry it belongs to.
The same pattern shows up in finance. On an invoice automation project, determining the correct general ledger account for a line is not an extraction problem, because the account number is not printed on the invoice. It is inferred from booking history, from the supplier, from what was done with similar lines before. Once we benchmarked a full year of invoices, roughly seven thousand of them, the system processed eighty-eight percent straight through, and the lines without any annotation landed on the right ledger account in ninety-three and a half percent of cases against that history. Those are matching numbers, not reading numbers, and they took the majority of the effort.
Two practical consequences follow. First, the quality of your master data sets a ceiling on matching accuracy, so a catalogue with duplicate entries and inconsistent naming will cap the system regardless of the model. Second, matching is where a feedback loop pays for itself, because every correction a person makes is a labelled example of exactly the ambiguity the system could not resolve.
How do you tell which problem you have?
Four questions, answered about the actual documents rather than about the ideal case.
- Is the input machine-readable already? If it arrives as a system-generated PDF, digitising is not your problem. If it arrives as scans or photographs, it is, and that changes both tooling and cost.
- Does the incoming stream contain more than one kind of document? A single stable document type removes the classification step entirely. Mixed inbound post makes classification the first thing you build.
- Do the values you need exist verbatim on the page? If yes, you have an extraction problem, which is the most tractable of the five. If the answer has to be derived, inferred or looked up somewhere else, you have a matching problem and should scope it as the main body of work.
- Is the answer expressed in text at all? Drawings, plans and diagrams put you in the interpreting category, where the honest first step is a feasibility check on your own real documents rather than a proposal.
One further question applies in Europe. If the documents drive a decision about a person, such as an application or an entitlement, the system may fall under the AI Act’s rules for high-risk systems, which change the documentation and oversight you owe. That is a scoping question rather than a technical one, and it is cheaper to answer at the start.
Most real projects contain several of these. That is fine and worth stating explicitly, because a project described as “invoice extraction” that is really extraction plus matching will be planned at roughly half its true size.
What accuracy should you expect?
Different numbers per step, and quoting a single figure for a whole pipeline is how expectations get set wrong.
Digitising clean digital documents is effectively solved. Classification into a small number of clearly distinct categories is reliable enough that it is rarely the constraint. Field extraction on structured documents reaches the high nineties per field, provided the field is genuinely present and unambiguous. Matching drops noticeably, and a straight-through rate in the high eighties on a real mixed stream is a good outcome rather than a disappointing one. Interpreting drawings has no useful benchmark, which is itself the answer: test it before you promise it.
Two things matter more than the headline percentage. Measure per field rather than per document, because the fields differ enormously in difficulty and an aggregate hides which one is hurting you. And separate the confidence question from the correctness question, because a system that knows when it is unsure is far more valuable operationally than one that is slightly more accurate but uniformly confident.
Where does the human belong?
On the exceptions, not on everything. Reviewing all output regardless of confidence removes most of the benefit and turns automation into a second pair of eyes on work that was already being done.
The design that works is confidence-based routing. Items above the threshold pass through, items below it go to a person, and unmatched values go to a person by definition because there is nothing to pass through to. Where the receiving system is financial or otherwise unforgiving, a review step before write-back is sensible regardless of confidence on the first batches, then tightened once the numbers hold up.
What matters most is what happens to the correction. A correction that is applied and then forgotten is a missed opportunity, since those corrections are the cheapest training signal available and they cluster exactly where the system is weakest. Design the capture of corrections at the start, because retrofitting it means throwing away months of the most useful data you will ever have.
The question to ask before you buy anything
Take twenty real documents from the stream you want to automate, not twenty tidy examples, and walk them through the five steps by hand. Note which step each one gets stuck on. That exercise takes an afternoon and tells you more about what you need than any vendor demonstration, because a demonstration is always run on documents chosen to look like step three.
If most of your twenty get stuck at step four, you do not have a reading problem. You have a master data problem wearing a document AI costume, and the fix starts somewhere other than a model.
Not sure where your bottleneck lies? DataNorth’s AI Feasibility Assessment helps evaluate your data, technical requirements and implementation complexity before you invest in a solution.
Once the problem is clear, our AI Development services help turn that understanding into custom AI solutions integrated with your existing systems. For a practical example, explore our Sales Order Automation solution, which connects document extraction with business data and ERP workflows.
Frequently asked questions about document AI
What is the difference between OCR and data extraction?
OCR converts an image of text into characters, answering what the page says. Extraction identifies which of those characters are the invoice number, the date or the rate, answering what the page means. A perfectly digitised document can still fail extraction if the fields are ambiguous.
What accuracy can you expect from document AI?
It depends entirely on the step. Digitising clean digital documents is close to solved, classification into a handful of categories is reliable, and field extraction on structured documents reaches high nineties. Matching extracted values to your own master data is where accuracy drops, often into the eighties.
Why do document AI projects fail?
Rarely because the model cannot read the page. Failures cluster around three things: the extracted value has no clean equivalent in the receiving system, exceptions were never scoped, and nobody defined what a correct answer looks like before building started.
Do you need a specialist tool or a general model?
For high-volume standard documents in a single format, a specialist tool is usually cheaper and more predictable. For varied documents, mixed languages or documents needing judgement, a general model with a validation layer handles the variation better. Many production systems combine both.
Where should the human sit in a document AI process?
On the exceptions rather than on everything. Route low-confidence and unmatched items to a person and let the rest pass, then use the corrections as a signal for what to improve. Reviewing all output regardless of confidence removes most of the benefit.
