Academy

Making AI reliable on real, messy documents

On a clean document, any model looks good. The difference is in the messy tail — where real documents live.

The Felesh team 5 min

Extracting information from a document always looks easy in a demo. You hand the model a clean invoice, and it pulls out the amount, the date, and the number correctly. But real documents are not like that. Skewed scans, broken OCR, layouts that differ from one organisation to the next, tables that have run together, sometimes handwriting. The difference between a demo system and a reliable one shows up exactly in that “messy tail.”

One invoice, two scansThe same invoice is extracted from a clean scan and a skewed scan. The clean scan gives total 4,380.50, date 2026-07-03 and supplier Nilgun Co., all correct. The skewed scan gives total 4,830.50, because OCR swapped the 3 and the 8, and date 2026-03-07, because the printed 03/07/2026 was read as 7 March. Its supplier is correct. The two wrong values carry no mark of doubt.fieldclean scanskewed scantotalprinted 4,380.504,380.504,830.503/8 swappeddateprinted 03/07/20262026-07-032026-03-07read as 7 Marsupplierin the headerNilgun Co.Nilgun Co.
The same invoice on two scans: the skewed one returns a wrong total and a swapped date, with no sign of doubt.

This is a general problem; its shape differs across domains — invoices and financial forms, insurance documents, administrative records, reports — but the core is the same: how do we make the model stay stable even on messy input? The answer is not a bigger model. It’s a better system.

Why reliability is not a property of the model

The common temptation is to think the problem is solved by a stronger model. But reliability isn’t something that lives only in the model’s weights; it is a property of the whole system — and that means it can be designed and measured.

A model might do fine on a clean document and misread a single figure on the same document with a bad scan. The most dangerous case isn’t that it makes a mistake; it’s that it makes the mistake with full confidence and you don’t notice. In a reliable system, the model is allowed to say “I’m not sure” — and the system knows what to do with that uncertainty.

Break the problem down, then measure

The first step is to break “document extraction” into specific fields and define, for each one, what “correct” means. A total amount is an exact figure; a date has a defined format; a party’s name must match the text of the document. Once you have defined correctness field by field, you can finally measure it.

A field with its rule and its source lineOn the left, the scan of invoice NF-0418: Invoice NF-0418, Subtotal 4,020.00, Tax 360.50, and Total due 4,380.50 on the fourth line, highlighted. A wire runs from that line to a field card on the right: field total, value 4,380.50, rule exact to the cent, source scan line 4.scan, page 1Invoice NF-0418Subtotal 4,020.00Tax 360.50Total due 4,380.50field: totalvalue4,380.50ruleexact to the centsource: scan line 4
A field is defined by its rule and the line it came from, so each value can be checked against both.

And measuring requires a real evaluation set: examples of the same messy documents you actually face, paired with the correct answer for each field. That set is your ground truth. You measure every change to the system against it: did this change make extraction better, or did it just look better on a few lucky examples?

The numbers we report are evaluation results on our own internal test set, not the outcome of a customer project. That distinction is fundamental to us: a number only means something when you know which set it was measured on and with what definition of “correct.”

A field is measured against the answer keyThe answer key and the model’s output for a delivery note are compared field by field. The cursor lands on quantity: 240 against 210, not equal. Then on delivery date: 2026-06-12 against 2026-06-12, a match. Then on supplier: Nilgun Co. against Nilgun Co., a match. Two right values do not fix the wrong one.fieldanswer keymodelcheckquantity240210not equaldelivery date2026-06-122026-06-12matchsupplierNilgun Co.Nilgun Co.matchper field: two right values do not fix the wrong one
Each field is measured against the answer key on its own, so one wrong value stays wrong.

Turn uncertainty into a feature

The most important design decision is this: what happens when the model is not sure? In a brittle system, uncertainty is hidden and the wrong output quietly moves on. In a reliable system, uncertainty is a signal: the uncertain field is flagged and set aside for human review.

This is what honest AI looks like. The system doesn’t need to be right about everything; it needs to know where it is sure and where it isn’t — and to surface that boundary instead of hiding it. Combining “automatic extraction on the confident cases” with “human review on the borderline cases” produces something that is both fast and dependable.

Unsure fields go to review, sure fields are savedFour fields come out of the skewed scan. Invoice NF-0418 and supplier Nilgun Co. are sure, and their wires draw into the saved box. Total 4,830.50 and date 03/07/2026 are unsure: a ring marks them, and their wires draw into the review box, where total and date wait for a person. Nothing in review is saved until a person sets it.extracted from the skewed scaninvoice no.NF-0418suresupplierNilgun Co.suretotal4,830.50unsuredate03/07/2026unsuresavedNF-0418Nilgun Co.reviewtotaldatenothing in review is saved until a person sets it
A field the model is unsure of is held out of the saved set and sent to a person; the sure ones go straight through.

The transferable idea

The core of this goes well beyond documents. Anywhere AI has to work on messy real-world input, the same principle holds: you don’t buy reliability, you build it and measure it. Define correctness precisely, measure it on real examples, and surface uncertainty instead of burying it.

On a clean document, any model is a hero. Trust is earned in the messy tail — and there, you win not with a bigger model, but with a better-measured system.