Detecting duplicate invoices when receipts arrive as photographs

A photographed receipt and the supplier’s emailed PDF of the same expense share nothing as files. Different names, different formats, different dates received, different byte content. Every deduplication tool that works on filenames or checksums will report the folder as clean.

They share only what they say. That is the level the comparison has to work at.

What to match on

Four fields, together, identify an expense with enough precision to be useful:

• Supplier — normalised, because the same supplier appears as ACME LTD, Acme Limited and acme ltd. across three documents.
• Amount — with a small tolerance, because one of the two documents has been through OCR.
• Date — with a few days’ tolerance, because the invoice date and the date on the till receipt are not always the same day.
• Line structure — the number of lines and their individual amounts. This is what separates a genuine duplicate from a coincidence.

Any one of these alone produces noise. Supplier and amount together, within a date window, is the practical starting point; line structure is what raises confidence from “possible” to “almost certainly”.

The two cases that break naive matching

Recurring charges. A monthly subscription from one supplier is genuinely identical in amount, month after month. Supplier and amount alone will flag every month against every other month. The date window is what rescues this — but only if the window is smaller than the recurrence interval. For a monthly charge, a window of a few days works; for a weekly charge, it has to be tighter than the week.

OCR drift. The photographed copy has been read by a machine, so its extracted amount may differ from the PDF’s by a misread digit. The tolerance on amount has to absorb that. But it must absorb it asymmetrically: a difference of a few cents is plausibly the same expense, and a difference of a factor of ten is a misread decimal point that should be flagged as a disagreement rather than silently treated as a match or a non-match.

Rank, do not delete

The output should be a ranked list of candidate pairs with the evidence for each shown — the matched supplier, the two amounts, the two dates, the line comparison — and the decision left to a person.

Deciding that two documents are the same expense has financial consequences, and the cases that are genuinely ambiguous are exactly the cases an automatic rule gets wrong. The value of the automation is in producing a short, ordered list of things worth looking at, out of a folder nobody had time to read. It is not in making the call.

That principle generalises well beyond duplicates: a system that reports what it could not resolve is more useful than one that resolves everything and tells you nothing.