A photographed receipt and the supplier's emailed PDF of the same expense share nothing as files. What to match on instead, and how to handle recurring charges.
A photographed receipt and the supplier’s emailed PDF of the same expense share nothing as files. Different names, different formats, different dates received, different byte content. Every deduplication tool that works on filenames or checksums will report the folder as clean.
They share only what they say. That is the level the comparison has to work at.
Why this is a firm’s problem rather than a client’s
Duplication is not a sign of a careless client. It is a structural consequence of having more than one intake channel, and firms have spent the last few years adding channels deliberately — a receipt capture app, a supplier email address, a portal upload, a monthly scan of whatever came on paper. Each one improves collection, and each one raises the probability that the same expense arrives twice by different routes.
The cost lands on the firm in two places. The obvious one is overstated expenditure, which is a correction and possibly a conversation with the client. The less obvious one is review time: a folder whose duplicate status is unknown has to be treated as suspect throughout, so a member of staff reads documents that are almost all fine in order to find the few that are not. That is the expensive pattern — qualified attention spread evenly across a population where the interesting cases are a small fraction.
The aim of matching on content is not to eliminate the review. It is to compress it from a folder into a short ranked list, so the attention lands where the doubt actually is.
What to match on
Four fields, together, identify an expense with enough precision to be useful:
- Supplier — normalised, because the same supplier appears as
ACME LTD,Acme Limitedandacme ltd.across three documents. - Amount — with a small tolerance, because one of the two documents has been through OCR.
- Date — with a few days’ tolerance, because the invoice date and the date on the till receipt are not always the same day.
- Line structure — the number of lines and their individual amounts. This is what separates a genuine duplicate from a coincidence.
Any one of these alone produces noise. Supplier and amount together, within a date window, is the practical starting point; line structure is what raises confidence from “possible” to “almost certainly”.
The two cases that break naive matching
Recurring charges. A monthly subscription from one supplier is genuinely identical in amount, month after month. Supplier and amount alone will flag every month against every other month. The date window is what rescues this — but only if the window is smaller than the recurrence interval. For a monthly charge, a window of a few days works; for a weekly charge, it has to be tighter than the week.
OCR drift. The photographed copy has been read by a machine, so its extracted amount may differ from the PDF’s by a misread digit. The tolerance on amount has to absorb that. But it must absorb it asymmetrically: a difference of a few cents is plausibly the same expense, and a difference of a factor of ten is a misread decimal point that should be flagged as a disagreement rather than silently treated as a match or a non-match.
Rank, do not delete
The output should be a ranked list of candidate pairs with the evidence for each shown — the matched supplier, the two amounts, the two dates, the line comparison — and the decision left to a person.
Deciding that two documents are the same expense has financial consequences, and the cases that are genuinely ambiguous are exactly the cases an automatic rule gets wrong. The value of the automation is in producing a short, ordered list of things worth looking at, out of a folder nobody had time to read. It is not in making the call.
That principle generalises well beyond duplicates: a system that reports what it could not resolve is more useful than one that resolves everything and tells you nothing.
Three cases that look like duplicates and are not
A matching process is only as useful as its false-positive rate, because a list that is mostly wrong gets ignored within two months. Three patterns produce pairs that satisfy every criterion above and are legitimately two separate expenses.
The split payment. One invoice settled in two instalments of equal value, a fortnight apart. The supplier matches, the amounts match exactly, the dates fall inside a generous window. What distinguishes it is that both documents reference the same invoice number while being payment records rather than invoices — so document type, not amount, is what separates them.
The genuine repeat purchase. A trade counter visited twice in a week for the same consumable at the same price. Nothing about the documents differs except the time printed on them. The line structure is identical because the purchase was identical. Here the only reliable discriminator is a timestamp finer than the date, which is why a card slip or till receipt carrying a time is worth extracting even though nothing else needs it.
The corrected reissue. A supplier issues an invoice, spots an error, and reissues with the same total under a new number. Both documents are real, only one is payable, and the correct handling is not deduplication but establishing which supersedes the other — usually decidable from the invoice numbers and any credit note issued alongside.
None of these should be resolved automatically, and all three are obvious in a second to a person looking at both documents. That is the argument for the ranked list in its strongest form: the mechanism is good at finding candidate pairs and genuinely bad at the final judgement, so the design should play to the first and refuse the second.
What the check looks like in practice
Run it at intake rather than at reconciliation. Every input it needs is a property of the folder, so there is nothing to wait for, and a duplicate caught before coding costs a fraction of one caught after posting:
- Normalise supplier names before comparing anything. The same supplier appears in three spellings across two documents, and no matching works until that is collapsed.
- Group by supplier and amount, with a tolerance sized for OCR drift rather than for genuine variation — a few units in the last digit, not a percentage.
- Apply a date window smaller than the recurrence interval of anything that recurs. This is the check that separates a genuine duplicate from a monthly subscription.
- Compare line structure within each surviving group: the number of lines and their individual amounts. This is what raises a pair from possible to near-certain.
- Rank by confidence and attach both documents. The output is a queue, not a decision.
- Treat order-of-magnitude discrepancies as disagreements, not as non-matches. A factor of ten is a misread decimal point and deserves a look for its own sake.
Steps one to three take minutes to run across a year. Step four is what makes the list short enough to be worth reading.
The wider pattern
Duplicates are one of six structural causes of a folder-to-ledger difference, and they are usually the largest by value — worth reading alongside why supplier invoice totals do not match, which covers the others and the order to test them in.
Where the volume makes this impractical by hand every month, it is built into invoice data entry automation and forms part of the detection layer in bookkeeping cleanup automation.
Questions this raises
Our accounting package already has duplicate detection. Why is this different?
Built-in detection generally compares file identity or exact field equality, which finds the easy case where the same file was uploaded twice. The expensive case is two different files describing the same expense — a photographed receipt and the supplier PDF — which share no field exactly and are invisible to that comparison.
How much duplication is normal in a client folder?
It scales with the number of intake channels rather than with the client. A business that emails everything to one address produces very little. One where staff photograph receipts, suppliers email PDFs and someone scans paper monthly will produce duplicates continuously, because the same expense genuinely arrives twice through different routes.
Why not delete duplicates automatically once confidence is high enough?
Because the cases that are genuinely ambiguous are the cases a threshold gets wrong, and a wrongly removed expense is both a financial error and an invisible one. A ranked list takes a person seconds per pair with both documents in front of them, and it leaves a record of the decision.
Does this work when one copy is a poor photograph?
That is the case it is built for. The amount tolerance absorbs OCR drift, and line structure carries the confidence when individual fields are unreliable. A large discrepancy — a factor of ten rather than a few units in the last digit — is treated as a misread decimal point and flagged as a disagreement rather than resolved either way.