Receipts photographed on site were restating invoices already filed as PDFs. The pairs shared no file characteristics, so no review of the folder could surface them.
Where documents arrive through more than one channel, the same expense will eventually arrive through two of them. Nothing about the two files matches, which is precisely why the duplication survives review.
The situation
Supplier documents reached the accounts folder three ways: emailed PDFs from suppliers, scans of posted paper, and photographs taken on site by people who would not see the office for days. Every document was legitimate. Some of them described the same expense twice.
The pattern
Consider one expense arriving as a crisp emailed PDF and, separately, as a photograph of the paper copy taken at an angle in poor light. The two files share nothing: different filenames, different formats, different dates received, different dimensions, different byte content. Deduplication by filename or file hash cannot find them, because they genuinely are different files.
They share only what they say — the same supplier, the same amount, the same date, the same line items. That is the only level at which the match exists, so that is the level the comparison has to work at.
Two further complications make naive matching unsafe. Recurring charges from a single supplier are often genuinely identical in amount month after month, so amount and supplier alone cannot decide the question. And a photographed document has been through OCR, so its extracted amount may differ from the PDF’s by a misread digit — meaning the comparison must tolerate small differences while still treating a factor-of-ten gap as a disagreement rather than a match.
The output is therefore a ranked list with the evidence for each pair shown, not an automatic deletion. Deciding that two documents are the same expense is a judgement with financial consequences; the system’s job is to put the candidates in front of a person who can make it.
Engineering takeaway — Match documents on the facts they contain — supplier, amount, date, line structure — never on file identity. Any deduplication that operates on filenames or hashes will report a clean folder while the double-count sits inside it.