A review procedure for checking automated transaction coding — what to sample, what to recompute, and the four failure patterns that never appear in a summary.
Any firm using automated transaction coding — a rule set in the ledger, a bought tool, or an external provider — carries the same exposure. The coding is applied at volume, it is applied consistently, and if it is wrong it is wrong consistently, in the same direction, for as long as nobody checks.
This is a procedure for checking it. It is written to be vendor-neutral, and it applies to our own output as readily as to anyone else’s; a firm that cannot audit what we produce should not be using it.
Why a summary tells you nothing
The natural review is to look at the totals. They balance, the categories look plausible against last year, and the exception count is low. Nothing there is evidence of correctness.
Systematic coding errors preserve balance. A payment between the client’s own accounts recorded as a cost leaves every reconciliation intact and inflates expenditure. A coding rule that matched a supplier name inside a longer word will have miscoded transactions consistently, so the category looks stable year on year — the stability is the error persisting, not the coding working.
What makes an audit informative is checking for specific mechanisms, each of which leaves a signature that a total conceals.
Check one: what each rule actually captured
For every automatic coding rule, list the transactions it matched, and read the list rather than the rule.
The failure this catches is a rule written as a substring match. A rule intended to catch a supplier name will also fire when that string appears inside a longer word — inside a different supplier’s name, inside a reference number, inside a description. It will do so silently and permanently.
Two things to look for in each rule’s captured set. Does it contain anything the author would not recognise as belonging there — a different supplier, an unrelated description? And is the count plausible for the frequency the supplier actually trades at? A rule matching several times the expected volume is matching something else as well.
Then run it in reverse: for each rule, list the transactions that should have matched and did not. Under-matching is invisible in exception reports because an unmatched transaction usually just falls to a default category and stops looking like anything.
Check two: transfers coded as costs
Identify every account the client controls, then look for payments between them.
Movements between a current account and a card account, or between two accounts of the same business, are not expenditure. Coded as costs they inflate spending — potentially by a large amount over a year, since card paydowns recur — while leaving the ledger balanced, which is why they survive review.
The signature is a matched pair: a debit in one controlled account and a credit of the same amount in another, within a day or two. That pattern is mechanically findable across the whole population and requires no sampling.
Check three: the arithmetic each document asserts
For any coding derived from extracted documents, recompute what the document itself claims. Subtotal plus tax should equal total. Line items should sum to subtotal. Tax should be a plausible rate of the net amount.
A failure here means the extraction was wrong before the coding was applied, and the coding decision was made on a bad figure. This check is independent of whatever produced the reading, which is what makes it worth running: it does not rely on the system being honest about its own confidence.
Watch the tax rate check specifically. A rate that is not one of the rates the jurisdiction actually uses is a strong indicator that a value was misread or that a registration identifier was captured as an amount.
Check four: document type before amount
Confirm that each source document coded as a cost is one.
Quotes, pro-forma invoices, order confirmations, delivery notes and payment receipts all resemble invoices and none of them is an expense. A payment receipt coded alongside the invoice it settles double-counts the cost. A quote that was never accepted is a cost that never existed.
The signature is duplication with a small date offset and an exact amount match, or a supplier whose recorded spend exceeds what was actually paid. Both are findable without reading every document.
Related: detecting duplicate invoices when receipts arrive as photographs, which covers the matching logic this depends on.
Check five: the shape of the exception list
Finally, audit the audit. Look at what the system flagged and — more informative — at what it did not.
An automated run that produced no exceptions on a real client’s books has not demonstrated accuracy. Real books contain ambiguity: documents that could reasonably be read two ways, transactions with no supporting evidence, dates that are genuinely uncertain. A system that resolved all of it either guessed or was never asked to notice.
So the useful questions are these. What proportion of items were flagged, and is that proportion credible for the state of the books? Are the flagged items ones a person would agree are ambiguous? And for any flagged item, can you trace the figure back to the region of the document it came from — because a value you cannot trace is a value you have to re-key in order to check, which removes the saving entirely.
What the checks cannot tell you
An honest procedure should name its own limits, or it becomes the thing it was written to guard against.
These five checks find mechanism errors — errors with a structural signature that repeats. They are very good at that, and it is where most of the value in an automated run is at risk, because a repeating error is the one that reaches a whole period rather than a single line.
They are correspondingly poor at one-off judgement errors. A transaction coded to a defensible but wrong account, where nothing about it is structurally unusual, will pass every check here. So will a misclassification that depends on knowledge the documents do not carry — that a particular purchase was personal, or that a supplier was engaged under a different arrangement this year. No mechanical check can reach those, and a vendor claiming otherwise is describing something that does not exist.
That is the honest division of labour. The checks compress the population to the items with a detectable problem; a person applies judgement to those, and applies sampling and client knowledge to the rest. What changes is not that review disappears but that it stops being spread evenly over thousands of records where almost nothing is wrong.
It also means the checks should be run in addition to a conventional review sample, not instead of one. A firm that replaces its sampling entirely with mechanical detection has swapped one blind spot for another, and the new one is harder to see because the report looks thorough.
Running it as routine
None of these checks require reading the ledger line by line, and none depends on trusting the system being audited. That is the property that makes them worth adopting permanently rather than running once after a scare.
Run them at the point coding is produced and before anything is posted, on every client, every period. What they return is a short list with evidence attached — which is a review your team can actually complete, rather than a sampling exercise that provides comfort proportional to the sample and no more.
The broader principle is one worth holding regardless of who supplies your automation: verify the deliverable, not the tool. Vendor assurances, accuracy percentages and internal confidence scores are all claims made by the thing being checked. Recomputed arithmetic, cross-referenced accounts and independent second readings are not. That argument is made at more length in verify the deliverable, not the library.
We build these checks as part of month-end close automation, and the same detection layer sits behind bookkeeping cleanup automation. But the procedure above is not proprietary and is not meant to be — it is how you should be checking any automation touching a client ledger, ours included.
Questions this raises
How large a sample should we audit?
For the targeted checks, none — those run across the whole population, because they are mechanical and cheap. Random sampling is only for the residual judgement question of whether the coding is sensible, and there a modest stratified sample across suppliers and months is more informative than a large random one, because errors are systematic rather than scattered.
Our tool reports a very high accuracy figure. Is this still necessary?
An accuracy figure is a property of the sample it was measured on, not of your books, and it says nothing about how the remaining percent is distributed. A small residual scattered randomly is noise. The same residual concentrated in a single supplier for a whole year is a restatement. The checks below distinguish those two cases; a headline figure cannot.
Can we run these checks without special tooling?
Most of them, yes — they are pivot tables and sorts against an exported ledger, and running them by hand once is a good way to find out whether they surface anything. Tooling matters when you want them run every month across every client, which is where doing it manually stops being realistic.
What if the audit finds errors in work we have already filed?
Then you have found them, which is the better of the two available outcomes. The procedure is deliberately designed to be run on historical work as well as current, because the systematic errors it targets are exactly the ones that persist unnoticed across periods.