Validation rules that fire on correct data
An obviously true rule about order documents turned out to be false for nine per cent of genuine ones. It was only obvious until it was measured.
What happened
We were replacing a paid document processing service with our own pipeline: purchase orders arrive by email as PDFs, get parsed, and become sales orders in an inventory system after a person approves them.
Getting the extraction right is the easy half. The hard half is deciding when to refuse, because the failure this kind of system has to prevent is not a missing order. It is an order that looks perfectly fine and is quietly wrong.
So we wrote guards. One of them looked beyond argument: a delivery date cannot be earlier than the date the order was raised.
Before shipping any of it we ran the whole pipeline over three weeks of real historical documents. Five hundred and thirteen of them. That rule rejected forty six.
They were not misparsed
Every one of those documents was read correctly. The buyer simply raises a proportion of their purchase orders retrospectively, one to seven days after the goods have already gone out. Forty different delivery sites did it. It is not an error, an exception or a system fault at their end. It is just how they work, and nobody involved had ever thought to mention it because to them it is unremarkable.
Had that rule shipped, one order in eleven would have been held for a human who would have looked at it, found nothing wrong, and released it.
Why that is worse than no guard
A review queue only works while the things in it are worth reading. Fill it with items that are always fine and people learn, correctly and quickly, that the queue does not mean anything. Then the one genuine failure arrives and gets released with the same glance as everything else.
A guard with a high false alarm rate does not just waste time. It disables every other guard sharing the same queue.
The distinction that actually predicts a good rule
Looking at which guards earned their place, the split was clean.
The rules derived from the document were all worth having. Every one of these documents prints its own order total, so the extracted lines can be summed and compared against it. That check caught a real bug during development: a filter written slightly too narrowly silently dropped entire product lines from two orders, and the totals disagreed immediately. Same for checking quantity times unit price against the printed line total, which is what stops a hundred being read as a ten.
The rules that encoded an assumption about the world were the ones that misfired. Nothing in the document says delivery follows ordering. We supplied that, from intuition, and intuition was wrong.
Ask where a rule gets its authority. If the document asserts it, keep it and fail hard. If you are asserting it on behalf of the document, treat it as a note rather than a rejection until real data says otherwise.
Run every rule over history before it goes live
This cost nothing to find. The mailbox had three weeks of real documents sitting in it, and running the finished pipeline over all of them took a few minutes. Without that pass the rule would have looked completely sensible on the desk, gone live, and produced a slow drip of false alarms that would have taken weeks to be recognised as a pattern rather than bad luck.
If you are building anything that classifies or rejects, you almost certainly have a backlog of real examples somewhere. Point the finished thing at all of them and count what it refuses. Then read the refusals, because the ones that turn out to be correct documents are telling you your model of the world has a gap in it.
The other half: what else is in the mailbox
The same exercise turned up something nobody had scoped for. Five per cent of the messages in that mailbox were not purchase orders at all. They were people writing to a human: a late delivery, damaged goods, a question about stock, an out of hours escalation.
Any mailbox that feeds an automated pipeline will carry these. Somebody has the address, so somebody will write to it.
The trap is that a document pipeline naturally has two outcomes, processed and failed, and neither is right for a person asking a question. Sent down the failure path they sit in a technical queue nobody reads as correspondence. Filtered out as unrecognised they vanish entirely, and the sender concludes they were ignored.
Sorting has to be a first class step with its own destination for human mail, and the default has to be that anything not unmistakably machine generated goes to a person. Being wrong in that direction costs somebody thirty seconds. Being wrong in the other direction loses a complaint.
Worth checking on your own setup
- Which of your validation rules came from the data, and which came from an assumption you made?
- Have you run the current rules over a month of real history and read what they reject?
- What happens to a message arriving at an automated address that is not the thing it expects?
- If your review queue is mostly items that turn out fine, does anyone still read it properly?
Need help with any of this?
These notes are free and always will be. If you would rather someone just set it up, or you are stuck on something similar, get in touch at hello@opsira.io.