Alerting that people actually read
The failure mode of alerting is not missing alerts. It is sending so many that nobody looks, and then missing the one that mattered.
Start from what you would want to be woken for
Almost nothing qualifies. Work backwards from there rather than forwards from what is technically observable.
A useful division:
- Money events. A sale, a refund, a payment that could not be matched. Low volume, high interest.
- Broken automation. Something that was supposed to run and did not, or ran and failed. Nobody notices these otherwise, which is exactly why they need to shout.
- Exceptions needing a decision. An unmatched payment, an item overdue past the threshold, a customer reporting a fault.
Everything else belongs in a dashboard you choose to look at.
Self-hosted push is enough
A small notification server gives you push to a phone with a topic per concern and no per message cost. Publish with a plain HTTP request from anywhere, which means every script and workflow can alert without a client library.
Use a separate publish only credential for anything that only sends. Nothing that publishes should be able to read or reconfigure.
Make the message self contained
The test is whether you can act, or decide not to, from the notification alone. Include the identifier, the amount, the customer, the reason. A message that says something needs attention and requires opening a dashboard to find out what has failed at its job.
Use priority properly, and sparingly. If everything is urgent, nothing is, and you will mute the topic.
Alert on the absence of things
The hardest failures are silent. A nightly job that stops running produces no error because it produces nothing at all. Have long running jobs report completion, and alert when an expected completion does not arrive.
Prune ruthlessly
Alerting decays. A notification that was interesting at three sales a week is noise at thirty a day. Every few months, ask of each alert whether you have ever acted on it. If not, delete it. The value of the channel is entirely in your willingness to look at it.
Need help with any of this?
These notes are free and always will be. If you would rather someone just set it up, or you are stuck on something similar, get in touch at hello@opsira.io.