The backup said OK and there was no offsite copy
A backup job can succeed at the part you watch and fail at the part that matters. Ours did, and the retention prune kept running against a remote nothing was reaching.
Our nightly backup dumps every database on the box, ships the folder to object storage, prunes old copies at both ends and pings a phone. On a Tuesday the upload started failing. The dumps kept working, so the job kept reporting DONE with 1 error(s), and the alert said one step had failed. Both statements were true. Neither said the thing that mattered, which was that there was no longer an offsite copy of anything.
It ran that way for five nights before anyone looked at it properly.
The part that did real damage
The retention prune was not conditional on the upload. Every morning the job would fail to transfer the new day, then walk the remote listing and delete anything past the retention window. So the remote was being trimmed from the oldest end while nothing new arrived at the newest end.
That is worse than a stalled backup. A stalled backup leaves you with yesterday. This arrangement had a fixed expiry date: the offsite window shrank by one day every day, and on current trend the last surviving copy would have been deleted about four weeks later, with the log still reporting a single routine error each morning.
Any cleanup step that runs unconditionally after a step that can fail will eventually run in exactly the circumstances where you needed it not to. Gate the destructive half on the result of the productive half.
Why the alerting did not help
The job counted failed steps and alerted when the count was above zero. That is a reasonable design, and it is why nobody reacted: the message it produced was indistinguishable from the message you get when a single unimportant dump fails, which had happened before and been fine.
Failures are not interchangeable. A missing dump of a cache database and the total absence of offsite storage should not produce the same sentence. We split the offsite failure into its own alert with its own wording, saying plainly that the local dump succeeded and there is no offsite copy of today.
The cause was somewhere else entirely
Storage had hit a cap. It hit the cap because a workflow engine on the same box was recording the full request and response of every automation run, and a small prototype had started calling four of its endpoints once a minute. That is under six thousand runs a day, each one storing a couple of hundred kilobytes it would never be read for.
The nightly dump of that one database went from about 47MB to 130MB to 460MB across a week. Nothing connected the two events. Somebody added a one line scheduled job to a side project, and disaster recovery for everything else on the machine stopped working thirty-six hours later.
Watch the size, not just the status
Every check we had was boolean. Did the dump run, did it produce a non-empty file, did the step exit zero. All of those stayed green throughout, because none of them were false.
The signal was in the trend. A backup that quietly triples in a week is telling you something changed upstream, and it is worth an alert on its own, well before it collides with a quota. Record the size of each artefact in the log and compare it against the last run.
Worth checking on your own setup
- Does your retention step run when the transfer step failed?
- Does your success alert distinguish a partial failure from having no offsite copy at all?
- Do you know the size of last night''s backup, and the one from a month ago?
- If your storage has a cap, does anything tell you before the cap tells you?
The general lesson is not about any one tool. A backup has two halves, making the copy and putting it somewhere else, and only the first half is easy to observe. A log line confirming the first half is not evidence of the second. Check the transfer, not the summary.
Need help with any of this?
These notes are free and always will be. If you would rather someone just set it up, or you are stuck on something similar, get in touch at hello@opsira.io.