Backups you have actually tested
Most backup setups protect the wrong layer, and the gap only becomes visible on the day it cannot be fixed.
Back up state, not machines
In a containerised setup the containers are disposable. What is not disposable is the state: databases, uploaded files, object storage, and the configuration that describes how it all fits together.
A backup that images the whole server feels thorough and is usually both too big to restore quickly and missing the one thing you needed, because the database was mid write when the snapshot was taken.
Dump databases properly
Copying database files while the service is running produces a file that may or may not restore. Use the database's own dump tool, which produces a consistent snapshot. Do it per database, on a schedule, and check the output is a plausible size rather than a zero byte file created by a failing command.
Get it off the machine
A backup on the same server protects against a mistake, not against losing the server. Push to object storage somewhere else, with credentials that allow writing but not deleting history where possible.
Encrypt before it leaves, and store the encryption key somewhere that is not the thing being backed up. A key stored only on the server you are protecting is not a key, it is a coincidence.
The configuration is state too
Compose files, proxy configuration, cron entries, environment files. Losing these turns a two hour restore into a two day reconstruction from memory. Keep them in version control, and keep the secrets they reference somewhere deliberate.
Rehearse the restore
This is the part that gets skipped, and it is the only part that proves anything. Restore into a scratch environment and see whether the thing comes up.
You will find gaps every time: a database you forgot existed, a file mount nobody documented, a credential that lived only in someone's shell history. Better to find them during a rehearsal.
Monitor the backup itself
Backups fail silently more than anything else, because success and failure both produce no visible change. Have the job report completion and alert on the absence of that report. A backup you assume is running is not a backup.
Need help with any of this?
These notes are free and always will be. If you would rather someone just set it up, or you are stuck on something similar, get in touch at hello@opsira.io.