Quadratic growth, a string limit, and a week long crash loop
A long running process was down for seven days and restarted twenty six thousand times. Two ordinary decisions combined into that.
The first mistake: a per-pair store with no cap
The application kept one record per pair of entities. That scales with the square of the population. At a few hundred entities it is unremarkable. At a few thousand it was nearly six million records and over 500MB, while everything else in the system totalled under 5MB.
Population was not the problem. The quadratic store was. Nobody had noticed because it grew slowly and invisibly, right up until it crossed a hard limit.
The limit it crossed
State was persisted by serialising the whole world to JSON. JavaScript engines cap the maximum length of a single string, somewhere around 512MB. Once the snapshot exceeded it, serialisation threw:
RangeError: Invalid string lengthThere is no gradual degradation. It works, then it does not.
The second mistake: throwing from a timer callback
The save ran inside a repeating timer. An uncaught throw there kills the process. The container was set to restart unless stopped, so it came straight back up, ran the same tick, hit the same limit, and died again.
One failed save became an infinite crash loop. Nothing alerted, because the service was technically always about to be running.
The fixes, in order of importance
- Cap the store. Keep only the strongest N entries per entity, and only prune once the total passes a soft threshold so the cost is not paid every tick. Check what actually reads the data first: here nothing needed more than a handful per entity, so the cap was free.
- Never let a persistence error escape a timer callback. Catch, attempt a recovery, and if that fails skip this save and continue. A bad tick should cost one tick.
- Wrap the whole periodic callback. Any throw inside it has the same consequence, not just the one you found.
The transferable lessons
Any per-pair or per-combination structure in a long lived process needs a bound from the day it is written. The population that makes it a problem is always further away than it feels.
And a restart policy is not resilience. Combined with an error that recurs deterministically, it is an amplifier: it converts one failure into thousands and hides the evidence in restart noise. Alert on restart counts, not just on whether something is up.
Need help with any of this?
These notes are free and always will be. If you would rather someone just set it up, or you are stuck on something similar, get in touch at hello@opsira.io.