Your build output is not your live site
Someone patched production directly, months ago, and it worked. The next deploy quietly undoes it.
A site had been deployed by copying a build directory onto a server. Straightforward, and it had worked for months.
Before deploying a routine content change, a diff of the build output against the live directory turned up analytics and attribution scripts that existed only in production. The built pages contained no reference to either. Deploying would have removed visitor tracking from every page on the site, silently, with no error and no visible difference on the page.
What had drifted
- Two JavaScript files present on the server and absent from the source.
- Script tags for those files, hand-added to the deployed HTML rather than to the layout template.
- An analytics beacon updated in production and never back-ported, so the source still held an older version pointing at a retired endpoint.
- Two text files added directly to the server by a later piece of work.
- Worst of the set: a configuration file that existed in both places with different contents. The source copy was older and would have overwritten a deliberate change.
Why nobody notices
Every one of those edits worked when it was made. Production was the fastest place to make them, and the change was verified on the live site immediately afterwards. The gap only opens on the next deploy, which might be weeks later and performed by someone with no reason to connect the two events.
Removal is also the quietest kind of failure. A missing analytics script does not throw. The page renders identically. You find out when someone asks why the traffic numbers stopped, and by then the deploy is not the obvious suspect.
The check that takes a minute
Before any file-copy deploy, list what exists in production and not in the build:
comm -13 <(ls dist | sort) <(ssh server 'ls /var/www/site' | sort)Then diff the rendered output of one page against the live version of the same page, looking specifically at script tags. File-level comparison will not catch a tag that was hand-added inside an HTML file that exists in both places.
Every result needs an answer: either it belongs in the source, or it is genuinely server-side and the deploy must not remove it.
Copy without deleting, verify, and only then consider pruning. Back up the live directory first. An extra stale file costs nothing; a removed one can be invisible for weeks.
The fix is not discipline
"Stop editing production" is good advice and will not hold, because at some point something will be broken at an inconvenient hour and the server will be the fastest route. Plan for it instead: put the diff in the deploy routine so drift is surfaced automatically rather than remembered.
When you do find drift, back-port it to the source rather than re-patching production afterwards. Otherwise you have agreed to perform the same repair after every deploy, forever.
The upside
The same comparison, run in the other direction, is worth doing too. In this case the build contained a finished page that had never been deployed at all. Nobody had noticed it was missing.
Need help with any of this?
These notes are free and always will be. If you would rather someone just set it up, or you are stuck on something similar, get in touch at hello@opsira.io.