Your sitemap is advertising pages that do not exist
Two hand-maintained lists of the same thing will drift. When one of them is your sitemap, the drift is search engines crawling 404s on your behalf.
A site listed twenty-odd articles in its sitemap. Four of them had never been written. The metadata existed, the slugs existed, the index page linked to them, and the pages themselves returned 404.
Nobody noticed because everything looked right from the inside. The article list rendered. The sitemap generated. It was only a crawl of every sitemapped URL that turned it up.
Why it happens
Adding an article took three steps: write metadata into a registry, create the page file, and add the slug to a list in the sitemap generator. Steps one and three were done for all four. Step two was not.
Nothing enforced the relationship, because the sitemap had its own copy of the truth. Two lists describing the same set of things, maintained by hand, will eventually disagree. That is not a discipline problem, it is a design problem.
Why it matters more than it looks
A sitemap is a direct statement to a search engine: these URLs exist and are worth crawling. Filling it with 404s spends crawl budget on nothing and undermines the credibility of every other URL in the file. Some engines are markedly less forgiving about this than others, and the one that feeds AI assistants is often the stricter one.
It is also the kind of fault that silently gets worse. Every crawl re-tries the dead URLs.
The fix
Derive the sitemap from the same list everything else uses, and filter it on a flag that means published:
// one source of truth
export const PUBLISHED = ALL_POSTS.filter((p) => !p.draft);
// the sitemap can no longer disagree with the site
const slugs = PUBLISHED.map((p) => p.slug);Marking the unwritten entries as drafts rather than deleting them keeps the titles and descriptions for whoever writes the articles later. Removing the flag republishes everywhere at once.
Crawl your own sitemap and assert every URL returns 200. It takes a few lines, runs in seconds, and catches this class of fault permanently. Any sitemap generated from a database or an API can develop it after a schema change or a failed migration, not just hand-written ones.
The general shape
Whenever you find yourself adding the same identifier to two places, one of them should be computed from the other. The question to ask during review is not whether both lists are correct now, it is what makes them stay correct.
Need help with any of this?
These notes are free and always will be. If you would rather someone just set it up, or you are stuck on something similar, get in touch at hello@opsira.io.