Soft 404s: when your error page returns success
A missing page that returns HTTP 200 is worse than one that returns 404, and the configuration that causes it looks entirely reasonable.
The mistake
A common static site configuration falls back to the error page when a file is not found:
location / {
try_files $uri $uri/ $uri.html /404.html;
}That serves your branded 404 page, which looks correct in a browser. But using a file path as the fallback serves it with an HTTP 200. As far as any crawler is concerned, every wrong URL on your site is a real page with real content.
Why it matters
Crawlers use status codes to decide what exists. Return 200 for everything and you get duplicate near identical pages indexed, crawl budget spent on URLs that were never real, and a site that looks larger and thinner than it is. On a site whose purpose is to be cited as a source, that is working directly against you.
It also hides mistakes. A broken internal link returns a page rather than an error, so nothing in your monitoring notices.
The fix
location / {
try_files $uri $uri/ $uri.html =404;
}
location = /404.html { internal; }
error_page 404 /404.html;The =404 raises a real not found condition. error_page then renders your custom page while preserving the status. The internal directive stops the error page being requestable at its own URL, which would itself be a 200.
Verify it
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/does-not-existIt must print 404. Checking in a browser proves nothing, because a browser renders the page identically either way. This is the whole reason the mistake survives so long.
The general shape
Anywhere a system has a friendly fallback, ask what status code it returns. Single page applications that serve index.html for unknown routes have exactly the same problem, and it is even easier to miss there because the app renders a plausible empty state.
Need help with any of this?
These notes are free and always will be. If you would rather someone just set it up, or you are stuck on something similar, get in touch at hello@opsira.io.