A rate limited crawl will invent problems that do not exist
An auditor that cannot tell "I did not see this page" from "I saw it and the field was empty" does not report less. It reports more, confidently, and wrongly.
A crawl of a 222 page store to audit titles, meta descriptions and headings came back with a headline finding: 121 product pages had no meta description. That is the kind of number that reorganises an afternoon.
The real number was one. The other 120 pages had perfectly good meta descriptions. What they had in common was that the crawler never saw them, because the site sat behind a CDN that started returning 429 Too Many Requests about sixty URLs into a serial crawl.
Why it is so convincing
The failure is silent in the only place that matters. The crawler caught the exception, wrote a row, and moved on. The row had a URL and a set of empty fields, which is indistinguishable from a page that genuinely has nothing in those fields.
And the fabricated finding is plausible. Missing meta descriptions on an e-commerce catalogue is exactly what you expect to find. Nothing about the output looks wrong. It looks like the answer.
Compare that with a crawler that crashes on the first 429. That is a much better failure, because it is obviously a failure. The dangerous version is the one that completes.
The shape of the bug
It is almost always the same three lines:
try:
html = fetch(url)
except Exception as e:
rows.append({"url": url, "err": str(e)})
continue
rows.append(parse(html))That is correct code. The bug arrives later, in whatever counts the results:
missing = sum(1 for r in rows if not r["meta"])Every error row has no meta key, or an empty one, so every error row is counted as a defect. The error column was recorded faithfully and then never consulted.
Three rules that prevent it
- Never let "not fetched" and "fetched, field empty" share a representation. They are different states and they need different values. If a field can be null, make null mean "not measured" and use an empty string for "measured, absent". Then a count of defects that reads null as a defect is a type error you can catch, rather than a number you have to notice.
- Print the failure count beside every statistic. "121 pages missing a meta description" next to "121 pages failed to fetch" is obviously suspect to anyone glancing at it. The same finding with the failure count in a different part of the output is not.
- Refuse to summarise a partial crawl. If coverage is under 100 per cent, either retry until it is, or label every figure as a lower bound. A tool that quietly summarises whatever it happened to collect is a tool that will mislead you on its worst day rather than its average one.
Before believing any audit number, ask what the tool does when it cannot see a page. If the answer is "the same thing it does when the page is fine but the field is empty", the number is unusable, and no amount of staring at it will reveal that.
Not getting rate limited in the first place
A serial crawl from a single datacentre IP with no delay will trip most CDN rate limits within a few dozen requests. Datacentre address space is treated more harshly than residential, so a crawl that is fine from a laptop can fail from a server.
- Put a delay between requests. One to two seconds is enough for almost any site, and a 222 page crawl is still under ten minutes.
- Back off exponentially on 429 and retry with a cap, rather than failing on the first one.
- Honour
Retry-Afterwhen the response carries it. - Send a User-Agent that identifies who you are and how to contact you. Blocks get lifted for crawlers that are polite and identifiable.
- Crawl your own site from an address you control, so if you do get throttled you can tell the difference between a limit and a fault.
The general version
This is the standard failure mode of any tool that measures the absence of something. Absence of evidence gets recorded as evidence of absence, and the report reads identically either way. Broken link checkers report working links as broken when the network hiccups. Config auditors report a missing setting when the file could not be opened. Uptime checks report an outage when the checker itself lost DNS.
The discipline is the same everywhere: a measurement and a failed measurement must never produce the same value. If they do, the tool is not measuring the thing. It is measuring its own reliability, and reporting the result as if it were the thing.
Need help with any of this?
These notes are free and always will be. If you would rather someone just set it up, or you are stuck on something similar, get in touch at hello@opsira.io.