Your bot filter is reading the one field bots control
A visit from a VPS provider, claiming to be Chrome on Windows, passed every filter and produced a notification at ten past ten at night.
Most homegrown analytics filters bots like this:
/bot|crawl|spider|headless|preview|curl|python|wget/i.test(userAgent)That catches the polite ones. Search crawlers, monitors and uptime checkers identify themselves honestly, so a string match removes them.
It catches nothing else, because the user agent is the one field the client fully controls. Anything with a reason to look like a person says it is Chrome on Windows, and your filter agrees.
What it looks like
A visit that seems entirely legitimate: desktop browser, plausible city, first visit, landing on the home page. The only thing out of place is the network it came from, and most analytics never shows you that.
Ours arrived from a company that sells virtual servers and proxies. Nobody sits on that connection. It was a scraper or a proxy hop, and it produced a real-time notification as though it were a prospect.
The field that actually distinguishes them
Network, not user agent. An IP intelligence lookup will usually return the owning organisation and whether the range is hosting, residential or mobile. Store both:
- Organisation name. Hosting providers are recognisable, and you can keep a short list of the ones you actually see.
- Connection type. Where a range is classified as datacentre, treat it as automated regardless of what the browser claims.
People do not browse from a cloud region. A residential browser fingerprint on a hosting IP is a contradiction, and the contradiction is the signal.
Expect the lookup to have gaps
This is where ours failed. Traffic from the large well-known clouds was correctly tagged as datacentre. Traffic from a smaller hosting company came back with the connection type blank, so nothing flagged it.
Do not rely on the classification alone. Keep your own list of organisation names you have seen and know to be hosting, and extend it whenever one gets through. It is a few lines, and it improves every time you are woken unnecessarily.
Filter at alerting, not at storage
Keep the record. Suppress the notification.
Discarding suspected bot traffic at write time means you cannot audit the decision later, and cannot reclassify when you find a source you were wrong about. Store everything with the network fields attached, then decide at read time and at alert time. A filter that turns out to be wrong then costs you a query rather than your history.
Need help with any of this?
These notes are free and always will be. If you would rather someone just set it up, or you are stuck on something similar, get in touch at hello@opsira.io.