One analytics source will mislead you, and filtering changes the answer
Three readings of one dataset produced three different business conclusions. Only one was right, and it was not the alarming one.
We looked at a week of traffic and concluded it had fallen off a cliff. It had not. Here is how one dataset produced three answers.
Reading one: raw rows. Down 67%.
Counting every row in the visits table put the busiest day of the previous week at 36 and the latest day at 1. Alarming, and wrong.
The table held bot traffic, datacentre traffic, and one day when an internal QA agent had walked the entire site, contributing 142 rows on its own. Anything compared against that baseline looks like a collapse.
Reading two: filtered. Down 24%.
Applying the bot and datacentre filters turned that 36 into 13 real visits from 9 people, and the QA day''s 142 into 20. Week on week became a 24% softening.
Better, and still wrong, because it was measuring the same instrument more carefully rather than asking whether the instrument was right.
Reading three: a second source. Flat.
An independent analytics tool on the same pages over the same days showed 62 sessions, then 57. An 8% wobble, and nothing to investigate.
It also showed every single day higher than the first source, by roughly a third. Privacy blockers, browsers that drop beacon requests, and sessions ending before the request fires all remove visits from a homegrown beacon. The two sources agreed on shape and disagreed on level, which is the useful outcome: trust the trend, distrust the absolute.
What to take from it
Every count is a filter, and the filter is a decision. Nobody chose to include QA traffic in the baseline. It happened because the default is to count rows.
Exclude your own agents explicitly. Internal tooling, deploy checks and anything that renders pages will otherwise appear as your most engaged users.
Two imperfect sources beat one careful one. Neither is accurate in absolute terms. Where they agree you can act. Where they disagree, you have learned something about the instruments.
Be most suspicious when data confirms a worry. A frightening number invites action rather than scrutiny. The 67% collapse had people looking in the wrong direction before anyone asked whether the baseline was real.
The knock-on effect on every rate
Undercounting does not only mislead about traffic. Every rate derived from it is wrong in the opposite direction. Conversion looks better than it is, because the numerator comes from your payment processor, which misses nothing, while the denominator comes from a browser script, which misses plenty.
If a conversion rate looks implausibly good, check the denominator before celebrating.
Need help with any of this?
These notes are free and always will be. If you would rather someone just set it up, or you are stuck on something similar, get in touch at hello@opsira.io.