Understanding Your Data
How traffic is classified
Before anything reaches a report, every request is sorted into one of four groups. This is the idea the whole dashboard is built on.
Most of what you read in Honeylog rests on one decision, taken once for every single request your site receives: was this a person, or was it a machine, and if it was a machine, can we trust what it says about itself?
Everything else, from the headline numbers on the dashboard home to your alerts, is a view over the four groups that come out of that decision.
#The two questions we ask
The first question is what the request says it is. A request carries a short description of the software making it, and well behaved crawlers use it to introduce themselves by name. Honeylog recognises a large catalogue of these names, which is how a request becomes "Googlebot" or "ClaudeBot" instead of an anonymous line in a log.
The second question is whether the evidence agrees. A name proves nothing on its own, because anyone can put any name on a request, and plenty of software does exactly that. So we look at where the request actually came from and compare it with what the claimed owner says about their own crawlers. Organisations that run serious crawlers publish the addresses those crawlers use, precisely so that sites can check.
#The four groups
Out of those two questions come the four groups you see everywhere in the product:
- Human visitors are requests with no sign of automation about them, arriving from the kind of connection people actually browse from.
- Verified bots name themselves as a known crawler, and the address they came from confirms it.
- Suspicious bots are clearly automated, but nobody can prove who is behind them, usually because there is no published list to check against.
- Fake bots claim an identity the evidence contradicts, or pretend to be people while arriving from somewhere no person browses from.
On top of the verified group there is a second, softer grouping by purpose. Verified crawlers do all sorts of jobs, from monitoring a site to fetching a feed, and two of those jobs matter enough to most sites to get a report of their own: SEO bots, which feed search engines, and AI bots, which work for assistants, answer engines and model training.
#Why it is worth understanding
The four groups are not labels on a chart, they change what the numbers mean. Traffic that a page based analytics tool would quietly count as visitors sits here in a group of its own, so your human numbers stay honest. A crawler that grew its appetite tenfold last week is visible as itself, rather than buried in a total. And when something is pretending to be Googlebot, you get told rather than reassured.
A last thing worth knowing: the verdict belongs to the request, not to the bot. The same name can be verified in one request and fake in the next, because one of the two came from the right place and the other did not.