Understanding Your Data

Verified bots

Crawlers that say who they are and can prove it. This is the traffic you can safely make decisions on.

A verified bot is a crawler that introduced itself by name and passed the check on that name. Googlebot that really is Google, ClaudeBot that really is Anthropic, Bingbot that really is Microsoft.

#How a bot earns the label

Companies that run crawlers at any scale publish the addresses those crawlers use, and they keep that list current, because they want site owners to be able to tell their traffic apart from everyone else pretending to be them.

So when a request arrives claiming a name we recognise, we compare where it came from against what that organisation says about itself. If the two agree, the request is verified. If they disagree, it is fake. If there is nothing trustworthy to compare against, it stays suspicious.

Two details matter here:

  • The check happens per request, not once per bot. A name is verified in the requests that came from the right place and not in the others, which is exactly what you want, because impersonation is common for the popular names.
  • A verified bot is not automatically a welcome bot. Verified means we know who it is, not that you want it there. Plenty of teams are happy to see one search crawler and much less happy about a scraper that is equally real.

#What you can see about them

For a verified bot, the reports can tell you the name of the bot, what kind of work it does, and which organisation runs it. That last one is what makes it possible to read several crawlers from the same company as one story, since a single organisation often runs a handful of them for different purposes.

The Verified Bots report shows this traffic on its own, and the same shared layout as every other report lets you group it by page, country, or response, and export it.

#Why you should care about this group

Verified traffic is the only bot traffic you can reason about with confidence. It is the right basis for questions like whether a search engine is actually reaching your new section, whether an AI crawler is reading the pages you want it to read, whether your server is spending real money serving a crawler that gives you nothing back, and whether a crawler has started behaving differently than it did last month.

Those questions have answers here. Ask them of the unverified traffic and the numbers move under your feet.