SEO bots | HoneyLog Documentation

SEO bots are the search engine crawlers: Googlebot, Bingbot, YandexBot, DuckDuckBot, Yahoo! Slurp and well over a hundred more that Honeylog recognises by name. They read your pages so that your pages can be found.

#A subcategory of verified bots

SEO is not a fifth verdict alongside human, verified, suspicious and fake. It sits one level below them.

Honeylog first decides whether a request is a verified bot at all, by checking the name it gave against where it actually came from. Only then does it ask what that bot is for, and SEO is one answer among many. The same verified group holds site and network monitors, feed fetchers, link validators, security checkers, social media agents collecting a preview, read-it-later services and more.

What makes SEO and AI different is not that they are the only ones, it is that they are the two most sites need to act on, so the dashboard gives each of them a report of its own. Everything else verified sits together in the general Verified Bots report.

Two consequences follow, and both are worth holding on to:

Nothing reaches this report without passing the identity check. Something calling itself Googlebot from a rented server never appears here. It is counted as fake, which is the whole reason the numbers on this page are usable.
The label describes the job, not your opinion of it. A crawler is filed under SEO because that is the work it does, not because you have decided to welcome it.

In the dashboard this family carries a single category, shown as Search Bot.

#What a search crawler is actually doing

The work has three parts that repeat forever: discovering URLs, fetching them, and coming back later to see whether anything changed.

The schedule is not fixed. How often a crawler returns is decided by its own algorithms, weighing how popular a site is, how fresh its content tends to be, how often it changes and how much authority it carries. Well behaved search crawlers respect robots.txt and throttle themselves so they do not overwhelm a server.

None of that scheduling is yours to control. What is yours is what they find when they arrive: which URLs exist, what you return for them, and how quickly.

#Why they matter

Crawling is the precondition for everything else in search. Nothing you do about titles, content or links has any effect on a page that never gets fetched. That makes this report one of the few places where an SEO problem is visible as a fact rather than an inference.

A page that is never crawled cannot rank. The most common version of this is a new section that nothing links to, or one quietly excluded by a rule nobody remembered writing. The report tells you whether it is being fetched at all.
Crawler attention is finite. It is ordinary to find a crawler spending most of its requests on filter combinations, sorting parameters, session URLs or endless paginated archives, while the pages you actually care about are visited rarely. Grouping the report by path shows you that imbalance directly.
Migrations show up here first. Crawl volume reacts to a redirect chain, a mistaken robots.txt or a slow server days before rankings do. A collapse in crawling after a release is an early warning you can still act on.
What you serve a crawler is part of the story. The same report carries response codes and response times per bot, so a section quietly answering 404 or 500 to Googlebot, or taking seconds to respond, stops being invisible.

#They also send people back

Search crawlers belong to the group that leads to human visits: someone reads a result and arrives. Bot Conversions follows that chain, from the crawl to the person who turns up afterwards, which is the honest way to judge whether the crawling is worth what it costs you to serve.

#Where to go next

The SEO Bots report under Bots Traffic holds this traffic on its own, with the usual chart, table and filters. Visibility Radar follows one crawler over time, which is the natural next step once you notice something changed. And Robots.txt shows whether the rules you wrote are being respected.