SEO log analysis

See what Googlebot really crawls, and what it misses

Every request of Googlebot, Bingbot and the other search engines, verified and URL by URL: where your crawl budget goes, the errors crawlers get, and the pages they never reach, straight from your server logs.

Get a demo

Free audit, no account needed. HoneyLog adds no JavaScript tag.

Visibility Radar

Know every search crawler, and catch the fake ones

Search Console shows a summary, days later. Your server sees every request. HoneyLog turns them into a list of the crawlers that visit you, what each one reads, and which ones are impostors.

  • Every crawler by name. Googlebot, Bingbot, Applebot and the SEO tools, with unique URLs, total crawls, status codes and last visit.

  • Fakes caught. A Googlebot from outside the IP ranges Google publishes is flagged as malicious and kept out of your crawl numbers.

  • Day by day. How the crawl of each bot moves over time, so a drop after a release or a robots.txt change shows up at once.

Example data

Status codes

See where your crawl budget really goes

Every request Googlebot spends on a redirect, an error, a parameter or a filter page is one it does not spend on the pages you want indexed. HoneyLog shows the crawl section by section, with the status codes it gets back.

  • Crawl by section. How many verified crawls each part of your site receives, from product and article pages to tags and parameters.

  • Waste made visible. The redirects, 404s and 5xx search engines keep hitting, URL by URL, as they happen.

  • An alert on errors. An email when server errors spike, before crawlers start slowing down on your site.

Example data

Sitemap

Find the pages search engines never reach

Your sitemaps list the pages you want found. Your logs show the pages bots really visit. HoneyLog puts the two side by side, so the gaps stand out.

  • Listed but never visited. Sitemap URLs nobody has visited yet, not even a crawler, so you know where to add internal links.

  • Crawled but not listed. URLs crawlers keep visiting that are in none of your sitemaps: old pages, parameters, duplicates.

  • Last status per URL. What each page answered the last time a bot asked, so a broken URL in a sitemap never goes unnoticed.

Example data

Questions from SEO teams

Search Console shows Google's own view, aggregated and a few days behind: how your pages appear in search and a summary of its crawl. HoneyLog shows every request Googlebot and the other search engines actually make to your server, URL by URL, as it happens, verified against the IP ranges the search engines publish. Use both: Search Console for rankings and indexing, HoneyLog for the crawl itself.

On a site with a few hundred pages, rarely. On sites with thousands of URLs, faceted navigation, parameters or frequent updates, it does: every request Googlebot spends on a redirect, an error, a filter combination or a duplicate is a request it doesn't spend on the pages you want indexed. HoneyLog shows where the crawl goes, section by section, and which status codes it gets back.

Anyone can write Googlebot in a user agent. HoneyLog checks every request that claims to be a known crawler against the IP ranges its company publishes: a Googlebot from outside Google's ranges is labeled a malicious bot and kept out of your Googlebot numbers, so impostors never inflate your crawl data.

No. SEO crawlers such as Screaming Frog simulate a crawl of your site at a given moment. HoneyLog doesn't crawl anything: it shows the requests that Googlebot, Bingbot, the AI crawlers and the SEO tools themselves really make to your server, continuously. The two answer different questions: what could be crawled, and what is crawled.

All the major ones, by name: Googlebot and its variants, Bingbot, Applebot, DuckDuckBot, YandexBot, Baiduspider and others, plus SEO tools such as AhrefsBot and SemrushBot and the AI crawlers. The search engines that publish their IP ranges are verified against them, so you see what the real bot did.

More answers in the HoneyLog FAQ

Free AI audit

Your logs already know what Googlebot crawls

Check how AI crawlers see your site in a minute, or book a demo and see HoneyLog with our team.

Get a demo

Free audit, no account needed. HoneyLog adds no JavaScript tag.