Using the Dashboard

Sitemap

Track how bots crawl the URLs of your sitemaps, page by page, and spot what is not being crawled.

The Sitemap section connects your sitemaps to your logs. You give Honeylog the sitemap URLs of your site, it collects every URL listed in them (following nested sitemaps, news sitemaps and feeds), and then crosses that list with your real traffic. The result: you can see which of your pages bots actually crawl, which ones they ignore, and which URLs receive traffic without being in any sitemap.

The section works on single sites, so it is not shown when you are viewing a network. It has three pages: Overview, Page Stats and Tags Lists.

#Overview

The top of the page holds your Sitemaps list. Use Add Sitemaps to register one or more sitemap URLs; they must live on your own domain. As soon as you save, Honeylog scans them, and from then on they are re-scanned about every hour, so new URLs show up on their own.

Below it, the Sitemap URLs table lists every collected URL together with its traffic:

  1. The URL, linking to its Page Stats.
  2. Which sitemap it came from.
  3. Total Visits in the selected date range, with the comparison value when the date picker comparison is on.
  4. First Visit and Last Visit, which cover your whole data retention and do not depend on the selected range.
  5. The Last Status code returned for it.

Two sets of filters make this table useful:

  • The Show dropdown switches between all URLs, URLs in your sitemaps, URLs not in any sitemap, and URLs without crawls. The last one is the interesting view: sitemap entries that received no visits in the range, meaning bots are skipping them.
  • The page type pills narrow the list to articles, categories, the homepage, author pages, company pages, tags, robot files or assets. Categories, authors, company pages and tags come from what you classified on the Tags Lists page.

A search box filters by URL, and the table is paginated at 10 rows.

#Page Stats

Page Stats is the single-page view. Type a pathname in the search bar (it autocompletes from your sitemap URLs) or arrive here by clicking a URL in the Overview. You can also scan a pathname that is not in any sitemap; in that case a badge tells you, and the dates come from your logs instead of the sitemap.

For the chosen page you get:

  1. A Verified Crawler Producers chart, a daily trend of visits with one line per bot producer (Google, OpenAI, Anthropic and so on) plus a line for everything else.
  2. The Latest Visits panel with the most recent bot hits on the page and a link into Raw Logs scoped to it.
  3. Two tiles with the page's creation date and last update as declared in your sitemap.
  4. A Verified Bots table, one row per bot that visited the page: total visits, last visit, first visit expressed relative to the page's creation ("3 days after the creation"), and how quickly the bot came back after your last update. It can be filtered by bot category, searched by name and sorted.

This is the page to answer questions like "did GPTBot pick up my article, and how long after I published it?".

#Tags Lists

Tags Lists is the settings page of the section. Here you teach Honeylog the structure of your site by classifying URLs into four groups: Categories, Authors, Company Pages (about us, privacy policy, contacts) and Tags.

Each group has an add button that opens a bulk modal: paste or type URLs, which autocomplete from the URLs already collected from your sitemaps, and save them all at once. The classifications feed the page type filters on the Overview, so the effort pays off immediately.

To see the crawl side aggregated by bot instead of by page, see Bot traffic. To follow a single crawler over time, see Visibility Radar.