Robots.txt | HoneyLog Documentation

Your robots.txt is a small file with a big influence on who can crawl your site. The Robots.txt section of the dashboard fetches it for you every day, tells you what each rule means in plain English, keeps every version you ever published and gives you practical advice for your kind of website. It also includes a leaderboard that shows how the biggest websites of the web treat AI crawlers.

The section only makes sense for a single site, because a network has no robots.txt of its own. When you are viewing a network the entry is hidden from the sidebar.

When you open the section you find six pages in its own sidebar (or as a row of buttons on mobile): Trends, Violations, Details, History, Tips and Leaderboard. Here is what each one does.

#Trends

This is the main page. Honeylog fetches your robots.txt once a day, in the morning of your site's timezone, and shows you the current state of things:

The robots.txt card with its status (Found, No robots.txt or Unreachable), the full content of the file and when it was last checked. If you are an admin of the site you also get a Check now button to fetch the live file right away, useful right after a deploy.
A trend chart with the daily hits your /robots.txt received from bots over the last 7 days, together with the bots that requested it most. A robots.txt nobody reads is a sign that crawlers are not paying attention to it, so this chart is worth a look now and then.
A Crawlers in robots.txt list, with one card for every user agent named in your file plus the * wildcard rule. Each card tells you whether that crawler is allowed or disallowed, what the bot is for, and lists its Allow, Disallow and Crawl-delay rules. When your file changes, small badges highlight what is new, what was removed and what changed compared to the previous version. A search box helps you jump to a specific bot.

#Violations

A robots.txt is a request, not a lock: nothing stops a bot from ignoring it. This page shows you which bots do. A violation is a request for a page that a Disallow rule of your robots.txt forbids to that bot. Your file changes over time, so every hit is judged against the version that was live at that moment, not against the current one.

The three Violations pages follow the relative date picker in the header ("Last 7 days", "Last 30 days", or any start date you pick), a range that always ends today.

Overview is the summary. Four numbers on top: the violating bots, the total disallowed hits, the distinct affected paths and the top offender, the bot with the most violations. Below, a chart of the disallowed hits over the range: a dashed marker shows every moment your robots.txt changed, so you can tell whether a new rule made a bot stop, or whether it simply keeps crawling. The versions in force during the range are listed above the chart. Then one card per violating bot, with its violations and a small trend; click a card to open that bot's page. The list can be exported as CSV.

Violating Bot looks at one bot. Pick it from the selector (only the bots that violated your robots.txt in the range are listed) or from the top offenders proposed on the empty page. You get its disallowed hits, the paths it hit, its most violated path and its last violation, the same chart filtered on the bot, and three lists of the pages it hit despite your rules, split by the answer your site gave: 2xx (the page was served), 3xx (redirected) and 4xx-5xx (blocked or failed). Each list has its own CSV export, and the button at the top right exports all of them together with the status code.

Violated Pathname is the opposite view. Type a pathname (or pick one of the most violated ones) to see the chart of the hits it received against your rules and the bots that made them, with one CSV export of those bots.

If a bot you blocked keeps showing up here, the robots.txt is not enough for it: consider blocking it at the CDN or server level, and track the decision in Bots Management.

#Details

A line-by-line explanation of your current robots.txt written for humans. Every user agent group gets its own card, and each Allow and Disallow rule is paired with a sentence that says what it actually does. This includes the cases people often get wrong, like rules that look restrictive but have no effect at all. Sitemap lines and non-standard directives are listed separately with a note on how crawlers treat them.

If you inherited a robots.txt from someone else, or you are not sure a rule does what you think, this is the page to read.

#History

Every distinct version of your robots.txt that Honeylog has captured, newest first. A new version is saved only when the content really changes, so each entry is a genuine revision and not just a daily copy. You can open the full text of any version and copy it, for example to roll back a change. The current version is marked, and the moments when the file was missing or unreachable are flagged too.

Whenever a change is detected, Honeylog also creates an automatic annotation ("robots.txt changed", "robots.txt added" or "robots.txt removed"), so you can spot the change directly on your traffic charts and see whether it had an effect. You can review these automatic annotations on the Annotations page under Settings.

#Tips

Practical guidance for writing a good robots.txt, tailored to the type of website you run. The page opens on the website type saved in your site options (publisher, ecommerce, SaaS, marketplace and so on), and a dropdown lets you preview the advice for the other types.

For the selected type you get a crawling strategy, what to keep crawlable, what to keep crawlers out of to save crawl budget, how to think about AI crawlers, an example robots.txt you can copy as a starting point, the most common mistakes, and a reference of the crawlers worth knowing about.

#Leaderboard

This page is not about your site. It ranks major websites of the web by how open their robots.txt is to AI crawlers, based on a scan that Honeylog runs every month. You can see the share of sites blocking each AI crawler (GPTBot, ClaudeBot, PerplexityBot and others), a grade distribution from A (wide open) to F (walled off), and how blocking has changed over time. The table can be searched, filtered by region and by blocked bot, and sorted.

Keep in mind that the leaderboard measures declared intent. A robots.txt rule says what a site asks crawlers to do, not what its CDN or firewall actually enforces.

#Where did Annotations and Bots Management go?

They used to be part of this section, but they are not really about robots.txt, so they now live under the Settings label of the main sidebar. You can read about both in Site settings.