Robots.txt Intelligence already showed you your rules and every version of the file. Now it answers the next question: who is ignoring them? A violation is a request for a page that one of your Disallow rules forbids to that bot, judged against the version of the file that was live at that moment. The Overview gives you the violating bots, the disallowed hits, the paths affected and the top offender. Below that, a chart marks every change to your robots.txt, so you can see whether a new rule actually made a bot stop. Click a bot to open its own page, with the paths it hit split by what your site answered: served, redirected or blocked. Violated URLs works the other way around: pick a path and see which bots went there anyway. If a bot you blocked keeps showing up, robots.txt isn't enough for it: block it at the CDN or server level and track the decision in Bots Management.
Changelog
Robots.txt Violations: see which bots ignore your rules
Your robots.txt is a request, not a lock. Now you can see exactly who isn't respecting it, on which pages, and what your site answered
Sensini Luca
14-day Free Trial
Try HoneyLog 14-day Free Trial
If you are interested in HoneyLog, you can try it with our Free Trial!
No credit card required