Robots.txt Intelligence: track, monitor and understand your robots.txt

Your robots.txt is the first thing AI crawlers, SEO spiders and monitoring bots read when they visit your site, yet almost nobody checks it regularly. The new Robots.txt Intelligence section does it for you. At its core is the Trends page, where you can see the current state of your robots.txt at a glance: whether it was found, is missing, or could not be reached at the last check, when it was last verified, and its full content in a clean and readable viewer. You can trigger a fresh check on demand with the Check now button, or simply let the automatic daily check keep everything up to date. If your site serves no robots.txt at all, we tell you, because in that case AI crawlers are free to access everything. Below the file itself, the Crawlers in robots.txt panel breaks the file down bot by bot. Every crawler explicitly named in your robots.txt gets its own card showing whether it is allowed or disallowed, what it is used for, and the exact Allow, Disallow and Crawl delay rules that apply to it. A search box lets you instantly find any bot by name or purpose. Since we keep previous versions of your file, each card is also compared with the prior version: bots that are new, removed or whose rules changed are clearly flagged, with the old rules shown next to the new ones. A silent edit to your robots.txt will never go unnoticed again. The page is completed by a view of the bot traffic from the last 7 days, connecting what the file says with what bots actually do. The section also brings everything together in one place. Alongside Trends you will find Details, History and Tips for your robots.txt, plus Annotations, Bots Management and the Robots Leaderboard, all reachable from the new Intelligence sidebar.