Sitemap: see what bots actually do with the pages you publish

Your sitemap tells crawlers which pages matter. Until now, though, you had no way to know whether they listened. The new Sitemap section answers exactly that question. Everything starts from the Overview page. Add one or more sitemap URLs from your domain and we scan them automatically, collecting every URL they contain. Each collected URL then appears in a single table cross referenced with real traffic: total visits with a comparison against the previous period, first and last visit, and the last HTTP status code the page returned. You can sort by visits or by most recent visit, search for any URL, and filter by page type. The most interesting part is the coverage filters. In one click you can see the URLs that are in your sitemaps, the visited URLs that are missing from your sitemaps, and the sitemap URLs that received no crawls at all in the selected period. That last view is gold: those are the pages you told crawlers about that nobody is coming to read. The Page Stats page lets you zoom in on a single page. Type a pathname and an autocomplete suggests matching entries from your sitemaps, exactly like a search bar. Once selected, you get a visit trend chart, the latest bot visits on that page, and its lifecycle: when it first appeared in your sitemap and when it was last updated, or, if the page is not in any sitemap, when bots first and last reached it. Pages that receive traffic but are missing from your sitemaps are clearly marked with a Not in sitemap badge. Finally, Tags Lists lets you teach us the structure of your site. Add the URLs of your categories, authors, company pages and tags, with the same autocomplete based on your sitemap entries, and those page types become available as filters throughout the section. Add your sitemaps from the Overview page and let the first scan run. From that moment on you will always know which of your pages crawlers visit, which ones they ignore, and which ones they find even though you never listed them.