Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 301 |
National Post
nationalpost.com
|
F | 18 | 9/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 302 |
NOS
nos.nl
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 303 |
New Straits Times
nst.com.my
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 304 |
Portafolio
portafolio.co
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Meta-ExternalAgent
|
LatAm |
| 305 |
The Post and Courier
postandcourier.com
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 306 |
Pulse (Maeil Business)
pulsenews.co.kr
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 307 |
Reuters
reuters.com
|
F | 18 | 9/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Global |
| 308 |
Salzburger Nachrichten
salzburg.com
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
EU |
| 309 |
Slate France
slate.fr
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Meta-ExternalAgent
|
EU |
| 310 |
SpaceNews
spacenews.com
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Meta-ExternalAgent
|
US |
| 311 |
IEEE Spectrum
spectrum.ieee.org
|
F | 18 | 9/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 312 |
Swarajya
swarajyamag.com
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
Asia |
| 313 |
Tages-Anzeiger
tagesanzeiger.ch
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 314 |
Techdirt
techdirt.com
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 315 |
The Africa Report
theafricareport.com
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 316 |
The Hindu
thehindu.com
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
Asia |
| 317 |
Saskatoon StarPhoenix
thestarphoenix.com
|
F | 18 | 9/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 318 |
Westdeutsche Allgemeine Zeitung
waz.de
|
F | 18 | 9/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 319 |
Windsor Star
windsorstar.com
|
F | 18 | 9/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 320 |
20 Minutes
20minutes.fr
|
D | 27 | 8/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Applebot-Extended
|
EU |
| 321 |
ABC News (Australia)
abc.net.au
|
D | 27 | 8/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Meta-ExternalAgent
|
Oceania |
| 322 |
Aftenposten
aftenposten.no
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 323 |
Aftonbladet
aftonbladet.se
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 324 |
Al-Ahram
ahram.org.eg
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 325 |
Allure
allure.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 326 |
American Banker
americanbanker.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 327 |
Android Police
androidpolice.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 328 |
AP News (org)
ap.org
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Global |
| 329 |
Architectural Digest
architecturaldigest.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 330 |
Ars Technica
arstechnica.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 331 |
Bon Appetit
bonappetit.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 332 |
The Boston Globe
bostonglobe.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 333 |
Brasil 247
brasil247.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
LatAm |
| 334 |
Bergens Tidende
bt.no
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 335 |
Novy Cas
cas.sk
|
D | 27 | 8/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
EU |
| 336 |
The Chronicle of Higher Education
chronicle.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 337 |
Conde Nast Traveler
cntraveler.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 338 |
Collider
collider.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 339 |
Complex
complex.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 340 |
Daily Trust
dailytrust.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 341 |
The Dallas Morning News
dallasnews.com
|
D | 27 | 8/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
US |
| 342 |
Dossier Center
dossier.center
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 343 |
E24
e24.no
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 344 |
Eater
eater.com
|
D | 27 | 8/11 |
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 345 |
El Tiempo
eltiempo.com
|
D | 27 | 8/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Meta-ExternalAgent
|
LatAm |
| 346 |
El Mercurio Online (Emol)
emol.com
|
D | 27 | 8/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
Bytespider
Amazonbot
Applebot-Extended
|
LatAm |
| 347 |
E! Online
eonline.com
|
D | 27 | 8/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
US |
| 348 |
Entertainment Weekly
ew.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 349 |
FlightGlobal
flightglobal.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 350 |
Food & Wine
foodandwine.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!