Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 451 |
France 24
france24.com
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
|
EU |
| 452 |
Financial Times
ft.com
|
D | 36 | 7/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 453 |
Teesside Gazette
gazettelive.co.uk
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 454 |
La Gazzetta dello Sport
gazzetta.it
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 455 |
Heraldo de Aragón
heraldo.es
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 456 |
iDNES.cz / MF Dnes
idnes.cz
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 457 |
Inc.
inc.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
US |
| 458 |
Index
index.hu
|
D | 36 | 7/11 |
GPTBot
ChatGPT-User
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
EU |
| 459 |
infoLibre
infolibre.es
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
|
EU |
| 460 |
InfoWorld
infoworld.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 461 |
JoongAng Ilbo
joongang.co.kr
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 462 |
La Jornada
jornada.com.mx
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
LatAm |
| 463 |
Jyllands-Posten
jyllands-posten.dk
|
D | 36 | 7/11 |
GPTBot
ChatGPT-User
Google-Extended
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 464 |
La Dépêche du Midi
ladepeche.fr
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
EU |
| 465 |
La Silla Vacia
lasillavacia.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
LatAm |
| 466 |
Le Figaro
lefigaro.fr
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
CCBot
Amazonbot
|
EU |
| 467 |
Leicester Mercury
leicestermercury.co.uk
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 468 |
Lidove noviny
lidovky.cz
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 469 |
Liverpool Echo
liverpoolecho.co.uk
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 470 |
Manchester Evening News
manchestereveningnews.co.uk
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 471 |
MarketWatch
marketwatch.com
|
D | 36 | 7/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
US |
| 472 |
Mitteldeutscher Rundfunk (MDR)
mdr.de
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 473 |
Mediapart
mediapart.fr
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 474 |
Midi Libre
midilibre.fr
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
EU |
| 475 |
The Mirror
mirror.co.uk
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 476 |
Mother Jones
motherjones.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
PerplexityBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 477 |
Norddeutscher Rundfunk (NDR)
ndr.de
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 478 |
Network World
networkworld.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 479 |
Yahoo News
news.yahoo.com
|
D | 36 | 7/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
|
US |
| 480 |
Newsweek
newsweek.com
|
D | 36 | 7/11 |
GPTBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
US |
| 481 |
NME
nme.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 482 |
Nottingham Post
nottinghampost.com
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 483 |
L'Obs
nouvelobs.com
|
D | 36 | 7/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
|
EU |
| 484 |
OilPrice.com
oilprice.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 485 |
PCWorld
pcworld.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 486 |
Polityka
polityka.pl
|
D | 36 | 7/11 |
GPTBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 487 |
Radio France Internationale
rfi.fr
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
|
EU |
| 488 |
Radio Télévision Suisse (RTS)
rts.ch
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 489 |
Scientific American
scientificamerican.com
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
|
US |
| 490 |
The Spokesman-Review
spokesman.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 491 |
Schweizer Radio und Fernsehen (SRF)
srf.ch
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 492 |
Südwestrundfunk (SWR)
swr.de
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 493 |
t-online
t-online.de
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 494 |
Tagesschau (ARD)
tagesschau.de
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 495 |
TechCrunch
techcrunch.com
|
D | 36 | 7/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
|
US |
| 496 |
Télérama
telerama.fr
|
D | 36 | 7/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
|
EU |
| 497 |
Tennis.com
tennis.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 498 |
The Australian
theaustralian.com.au
|
D | 36 | 7/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
Oceania |
| 499 |
The Guardian
theguardian.com
|
D | 36 | 7/11 |
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 500 |
The Sun
thesun.co.uk
|
D | 36 | 7/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
UK |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!