Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 51 |
Tempo (English)
en.tempo.co
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 52 |
Yonhap (English)
en.yna.co.kr
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 53 |
Expansión
expansion.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 54 |
Expresso
expresso.pt
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 55 |
FAZ
faz.net
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 56 |
Het Financieele Dagblad
fd.nl
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 57 |
France Info
francetvinfo.fr
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 58 |
Detroit Free Press
freep.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 59 |
Frontline
frontline.thehindu.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 60 |
The Gainesville Sun
gainesville.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 61 |
Grand Forks Herald
grandforksherald.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 62 |
Green Bay Press-Gazette
greenbaypressgazette.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 63 |
The Greenville News
greenvilleonline.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 64 |
Haaretz
haaretz.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 65 |
Sarasota Herald-Tribune
heraldtribune.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 66 |
Helsingin Sanomat
hs.fi
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 67 |
HuffPost
huffpost.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 68 |
L'Humanité
humanite.fr
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 69 |
The Hutchinson News
hutchnews.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 70 |
Iltalehti
iltalehti.fi
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 71 |
The Indianapolis Star
indystar.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 72 |
i
inews.co.uk
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 73 |
The Forum (Fargo)
inforum.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 74 |
Ilta-Sanomat
is.fi
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 75 |
Jacksonville.com
jacksonville.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 76 |
Journal & Courier (Lafayette)
jconline.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 77 |
Milwaukee Journal Sentinel
jsonline.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 78 |
Knoxville News Sentinel
knoxnews.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 79 |
Kyodo News
kyodonews.net
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 80 |
La Libre Belgique
lalibre.be
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 81 |
LancasterOnline
lancasteronline.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 82 |
La Presse
lapresse.ca
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 83 |
La Tribune
latribune.ca
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 84 |
La Tribune
latribune.fr
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 85 |
La Voix du Nord
lavoixdunord.fr
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 86 |
Le Devoir
ledevoir.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 87 |
Le Droit
ledroit.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 88 |
Le Nouvelliste
lenouvelliste.ca
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 89 |
Les Échos
lesechos.fr
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 90 |
Le Soir
lesoir.be
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 91 |
Le Soleil
lesoleil.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 92 |
L'Est Éclair
lest-eclair.fr
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 93 |
Libération
liberation.fr
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 94 |
The Journal News (lohud)
lohud.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 95 |
Lubbock Avalanche-Journal
lubbockonline.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 96 |
The Mainichi
mainichi.jp
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 97 |
Marca
marca.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 98 |
The Mercury News
mercurynews.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 99 |
Metro
metro.co.uk
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 100 |
MMA Junkie
mmajunkie.usatoday.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!