Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 201 |
De Morgen
demorgen.be
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 202 |
El Diario Vasco
diariovasco.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 203 |
eKathimerini
ekathimerini.com
|
F | 9 | 10/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 204 |
Ekstra Bladet
ekstrabladet.dk
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 205 |
El Universal
eluniversal.com.mx
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
LatAm |
| 206 |
Expressen
expressen.se
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
EU |
| 207 |
Hufvudstadsbladet
hbl.fi
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
EU |
| 208 |
The Herald (Glasgow)
heraldscotland.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 209 |
Herald Sun
heraldsun.com.au
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
Oceania |
| 210 |
Het Laatste Nieuws
hln.be
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 211 |
The Hollywood Reporter
hollywoodreporter.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 212 |
IGN
ign.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 213 |
Il Gazzettino
ilgazzettino.it
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 214 |
Il Mattino
ilmattino.it
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 215 |
Il Messaggero
ilmessaggero.it
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 216 |
Irish Independent
independent.ie
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
Ireland |
| 217 |
IndieWire
indiewire.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 218 |
Dagbladet Information
information.dk
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 219 |
The Joplin Globe
joplinglobe.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 220 |
Lincoln Journal Star
journalstar.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 221 |
Kathimerini
kathimerini.gr
|
F | 9 | 10/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 222 |
Kauppalehti
kauppalehti.fi
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
EU |
| 223 |
Kompas
kompas.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 224 |
La Cuarta
lacuarta.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
LatAm |
| 225 |
La Tercera
latercera.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
LatAm |
| 226 |
L'Express
lexpress.fr
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 227 |
Luzerner Zeitung
luzernerzeitung.ch
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 228 |
Wisconsin State Journal
madison.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 229 |
Mashable
mashable.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 230 |
MassLive / The Republican
masslive.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
US |
| 231 |
Nature
nature.com
|
F | 9 | 10/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Global |
| 232 |
The News-Gazette (Champaign)
news-gazette.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 233 |
news.com.au
news.com.au
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
Oceania |
| 234 |
NewsNation
newsnationnow.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 235 |
Het Nieuwsblad
nieuwsblad.be
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
EU |
| 236 |
The Star-Ledger (NJ.com)
nj.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
US |
| 237 |
The Times-Picayune | NOLA.com
nola.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 238 |
NT News
ntnews.com.au
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
Oceania |
| 239 |
The New York Times
nytimes.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 240 |
Omaha World-Herald
omaha.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 241 |
The Oregonian
oregonlive.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
US |
| 242 |
Oxford Mail
oxfordmail.co.uk
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 243 |
Het Parool
parool.nl
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 244 |
PennLive / The Patriot-News
pennlive.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
US |
| 245 |
RNZ (Radio New Zealand)
rnz.co.nz
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
Oceania |
| 246 |
The Roanoke Times
roanoke.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 247 |
Rolling Stone
rollingstone.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 248 |
The Keene Sentinel
sentinelsource.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 249 |
San Francisco Examiner
sfexaminer.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 250 |
Sky News Australia
skynews.com.au
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
Oceania |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!