Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 251 |
Sowetan LIVE
sowetanlive.co.za
|
F | 9 | 10/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 252 |
De Standaard
standaard.be
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
EU |
| 253 |
St. Louis Post-Dispatch
stltoday.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 254 |
Sydsvenskan
sydsvenskan.se
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
EU |
| 255 |
The Post-Standard (Syracuse)
syracuse.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
US |
| 256 |
Talouselama
talouselama.fi
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
EU |
| 257 |
De Telegraaf
telegraaf.nl
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
EU |
| 258 |
The Telegraph
telegraph.co.uk
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 259 |
The Advocate (New Orleans)
theadvocate.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 260 |
The Globe and Mail
theglobeandmail.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
CA |
| 261 |
The Hill
thehill.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 262 |
The Mercury (Hobart)
themercury.com.au
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
Oceania |
| 263 |
The National (Scotland)
thenational.scot
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 264 |
Trouw
trouw.nl
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 265 |
Truck News
trucknews.com
|
F | 9 | 10/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 266 |
TV 2 Norway
tv2.no
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 267 |
Valor Economico
valor.globo.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
LatAm |
| 268 |
Variety
variety.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 269 |
Vibe
vibe.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 270 |
de Volkskrant
volkskrant.nl
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 271 |
Charleston Gazette-Mail
wvgazettemail.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 272 |
ZDNET
zdnet.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 273 |
24 heures
24heures.ch
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 274 |
3News (Ghana)
3news.com
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 275 |
Hamburger Abendblatt
abendblatt.de
|
F | 18 | 9/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 276 |
ANSA
ansa.it
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
EU |
| 277 |
Basler Zeitung
bazonline.ch
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 278 |
Bloomberg
bloomberg.com
|
F | 18 | 9/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 279 |
BreakingNews.ie
breakingnews.ie
|
F | 18 | 9/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Ireland |
| 280 |
BusinessMirror
businessmirror.com.ph
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
Asia |
| 281 |
Challenges
challenges.fr
|
F | 18 | 9/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 282 |
CNN
cnn.com
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
US |
| 283 |
Corriere della Sera
corriere.it
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 284 |
Crikey
crikey.com.au
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
Oceania |
| 285 |
Deccan Herald
deccanherald.com
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
Asia |
| 286 |
Der Standard
derstandard.at
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 287 |
The Economist
economist.com
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
UK |
| 288 |
Hankyoreh English
english.hani.co.kr
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 289 |
Financial Post
financialpost.com
|
F | 18 | 9/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 290 |
Goteborgs-Posten
gp.se
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 291 |
The Hankyoreh
hani.co.kr
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 292 |
Hong Kong Free Press
hongkongfp.com
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Meta-ExternalAgent
|
Asia |
| 293 |
HuffPost UK
huffingtonpost.co.uk
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
UK |
| 294 |
Il Sole 24 Ore
ilsole24ore.com
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
Google-Extended
PerplexityBot
CCBot
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 295 |
The Island
island.lk
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 296 |
Regina Leader-Post
leaderpost.com
|
F | 18 | 9/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 297 |
Le Temps
letemps.ch
|
F | 18 | 9/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 298 |
London Free Press
lfpress.com
|
F | 18 | 9/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 299 |
Berliner Morgenpost
morgenpost.de
|
F | 18 | 9/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 300 |
Nation (Kenya)
nation.africa
|
F | 18 | 9/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!