Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 401 |
SVT (Sveriges Television)
svt.se
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 402 |
Tal Cual
talcualdigital.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
LatAm |
| 403 |
talkSPORT
talksport.com
|
D | 27 | 8/11 |
OAI-SearchBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
UK |
| 404 |
The Athletic
theathletic.com
|
D | 27 | 8/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
US |
| 405 |
The Atlantic
theatlantic.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 406 |
TheCable
thecable.ng
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 407 |
The EastAfrican
theeastafrican.co.ke
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 408 |
The Financial Express (BD)
thefinancialexpress.com.bd
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 409 |
The Local Sweden
thelocal.se
|
D | 27 | 8/11 |
GPTBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
EU |
| 410 |
The Malaysian Reserve
themalaysianreserve.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 411 |
The New Humanitarian
thenewhumanitarian.org
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Global |
| 412 |
The Register
theregister.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 413 |
The Reporter (Ethiopia)
thereporterethiopia.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 414 |
The Saturday Paper
thesaturdaypaper.com.au
|
D | 27 | 8/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
Oceania |
| 415 |
The Verge
theverge.com
|
D | 27 | 8/11 |
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 416 |
Thrillist
thrillist.com
|
D | 27 | 8/11 |
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 417 |
Travel + Leisure
travelandleisure.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 418 |
Trend
trend.sk
|
D | 27 | 8/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
EU |
| 419 |
UOL
uol.com.br
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
LatAm |
| 420 |
Vanity Fair
vanityfair.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 421 |
VG (Verdens Gang)
vg.no
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 422 |
Vogue
vogue.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 423 |
Vox
vox.com
|
D | 27 | 8/11 |
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 424 |
Vulture
vulture.com
|
D | 27 | 8/11 |
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 425 |
WIRED UK
wired.co.uk
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 426 |
WIRED
wired.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 427 |
WirtschaftsWoche
wiwo.de
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 428 |
Yle
yle.fi
|
D | 27 | 8/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
EU |
| 429 |
Die Zeit
zeit.de
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 430 |
20 Minutos
20minutos.es
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 431 |
Albuquerque Journal
abqjournal.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 432 |
Anchorage Daily News
adn.com
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
|
US |
| 433 |
Barron's
barrons.com
|
D | 36 | 7/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
US |
| 434 |
Birmingham Mail
birminghammail.co.uk
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 435 |
Bayerischer Rundfunk
br.de
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 436 |
Bristol Post
bristolpost.co.uk
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 437 |
Cambridge News
cambridge-news.co.uk
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 438 |
Chicago Sun-Times
chicago.suntimes.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
US |
| 439 |
Chicago Sun-Times
chicagosuntimes.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
US |
| 440 |
The Chronicle (Newcastle)
chroniclelive.co.uk
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 441 |
CIO
cio.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 442 |
Computerworld
computerworld.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 443 |
Crunchbase News
crunchbase.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 444 |
Daily Record
dailyrecord.co.uk
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 445 |
Daily Star
dailystar.co.uk
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 446 |
Dark Reading
darkreading.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 447 |
Die Presse
diepresse.com
|
D | 36 | 7/11 |
GPTBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
EU |
| 448 |
Huddersfield Examiner
examinerlive.co.uk
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 449 |
Fast Company
fastcompany.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
US |
| 450 |
Finans
finans.dk
|
D | 36 | 7/11 |
GPTBot
ChatGPT-User
Google-Extended
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
EU |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!