Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 501 |
The Irish Sun
thesun.ie
|
D | 36 | 7/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
Ireland |
| 502 |
The Times
thetimes.co.uk
|
D | 36 | 7/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
UK |
| 503 |
The Sunday Times
thetimes.com
|
D | 36 | 7/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
UK |
| 504 |
Tygodnik Powszechny
tygodnikpowszechny.pl
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 505 |
Vozpópuli
vozpopuli.com
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 506 |
WalesOnline
walesonline.co.uk
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 507 |
The Washington Post
washingtonpost.com
|
D | 36 | 7/11 |
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 508 |
Westdeutscher Rundfunk (WDR)
wdr.de
|
D | 36 | 7/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 509 |
Winnipeg Free Press
winnipegfreepress.com
|
D | 36 | 7/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
CCBot
Bytespider
Meta-ExternalAgent
|
CA |
| 510 |
The Wall Street Journal
wsj.com
|
D | 36 | 7/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
US |
| 511 |
Gazeta Wyborcza
wyborcza.pl
|
D | 36 | 7/11 |
GPTBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
EU |
| 512 |
Associated Press
apnews.com
|
D | 45 | 6/11 |
GPTBot
ClaudeBot
PerplexityBot
CCBot
Amazonbot
Applebot-Extended
|
US |
| 513 |
ARY News
arynews.tv
|
D | 45 | 6/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
Asia |
| 514 |
Houston Chronicle
chron.com
|
D | 45 | 6/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
US |
| 515 |
Connecticut Post
ctpost.com
|
D | 45 | 6/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
US |
| 516 |
Deseret News
deseret.com
|
D | 45 | 6/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Applebot-Extended
|
US |
| 517 |
Ahram Online
english.ahram.org.eg
|
D | 45 | 6/11 |
ClaudeBot
Google-Extended
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 518 |
Eyewitness News
ewn.co.za
|
D | 45 | 6/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 519 |
Daily Express
express.co.uk
|
D | 45 | 6/11 |
GPTBot
OAI-SearchBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
|
UK |
| 520 |
L'HuffPost Italia
huffingtonpost.it
|
D | 45 | 6/11 |
GPTBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
EU |
| 521 |
The Irish News
irishnews.com
|
D | 45 | 6/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 522 |
La Razón
larazon.es
|
D | 45 | 6/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 523 |
Le Dauphiné Libéré
ledauphine.com
|
D | 45 | 6/11 |
GPTBot
ChatGPT-User
Google-Extended
CCBot
Bytespider
Amazonbot
|
EU |
| 524 |
Le Monde
lemonde.fr
|
D | 45 | 6/11 |
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 525 |
Le Progrès
leprogres.fr
|
D | 45 | 6/11 |
GPTBot
ChatGPT-User
Google-Extended
CCBot
Bytespider
Amazonbot
|
EU |
| 526 |
Lettera43
lettera43.it
|
D | 45 | 6/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
CCBot
Bytespider
|
EU |
| 527 |
MyJoyOnline
myjoyonline.com
|
D | 45 | 6/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 528 |
ORF
orf.at
|
D | 45 | 6/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
|
EU |
| 529 |
Politiken
politiken.dk
|
D | 45 | 6/11 |
GPTBot
ChatGPT-User
Google-Extended
CCBot
Bytespider
Amazonbot
|
EU |
| 530 |
La Repubblica
repubblica.it
|
D | 45 | 6/11 |
GPTBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
EU |
| 531 |
RPP Noticias
rpp.pe
|
D | 45 | 6/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
|
LatAm |
| 532 |
San Francisco Chronicle
sfchronicle.com
|
D | 45 | 6/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
US |
| 533 |
Der Tagesspiegel
tagesspiegel.de
|
D | 45 | 6/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 534 |
The Intercept
theintercept.com
|
D | 45 | 6/11 |
GPTBot
ChatGPT-User
Google-Extended
PerplexityBot
CCBot
Amazonbot
|
US |
| 535 |
Times Union (Albany)
timesunion.com
|
D | 45 | 6/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
US |
| 536 |
9to5Google
9to5google.com
|
C | 55 | 5/11 |
GPTBot
ClaudeBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 537 |
9to5Mac
9to5mac.com
|
C | 55 | 5/11 |
GPTBot
ClaudeBot
CCBot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 538 |
ABC News
abcnews.go.com
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
Google-Extended
CCBot
Bytespider
|
US |
| 539 |
Al Jazeera
aljazeera.com
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
ClaudeBot
PerplexityBot
Bytespider
|
MENA-Africa |
| 540 |
Ara
ara.cat
|
C | 55 | 5/11 |
GPTBot
ClaudeBot
PerplexityBot
CCBot
Applebot-Extended
|
EU |
| 541 |
Diario AS
as.com
|
C | 55 | 5/11 |
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
|
EU |
| 542 |
Bernama
bernama.com
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
ClaudeBot
CCBot
Bytespider
|
Asia |
| 543 |
Bild
bild.de
|
C | 55 | 5/11 |
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 544 |
BlogTO
blogto.com
|
C | 55 | 5/11 |
OAI-SearchBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
CA |
| 545 |
BNN Bloomberg
bnnbloomberg.ca
|
C | 55 | 5/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
PerplexityBot
CCBot
|
CA |
| 546 |
BNR Nieuwsradio
bnr.nl
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
CCBot
|
EU |
| 547 |
Consequence
consequence.net
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
ClaudeBot
CCBot
Bytespider
|
US |
| 548 |
Cosmopolitan
cosmopolitan.com
|
C | 55 | 5/11 |
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
US |
| 549 |
CP24
cp24.com
|
C | 55 | 5/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
PerplexityBot
CCBot
|
CA |
| 550 |
CTV News
ctvnews.ca
|
C | 55 | 5/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
PerplexityBot
CCBot
|
CA |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!