Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 601 |
Tampa Bay Times
tampabay.com
|
C | 64 | 4/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
|
US |
| 602 |
TechRadar
techradar.com
|
C | 64 | 4/11 |
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 603 |
The Week (US)
theweek.com
|
C | 64 | 4/11 |
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 604 |
Tom's Hardware
tomshardware.com
|
C | 64 | 4/11 |
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 605 |
Tortoise Media
tortoisemedia.com
|
C | 64 | 4/11 |
GPTBot
ChatGPT-User
Google-Extended
CCBot
|
UK |
| 606 |
Us Weekly
usmagazine.com
|
C | 64 | 4/11 |
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 607 |
Vanguardia (Colombia)
vanguardia.com
|
C | 64 | 4/11 |
GPTBot
ChatGPT-User
Google-Extended
CCBot
|
LatAm |
| 608 |
VentureBeat
venturebeat.com
|
C | 64 | 4/11 |
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
US |
| 609 |
The A.V. Club
avclub.com
|
B | 73 | 3/11 |
GPTBot
ChatGPT-User
Applebot-Extended
|
US |
| 610 |
Axios
axios.com
|
B | 73 | 3/11 |
CCBot
Bytespider
Amazonbot
|
US |
| 611 |
Bleacher Report
bleacherreport.com
|
B | 73 | 3/11 |
GPTBot
Google-Extended
CCBot
|
US |
| 612 |
BleepingComputer
bleepingcomputer.com
|
B | 73 | 3/11 |
ClaudeBot
CCBot
Applebot-Extended
|
US |
| 613 |
Business Insider
businessinsider.com
|
B | 73 | 3/11 |
ClaudeBot
CCBot
Bytespider
|
US |
| 614 |
CartaCapital
cartacapital.com.br
|
B | 73 | 3/11 |
GPTBot
Google-Extended
CCBot
|
LatAm |
| 615 |
Fakt
fakt.pl
|
B | 73 | 3/11 |
Bytespider
Amazonbot
Meta-ExternalAgent
|
EU |
| 616 |
Free Malaysia Today
freemalaysiatoday.com
|
B | 73 | 3/11 |
CCBot
Bytespider
Amazonbot
|
Asia |
| 617 |
Gazeta do Povo
gazetadopovo.com.br
|
B | 73 | 3/11 |
GPTBot
PerplexityBot
CCBot
|
LatAm |
| 618 |
Gizmodo
gizmodo.com
|
B | 73 | 3/11 |
CCBot
Bytespider
Meta-ExternalAgent
|
US |
| 619 |
O Globo / G1
globo.com
|
B | 73 | 3/11 |
GPTBot
Google-Extended
CCBot
|
LatAm |
| 620 |
Harvard Business Review
hbr.org
|
B | 73 | 3/11 |
GPTBot
Google-Extended
CCBot
|
US |
| 621 |
Irish Examiner
irishexaminer.com
|
B | 73 | 3/11 |
GPTBot
ChatGPT-User
CCBot
|
Ireland |
| 622 |
Kotaku
kotaku.com
|
B | 73 | 3/11 |
CCBot
Bytespider
Meta-ExternalAgent
|
US |
| 623 |
La Stampa
lastampa.it
|
B | 73 | 3/11 |
PerplexityBot
Bytespider
Applebot-Extended
|
EU |
| 624 |
Los Angeles Times
latimes.com
|
B | 73 | 3/11 |
GPTBot
ClaudeBot
Bytespider
|
US |
| 625 |
L'Orient-Le Jour
lorientlejour.com
|
B | 73 | 3/11 |
OAI-SearchBot
Bytespider
Meta-ExternalAgent
|
MENA-Africa |
| 626 |
Onet
onet.pl
|
B | 73 | 3/11 |
Bytespider
Amazonbot
Meta-ExternalAgent
|
EU |
| 627 |
ABP Live
scribd.com
|
B | 73 | 3/11 |
ClaudeBot
CCBot
Bytespider
|
Asia |
| 628 |
Sud Ouest
sudouest.fr
|
B | 73 | 3/11 |
GPTBot
ChatGPT-User
CCBot
|
EU |
| 629 |
The Diplomat
thediplomat.com
|
B | 73 | 3/11 |
GPTBot
Google-Extended
CCBot
|
Asia |
| 630 |
The News International
thenews.com.pk
|
B | 73 | 3/11 |
ClaudeBot
Bytespider
Amazonbot
|
Asia |
| 631 |
TV Noviny (Markiza)
tvnoviny.sk
|
B | 73 | 3/11 |
GPTBot
ChatGPT-User
CCBot
|
EU |
| 632 |
Yabiladi
yabiladi.com
|
B | 73 | 3/11 |
GPTBot
ClaudeBot
Bytespider
|
MENA-Africa |
| 633 |
1News (TVNZ)
1news.co.nz
|
B | 82 | 2/11 |
GPTBot
CCBot
|
Oceania |
| 634 |
7NEWS
7news.com.au
|
B | 82 | 2/11 |
GPTBot
ChatGPT-User
|
Oceania |
| 635 |
Aktuality.sk
aktuality.sk
|
B | 82 | 2/11 |
Bytespider
Amazonbot
|
EU |
| 636 |
Berlingske
berlingske.dk
|
B | 82 | 2/11 |
GPTBot
CCBot
|
EU |
| 637 |
Blu Radio
bluradio.com
|
B | 82 | 2/11 |
GPTBot
ChatGPT-User
|
LatAm |
| 638 |
BT
bt.dk
|
B | 82 | 2/11 |
GPTBot
CCBot
|
EU |
| 639 |
Caixin Global
caixinglobal.com
|
B | 82 | 2/11 |
GPTBot
CCBot
|
Asia |
| 640 |
Deadspin
deadspin.com
|
B | 82 | 2/11 |
GPTBot
CCBot
|
US |
| 641 |
Euronews
euronews.com
|
B | 82 | 2/11 |
GPTBot
CCBot
|
EU |
| 642 |
Fortune
fortune.com
|
B | 82 | 2/11 |
Google-Extended
CCBot
|
US |
| 643 |
Gazeta.pl
gazeta.pl
|
B | 82 | 2/11 |
GPTBot
CCBot
|
EU |
| 644 |
Golem.de
golem.de
|
B | 82 | 2/11 |
Bytespider
Amazonbot
|
EU |
| 645 |
Il Post
ilpost.it
|
B | 82 | 2/11 |
GPTBot
PerplexityBot
|
EU |
| 646 |
Inter Press Service
ipsnews.net
|
B | 82 | 2/11 |
GPTBot
CCBot
|
Global |
| 647 |
The Irish Times
irishtimes.com
|
B | 82 | 2/11 |
GPTBot
ChatGPT-User
|
EU |
| 648 |
Jakarta Globe
jakartaglobe.id
|
B | 82 | 2/11 |
CCBot
Meta-ExternalAgent
|
Asia |
| 649 |
Jornal de Notícias
jn.pt
|
B | 82 | 2/11 |
GPTBot
CCBot
|
EU |
| 650 |
The Korea Times
koreatimes.co.kr
|
B | 82 | 2/11 |
Bytespider
Amazonbot
|
Asia |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!