Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 1051 |
The Philippine Star
philstar.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1052 |
Phys.org
phys.org
|
A | 100 | 0/11 | Allows all | US |
| 1053 |
The Virginian-Pilot
pilotonline.com
|
A | 100 | 0/11 | Allows all | US |
| 1054 |
Poder360
poder360.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 1055 |
Politico
politico.com
|
A | 100 | 0/11 | Allows all | US |
| 1056 |
Popular Science
popsci.com
|
A | 100 | 0/11 | Allows all | US |
| 1057 |
Portfolio
portfolio.hu
|
A | 100 | 0/11 | Allows all | EU |
| 1058 |
Pittsburgh Post-Gazette
post-gazette.com
|
A | 100 | 0/11 | Allows all | US |
| 1059 |
Ukrainska Pravda
pravda.com.ua
|
A | 100 | 0/11 | Allows all | EU |
| 1060 |
Pravda
pravda.sk
|
A | 100 | 0/11 | Allows all | EU |
| 1061 |
Premium Times
premiumtimesng.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1062 |
The Press and Journal
pressandjournal.co.uk
|
A | 100 | 0/11 | Allows all | UK |
| 1063 |
PA Media
pressassociation.com
|
A | 100 | 0/11 | Allows all | UK |
| 1064 |
The Press (Christchurch)
pressreader.com
|
A | 100 | 0/11 | Allows all | Oceania |
| 1065 |
Press TV
presstv.ir
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1066 |
Proceso
proceso.com.mx
|
A | 100 | 0/11 | Allows all | LatAm |
| 1067 |
Proekt
proekt.media
|
A | 100 | 0/11 | Allows all | EU |
| 1068 |
Pro Football Talk
profootballtalk.nbcsports.com
|
A | 100 | 0/11 | Allows all | US |
| 1069 |
ProPublica
propublica.org
|
A | 100 | 0/11 | Allows all | US |
| 1070 |
Prothom Alo
prothomalo.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1071 |
Protocol
protocol.com
|
A | 100 | 0/11 | Allows all | US |
| 1072 |
Proto Thema
protothema.gr
|
A | 100 | 0/11 | Allows all | EU |
| 1073 |
Público
publico.es
|
A | 100 | 0/11 | Allows all | EU |
| 1074 |
Público (Portugal)
publico.pt
|
A | 100 | 0/11 | Allows all | EU |
| 1075 |
Pulzo
pulzo.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 1076 |
The Punch
punchng.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1077 |
PYMNTS
pymnts.com
|
A | 100 | 0/11 | Allows all | US |
| 1078 |
Quad-City Times
qctimes.com
|
A | 100 | 0/11 | Allows all | US |
| 1079 |
Quanta Magazine
quantamagazine.org
|
A | 100 | 0/11 | Allows all | US |
| 1080 |
Quartz
quartz.com
|
A | 100 | 0/11 | Allows all | US |
| 1081 |
Quartz
qz.com
|
A | 100 | 0/11 | Allows all | Global |
| 1082 |
R7
r7.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 1083 |
Radio-Canada
radio-canada.ca
|
A | 100 | 0/11 | Allows all | CA |
| 1084 |
RAI
rai.it
|
A | 100 | 0/11 | Allows all | EU |
| 1085 |
La Razon
razon.com.mx
|
A | 100 | 0/11 | Allows all | LatAm |
| 1086 |
RCN Radio
rcnradio.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 1087 |
RealClearPolitics
realclearpolitics.com
|
A | 100 | 0/11 | Allows all | US |
| 1088 |
Reason
reason.com
|
A | 100 | 0/11 | Allows all | US |
| 1089 |
Rediff News
rediff.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1090 |
Reforma
reforma.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 1091 |
La Republica (Uruguay)
republica.com.uy
|
A | 100 | 0/11 | Allows all | LatAm |
| 1092 |
Respekt
respekt.cz
|
A | 100 | 0/11 | Allows all | EU |
| 1093 |
Retail Dive
retaildive.com
|
A | 100 | 0/11 | Allows all | US |
| 1094 |
RICE Media
ricemedia.co
|
A | 100 | 0/11 | Allows all | Asia |
| 1095 |
Richmond Times-Dispatch
richmond.com
|
A | 100 | 0/11 | Allows all | US |
| 1096 |
Rigzone
rigzone.com
|
A | 100 | 0/11 | Allows all | US |
| 1097 |
Romania Insider
romania-insider.com
|
A | 100 | 0/11 | Allows all | EU |
| 1098 |
Rzeczpospolita
rp.pl
|
A | 100 | 0/11 | Allows all | EU |
| 1099 |
RT
rt.com
|
A | 100 | 0/11 | Allows all | Global |
| 1100 |
RTHK
rthk.hk
|
A | 100 | 0/11 | Allows all | Asia |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!