Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 1101 |
RTP
rtp.pt
|
A | 100 | 0/11 | Allows all | EU |
| 1102 |
RTVE
rtve.es
|
A | 100 | 0/11 | Allows all | EU |
| 1103 |
Rudaw
rudaw.net
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1104 |
Runrunes
runrun.es
|
A | 100 | 0/11 | Allows all | LatAm |
| 1105 |
SABC News
sabcnews.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1106 |
Salon
salon.com
|
A | 100 | 0/11 | Allows all | US |
| 1107 |
Samaa TV
samaa.tv
|
A | 100 | 0/11 | Allows all | Asia |
| 1108 |
Saudi Gazette
saudigazette.com.sa
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1109 |
ScienceDaily
sciencedaily.com
|
A | 100 | 0/11 | Allows all | US |
| 1110 |
South China Morning Post
scmp.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1111 |
The Scotsman
scotsman.com
|
A | 100 | 0/11 | Allows all | UK |
| 1112 |
Scroll.in
scroll.in
|
A | 100 | 0/11 | Allows all | Asia |
| 1113 |
SDP Noticias
sdpnoticias.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 1114 |
Semana
semana.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 1115 |
Seznam Zpravy
seznamzpravy.cz
|
A | 100 | 0/11 | Allows all | EU |
| 1116 |
Shanghai Daily (SHINE)
shine.cn
|
A | 100 | 0/11 | Allows all | Asia |
| 1117 |
Al-Shorouk
shorouknews.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1118 |
Sports Illustrated
si.com
|
A | 100 | 0/11 | Allows all | US |
| 1119 |
Sioux City Journal
siouxcityjournal.com
|
A | 100 | 0/11 | Allows all | US |
| 1120 |
Sky News
sky.com
|
A | 100 | 0/11 | Allows all | UK |
| 1121 |
Sky & Telescope
skyandtelescope.org
|
A | 100 | 0/11 | Allows all | US |
| 1122 |
Slashdot
slashdot.org
|
A | 100 | 0/11 | Allows all | US |
| 1123 |
Smithsonian Magazine
smithsonianmag.com
|
A | 100 | 0/11 | Allows all | US |
| 1124 |
The Spectator
spectator.co.uk
|
A | 100 | 0/11 | Allows all | UK |
| 1125 |
Caixin (alt)
spglobal.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1126 |
Evening Standard
standard.co.uk
|
A | 100 | 0/11 | Allows all | UK |
| 1127 |
The Standard (Kenya)
standardmedia.co.ke
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1128 |
Star Tribune
startribune.com
|
A | 100 | 0/11 | Allows all | US |
| 1129 |
Austin American-Statesman
statesman.com
|
A | 100 | 0/11 | Allows all | US |
| 1130 |
Stereogum
stereogum.com
|
A | 100 | 0/11 | Allows all | US |
| 1131 |
Subrayado
subrayado.com.uy
|
A | 100 | 0/11 | Allows all | LatAm |
| 1132 |
South Florida Sun Sentinel
sun-sentinel.com
|
A | 100 | 0/11 | Allows all | US |
| 1133 |
The Sun (Nigeria)
sunnewsonline.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1134 |
Supply Chain Dive
supplychaindive.com
|
A | 100 | 0/11 | Allows all | US |
| 1135 |
Sveriges Radio
sverigesradio.se
|
A | 100 | 0/11 | Allows all | EU |
| 1136 |
Radio Svoboda
svoboda.org
|
A | 100 | 0/11 | Allows all | EU |
| 1137 |
Swissinfo
swissinfo.ch
|
A | 100 | 0/11 | Allows all | EU |
| 1138 |
Syria Direct
syriadirect.org
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1139 |
Teletrece (T13)
t13.cl
|
A | 100 | 0/11 | Allows all | LatAm |
| 1140 |
Tunis Afrique Presse
tap.info.tn
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1141 |
TASS
tass.com
|
A | 100 | 0/11 | Allows all | Global |
| 1142 |
The Daily News (Longview)
tdn.com
|
A | 100 | 0/11 | Allows all | US |
| 1143 |
Tehran Times
tehrantimes.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1144 |
Telam
telam.com.ar
|
A | 100 | 0/11 | Allows all | LatAm |
| 1145 |
The Telegraph India
telegraphindia.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1146 |
TASR / Teraz.sk
teraz.sk
|
A | 100 | 0/11 | Allows all | EU |
| 1147 |
Terra
terra.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 1148 |
TGCom24
tgcom24.mediaset.it
|
A | 100 | 0/11 | Allows all | EU |
| 1149 |
Thai PBS World
thaipbsworld.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1150 |
Thanh Nien
thanhnien.vn
|
A | 100 | 0/11 | Allows all | Asia |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!