Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 1151 |
The Scientist
the-scientist.com
|
A | 100 | 0/11 | Allows all | US |
| 1152 |
The Star (Kenya)
the-star.co.ke
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1153 |
The American Conservative
theamericanconservative.com
|
A | 100 | 0/11 | Allows all | Global |
| 1154 |
The Bell
thebell.io
|
A | 100 | 0/11 | Allows all | EU |
| 1155 |
The Better India
thebetterindia.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1156 |
The Blaze
theblaze.com
|
A | 100 | 0/11 | Allows all | US |
| 1157 |
The Bureau of Investigative Journalism
thebureauinvestigates.com
|
A | 100 | 0/11 | Allows all | UK |
| 1158 |
The Canary
thecanary.co
|
A | 100 | 0/11 | Allows all | UK |
| 1159 |
The Chronicle Herald
thechronicleherald.ca
|
A | 100 | 0/11 | Allows all | CA |
| 1160 |
The Citizen (Tanzania)
thecitizen.co.tz
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1161 |
The Cork News
thecork.ie
|
A | 100 | 0/11 | Allows all | Ireland |
| 1162 |
The Courier (Dundee)
thecourier.co.uk
|
A | 100 | 0/11 | Allows all | UK |
| 1163 |
The Daily Beast
thedailybeast.com
|
A | 100 | 0/11 | Allows all | US |
| 1164 |
The Daily Caller
thedailycaller.com
|
A | 100 | 0/11 | Allows all | US |
| 1165 |
The Daily Star
thedailystar.net
|
A | 100 | 0/11 | Allows all | Asia |
| 1166 |
The Fader
thefader.com
|
A | 100 | 0/11 | Allows all | US |
| 1167 |
The Federalist
thefederalist.com
|
A | 100 | 0/11 | Allows all | US |
| 1168 |
The Gazette (Cedar Rapids)
thegazette.com
|
A | 100 | 0/11 | Allows all | US |
| 1169 |
The Hacker News
thehackernews.com
|
A | 100 | 0/11 | Allows all | US |
| 1170 |
The Information
theinformation.com
|
A | 100 | 0/11 | Allows all | US |
| 1171 |
The Jakarta Post
thejakartapost.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1172 |
The Morning
themorning.lk
|
A | 100 | 0/11 | Allows all | Asia |
| 1173 |
The Moscow Times
themoscowtimes.com
|
A | 100 | 0/11 | Allows all | EU |
| 1174 |
The Narwhal
thenarwhal.ca
|
A | 100 | 0/11 | Allows all | CA |
| 1175 |
The Nation
thenation.com
|
A | 100 | 0/11 | Allows all | US |
| 1176 |
The National (UAE)
thenationalnews.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1177 |
The New Daily
thenewdaily.com.au
|
A | 100 | 0/11 | Allows all | Oceania |
| 1178 |
TNW
thenextweb.com
|
A | 100 | 0/11 | Allows all | EU |
| 1179 |
The Quint
thequint.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1180 |
The Ringer
theringer.com
|
A | 100 | 0/11 | Allows all | US |
| 1181 |
The Shovel
theshovel.com.au
|
A | 100 | 0/11 | Allows all | Oceania |
| 1182 |
The Standard
thestandard.com.hk
|
A | 100 | 0/11 | Allows all | Asia |
| 1183 |
The Star
thestar.com.my
|
A | 100 | 0/11 | Allows all | Asia |
| 1184 |
The Statesman
thestatesman.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1185 |
TheStreet
thestreet.com
|
A | 100 | 0/11 | Allows all | US |
| 1186 |
The Tab
thetab.com
|
A | 100 | 0/11 | Allows all | UK |
| 1187 |
The Telegram
thetelegram.com
|
A | 100 | 0/11 | Allows all | CA |
| 1188 |
The Tyee
thetyee.ca
|
A | 100 | 0/11 | Allows all | CA |
| 1189 |
The Walrus
thewalrus.ca
|
A | 100 | 0/11 | Allows all | CA |
| 1190 |
The West Australian
thewest.com.au
|
A | 100 | 0/11 | Allows all | Oceania |
| 1191 |
The Wire
thewire.in
|
A | 100 | 0/11 | Allows all | Asia |
| 1192 |
De Tijd
tijd.be
|
A | 100 | 0/11 | Allows all | EU |
| 1193 |
TIME
time.com
|
A | 100 | 0/11 | Allows all | US |
| 1194 |
Chattanooga Times Free Press
timesfreepress.com
|
A | 100 | 0/11 | Allows all | US |
| 1195 |
TimesLIVE
timeslive.co.za
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1196 |
The Times of Israel
timesofisrael.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1197 |
Times of Oman
timesofoman.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1198 |
TMZ
tmz.com
|
A | 100 | 0/11 | Allows all | US |
| 1199 |
Todo Noticias (TN)
tn.com.ar
|
A | 100 | 0/11 | Allows all | LatAm |
| 1200 |
The New Paper
tnp.sg
|
A | 100 | 0/11 | Allows all | Asia |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!