Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 901 |
Holod Media
holod.media
|
A | 100 | 0/11 | Allows all | EU |
| 902 |
HotNews
hotnews.ro
|
A | 100 | 0/11 | Allows all | EU |
| 903 |
HR Dive
hrdive.com
|
A | 100 | 0/11 | Allows all | US |
| 904 |
Hromadske
hromadske.ua
|
A | 100 | 0/11 | Allows all | EU |
| 905 |
Hurriyet Daily News
hurriyetdailynews.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 906 |
HVG
hvg.hu
|
A | 100 | 0/11 | Allows all | EU |
| 907 |
i24NEWS
i24news.tv
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 908 |
Iceland Review
icelandreview.com
|
A | 100 | 0/11 | Allows all | EU |
| 909 |
IEEE Spectrum
ieee.org
|
A | 100 | 0/11 | Allows all | US |
| 910 |
iEfimerida
iefimerida.gr
|
A | 100 | 0/11 | Allows all | EU |
| 911 |
iG
ig.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 912 |
Il Fatto Quotidiano
ilfattoquotidiano.it
|
A | 100 | 0/11 | Allows all | EU |
| 913 |
Il Giornale
ilgiornale.it
|
A | 100 | 0/11 | Allows all | EU |
| 914 |
In.gr
in.gr
|
A | 100 | 0/11 | Allows all | EU |
| 915 |
The Independent
independent.co.uk
|
A | 100 | 0/11 | Allows all | UK |
| 916 |
India Today
indiatoday.in
|
A | 100 | 0/11 | Allows all | Asia |
| 917 |
Infobae
infobae.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 918 |
El Informador
informador.mx
|
A | 100 | 0/11 | Allows all | LatAm |
| 919 |
The Philadelphia Inquirer
inquirer.com
|
A | 100 | 0/11 | Allows all | US |
| 920 |
Inside Higher Ed
insidehighered.com
|
A | 100 | 0/11 | Allows all | US |
| 921 |
Institutional Investor
institutionalinvestor.com
|
A | 100 | 0/11 | Allows all | US |
| 922 |
interest.co.nz
interest.co.nz
|
A | 100 | 0/11 | Allows all | Oceania |
| 923 |
Interia
interia.pl
|
A | 100 | 0/11 | Allows all | EU |
| 924 |
iPolitics
ipolitics.ca
|
A | 100 | 0/11 | Allows all | CA |
| 925 |
Israel Hayom
israelhayom.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 926 |
iStories
istories.media
|
A | 100 | 0/11 | Allows all | EU |
| 927 |
The Japan News
japannews.yomiuri.co.jp
|
A | 100 | 0/11 | Allows all | Asia |
| 928 |
The Japan Times
japantimes.co.jp
|
A | 100 | 0/11 | Allows all | Asia |
| 929 |
Japan Today
japantoday.com
|
A | 100 | 0/11 | Allows all | Asia |
| 930 |
JOE.ie
joe.ie
|
A | 100 | 0/11 | Allows all | Ireland |
| 931 |
The Jordan Times
jordantimes.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 932 |
Winston-Salem Journal
journalnow.com
|
A | 100 | 0/11 | Allows all | US |
| 933 |
The Jerusalem Post
jpost.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 934 |
KCCI Des Moines
kcci.com
|
A | 100 | 0/11 | Allows all | US |
| 935 |
Khaosod English
khaosodenglish.com
|
A | 100 | 0/11 | Allows all | Asia |
| 936 |
The Korea Herald
koreaherald.com
|
A | 100 | 0/11 | Allows all | Asia |
| 937 |
Krebs on Security
krebsonsecurity.com
|
A | 100 | 0/11 | Allows all | US |
| 938 |
Kronen Zeitung
krone.at
|
A | 100 | 0/11 | Allows all | EU |
| 939 |
Kölner Stadt-Anzeiger
ksta.de
|
A | 100 | 0/11 | Allows all | EU |
| 940 |
Kurdistan 24
kurdistan24.net
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 941 |
Kurier
kurier.at
|
A | 100 | 0/11 | Allows all | EU |
| 942 |
The Kyiv Independent
kyivindependent.com
|
A | 100 | 0/11 | Allows all | EU |
| 943 |
Kyiv Post
kyivpost.com
|
A | 100 | 0/11 | Allows all | EU |
| 944 |
La Capital
lacapital.com.ar
|
A | 100 | 0/11 | Allows all | LatAm |
| 945 |
la diaria
ladiaria.com.uy
|
A | 100 | 0/11 | Allows all | LatAm |
| 946 |
La Gaceta
lagaceta.com.ar
|
A | 100 | 0/11 | Allows all | LatAm |
| 947 |
La Nacion
lanacion.com.ar
|
A | 100 | 0/11 | Allows all | LatAm |
| 948 |
La Republica
larepublica.co
|
A | 100 | 0/11 | Allows all | LatAm |
| 949 |
La Republica
larepublica.pe
|
A | 100 | 0/11 | Allows all | LatAm |
| 950 |
La Voz de Galicia
lavozdegalicia.es
|
A | 100 | 0/11 | Allows all | EU |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!