Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 801 |
Daily Maverick
dailymaverick.co.za
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 802 |
Daily Mirror
dailymirror.lk
|
A | 100 | 0/11 | Allows all | Asia |
| 803 |
Daily News (Sri Lanka)
dailynews.lk
|
A | 100 | 0/11 | Allows all | Asia |
| 804 |
Daily News Egypt
dailynewsegypt.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 805 |
Daily Press (Newport News)
dailypress.com
|
A | 100 | 0/11 | Allows all | US |
| 806 |
Daily Sabah
dailysabah.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 807 |
The Daily Star (Lebanon)
dailystar.com.lb
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 808 |
The Daily Wire
dailywire.com
|
A | 100 | 0/11 | Allows all | US |
| 809 |
Dawn
dawn.com
|
A | 100 | 0/11 | Allows all | Asia |
| 810 |
Deccan Chronicle
deccanchronicle.com
|
A | 100 | 0/11 | Allows all | Asia |
| 811 |
Detik
detik.com
|
A | 100 | 0/11 | Allows all | Asia |
| 812 |
Deutschlandfunk
deutschlandfunk.de
|
A | 100 | 0/11 | Allows all | EU |
| 813 |
Diario Financiero
df.cl
|
A | 100 | 0/11 | Allows all | LatAm |
| 814 |
Dhaka Tribune
dhakatribune.com
|
A | 100 | 0/11 | Allows all | Asia |
| 815 |
Diario do Nordeste
diariodonordeste.verdesmares.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 816 |
Digi24
digi24.ro
|
A | 100 | 0/11 | Allows all | EU |
| 817 |
Digiday
digiday.com
|
A | 100 | 0/11 | Allows all | US |
| 818 |
Digital Trends
digitaltrends.com
|
A | 100 | 0/11 | Allows all | US |
| 819 |
Discover Magazine
discovermagazine.com
|
A | 100 | 0/11 | Allows all | US |
| 820 |
Diário de Notícias
dn.pt
|
A | 100 | 0/11 | Allows all | EU |
| 821 |
The Dong-A Ilbo
donga.com
|
A | 100 | 0/11 | Allows all | Asia |
| 822 |
Deutsche Presse-Agentur
dpa.com
|
A | 100 | 0/11 | Allows all | Global |
| 823 |
Dunya News
dunyanews.tv
|
A | 100 | 0/11 | Allows all | Asia |
| 824 |
DV
dv.is
|
A | 100 | 0/11 | Allows all | EU |
| 825 |
Dziennik Gazeta Prawna
dziennik.pl
|
A | 100 | 0/11 | Allows all | EU |
| 826 |
ECO
eco.sapo.pt
|
A | 100 | 0/11 | Allows all | EU |
| 827 |
EFE
efe.com
|
A | 100 | 0/11 | Allows all | Global |
| 828 |
Efecto Cocuyo
efectococuyo.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 829 |
Egypt Independent
egyptindependent.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 830 |
Egypt Today
egypttoday.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 831 |
EJ Insight
ejinsight.com
|
A | 100 | 0/11 | Allows all | Asia |
| 832 |
El Nacional
el-nacional.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 833 |
El Comercio
elcomercio.pe
|
A | 100 | 0/11 | Allows all | LatAm |
| 834 |
elDiario.es
eldiario.es
|
A | 100 | 0/11 | Allows all | EU |
| 835 |
elDiarioAR
eldiarioar.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 836 |
El Economista
eleconomista.com.mx
|
A | 100 | 0/11 | Allows all | LatAm |
| 837 |
El Economista
eleconomista.es
|
A | 100 | 0/11 | Allows all | EU |
| 838 |
El Español
elespanol.com
|
A | 100 | 0/11 | Allows all | EU |
| 839 |
El Espectador
elespectador.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 840 |
El Financiero
elfinanciero.com.mx
|
A | 100 | 0/11 | Allows all | LatAm |
| 841 |
El Mostrador
elmostrador.cl
|
A | 100 | 0/11 | Allows all | LatAm |
| 842 |
El Nacional (alt)
elnacional.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 843 |
El Norte
elnorte.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 844 |
El Observador
elobservador.com.uy
|
A | 100 | 0/11 | Allows all | LatAm |
| 845 |
El Pais (Cali)
elpais.com.co
|
A | 100 | 0/11 | Allows all | LatAm |
| 846 |
El Pais (Uruguay)
elpais.com.uy
|
A | 100 | 0/11 | Allows all | LatAm |
| 847 |
El Popular
elpopular.pe
|
A | 100 | 0/11 | Allows all | LatAm |
| 848 |
El Sol de Mexico
elsoldemexico.com.mx
|
A | 100 | 0/11 | Allows all | LatAm |
| 849 |
Estado de Minas
em.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 850 |
Endpoints News
endpts.com
|
A | 100 | 0/11 | Allows all | US |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!