Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 851 |
Engadget
engadget.com
|
A | 100 | 0/11 | Allows all | US |
| 852 |
Eos
eos.org
|
A | 100 | 0/11 | Allows all | US |
| 853 |
ERT
ert.gr
|
A | 100 | 0/11 | Allows all | EU |
| 854 |
O Estado de S. Paulo
estadao.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 855 |
Exame
exame.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 856 |
Excelsior
excelsior.com.mx
|
A | 100 | 0/11 | Allows all | LatAm |
| 857 |
Expansion
expansion.mx
|
A | 100 | 0/11 | Allows all | LatAm |
| 858 |
Express
express.de
|
A | 100 | 0/11 | Allows all | EU |
| 859 |
Extra.ie
extra.ie
|
A | 100 | 0/11 | Allows all | Ireland |
| 860 |
Fana Broadcasting
fanabc.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 861 |
Fanpage.it
fanpage.it
|
A | 100 | 0/11 | Allows all | EU |
| 862 |
Fierce Biotech
fiercebiotech.com
|
A | 100 | 0/11 | Allows all | US |
| 863 |
Fierce Pharma
fiercepharma.com
|
A | 100 | 0/11 | Allows all | US |
| 864 |
Firstpost
firstpost.com
|
A | 100 | 0/11 | Allows all | Asia |
| 865 |
The Florida Times-Union
floridatimesunion.com
|
A | 100 | 0/11 | Allows all | US |
| 866 |
Focus
focus.de
|
A | 100 | 0/11 | Allows all | EU |
| 867 |
Folha de S.Paulo
folha.uol.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 868 |
Foreign Affairs
foreignaffairs.com
|
A | 100 | 0/11 | Allows all | Global |
| 869 |
Foreign Policy
foreignpolicy.com
|
A | 100 | 0/11 | Allows all | Global |
| 870 |
Fox News
foxnews.com
|
A | 100 | 0/11 | Allows all | US |
| 871 |
Fox Sports
foxsports.com
|
A | 100 | 0/11 | Allows all | US |
| 872 |
Washington Free Beacon
freebeacon.com
|
A | 100 | 0/11 | Allows all | US |
| 873 |
The Free Press Journal
freepressjournal.in
|
A | 100 | 0/11 | Allows all | Asia |
| 874 |
Frettabladid
frettabladid.is
|
A | 100 | 0/11 | Allows all | EU |
| 875 |
Daily FT
ft.lk
|
A | 100 | 0/11 | Allows all | Asia |
| 876 |
G4Media
g4media.ro
|
A | 100 | 0/11 | Allows all | EU |
| 877 |
Gandul
gandul.ro
|
A | 100 | 0/11 | Allows all | EU |
| 878 |
The Gazette (Colorado Springs)
gazette.com
|
A | 100 | 0/11 | Allows all | US |
| 879 |
GB News
gbnews.com
|
A | 100 | 0/11 | Allows all | UK |
| 880 |
Gulf Daily News (Bahrain)
gdnonline.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 881 |
Geo News
geo.tv
|
A | 100 | 0/11 | Allows all | Asia |
| 882 |
Gestion
gestion.pe
|
A | 100 | 0/11 | Allows all | LatAm |
| 883 |
GhanaWeb
ghanaweb.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 884 |
The Gisborne Herald
gisborneherald.co.nz
|
A | 100 | 0/11 | Allows all | Oceania |
| 885 |
Global News
globalnews.ca
|
A | 100 | 0/11 | Allows all | CA |
| 886 |
Global Times
globaltimes.cn
|
A | 100 | 0/11 | Allows all | Asia |
| 887 |
GlobeNewswire
globenewswire.com
|
A | 100 | 0/11 | Allows all | US |
| 888 |
Globes
globes.co.il
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 889 |
GMA News
gmanetwork.com
|
A | 100 | 0/11 | Allows all | Asia |
| 890 |
Golf Digest
golfdigest.com
|
A | 100 | 0/11 | Allows all | US |
| 891 |
News & Record (Greensboro)
greensboro.com
|
A | 100 | 0/11 | Allows all | US |
| 892 |
Greentech Media
greentechmedia.com
|
A | 100 | 0/11 | Allows all | US |
| 893 |
Grist
grist.org
|
A | 100 | 0/11 | Allows all | Global |
| 894 |
The Guardian Nigeria
guardian.ng
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 895 |
Gulf Times
gulf-times.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 896 |
Gulf News
gulfnews.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 897 |
Heise Online
heise.de
|
A | 100 | 0/11 | Allows all | EU |
| 898 |
Hespress
hespress.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 899 |
Hirado
hirado.hu
|
A | 100 | 0/11 | Allows all | EU |
| 900 |
Hollywood Life
hollywoodlife.com
|
A | 100 | 0/11 | Allows all | US |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!