Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 951 |
Le360
le360.ma
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 952 |
Leadership (Nigeria)
leadership.ng
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 953 |
L'Echo
lecho.be
|
A | 100 | 0/11 | Allows all | EU |
| 954 |
L'Economiste (Morocco)
leconomiste.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 955 |
Le Point
lepoint.fr
|
A | 100 | 0/11 | Allows all | EU |
| 956 |
Libertatea
libertatea.ro
|
A | 100 | 0/11 | Allows all | EU |
| 957 |
Libya Herald
libyaherald.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 958 |
LIGA.net
liga.net
|
A | 100 | 0/11 | Allows all | EU |
| 959 |
Limerick Leader
limerickleader.ie
|
A | 100 | 0/11 | Allows all | Ireland |
| 960 |
Maclean's
macleans.ca
|
A | 100 | 0/11 | Allows all | CA |
| 961 |
MacRumors
macrumors.com
|
A | 100 | 0/11 | Allows all | US |
| 962 |
Magyar Nemzet
magyarnemzet.hu
|
A | 100 | 0/11 | Allows all | EU |
| 963 |
Malay Mail
malaymail.com
|
A | 100 | 0/11 | Allows all | Asia |
| 964 |
Malaysiakini
malaysiakini.com
|
A | 100 | 0/11 | Allows all | Asia |
| 965 |
The Manila Times
manilatimes.net
|
A | 100 | 0/11 | Allows all | Asia |
| 966 |
Masrawy
masrawy.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 967 |
Manila Bulletin
mb.com.ph
|
A | 100 | 0/11 | Allows all | Asia |
| 968 |
Morgunbladid
mbl.is
|
A | 100 | 0/11 | Allows all | EU |
| 969 |
The Morning Call (Allentown)
mcall.com
|
A | 100 | 0/11 | Allows all | US |
| 970 |
Mediafax
mediafax.ro
|
A | 100 | 0/11 | Allows all | EU |
| 971 |
Mediazona
mediazona.ca
|
A | 100 | 0/11 | Allows all | EU |
| 972 |
Meduza
meduza.io
|
A | 100 | 0/11 | Allows all | EU |
| 973 |
Metropoles
metropoles.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 974 |
Mail & Guardian
mg.co.za
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 975 |
Michael West Media
michaelwest.com.au
|
A | 100 | 0/11 | Allows all | Oceania |
| 976 |
Mid-Day
mid-day.com
|
A | 100 | 0/11 | Allows all | Asia |
| 977 |
Middle East Eye
middleeasteye.net
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 978 |
Middle East Monitor
middleeastmonitor.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 979 |
Minuto Uno
minutouno.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 980 |
Missoulian
missoulian.com
|
A | 100 | 0/11 | Allows all | US |
| 981 |
MLB.com
mlb.com
|
A | 100 | 0/11 | Allows all | US |
| 982 |
Modern Healthcare
modernhealthcare.com
|
A | 100 | 0/11 | Allows all | US |
| 983 |
Moneycontrol
moneycontrol.com
|
A | 100 | 0/11 | Allows all | Asia |
| 984 |
Daily Monitor (Uganda)
monitor.co.ug
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 985 |
Monocle
monocle.com
|
A | 100 | 0/11 | Allows all | UK |
| 986 |
Montevideo Portal
montevideo.com.uy
|
A | 100 | 0/11 | Allows all | LatAm |
| 987 |
Morningstar
morningstar.com
|
A | 100 | 0/11 | Allows all | US |
| 988 |
Morning Star
morningstaronline.co.uk
|
A | 100 | 0/11 | Allows all | UK |
| 989 |
Morocco World News
moroccoworldnews.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 990 |
Mothership
mothership.sg
|
A | 100 | 0/11 | Allows all | Asia |
| 991 |
Midland Reporter-Telegram
mrt.com
|
A | 100 | 0/11 | Allows all | US |
| 992 |
Naftemporiki
naftemporiki.gr
|
A | 100 | 0/11 | Allows all | EU |
| 993 |
Naharnet
naharnet.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 994 |
Narcity
narcity.com
|
A | 100 | 0/11 | Allows all | CA |
| 995 |
NASCAR
nascar.com
|
A | 100 | 0/11 | Allows all | US |
| 996 |
The Nation
nation.com.pk
|
A | 100 | 0/11 | Allows all | Asia |
| 997 |
Nation.Cymru
nation.cymru
|
A | 100 | 0/11 | Allows all | UK |
| 998 |
National Geographic
nationalgeographic.com
|
A | 100 | 0/11 | Allows all | US |
| 999 |
National Review
nationalreview.com
|
A | 100 | 0/11 | Allows all | US |
| 1000 |
The Nation Thailand
nationthailand.com
|
A | 100 | 0/11 | Allows all | Asia |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!