Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 1001 |
NBC Sports
nbcsports.com
|
A | 100 | 0/11 | Allows all | US |
| 1002 |
National Business Review
nbr.co.nz
|
A | 100 | 0/11 | Allows all | Oceania |
| 1003 |
NDTV
ndtv.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1004 |
Nepszava
nepszava.hu
|
A | 100 | 0/11 | Allows all | EU |
| 1005 |
NerdWallet
nerdwallet.com
|
A | 100 | 0/11 | Allows all | US |
| 1006 |
New Age
newagebd.net
|
A | 100 | 0/11 | Allows all | Asia |
| 1007 |
The New Arab
newarab.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1008 |
New Haven Register
newhavenregister.com
|
A | 100 | 0/11 | Allows all | US |
| 1009 |
The New Indian Express
newindianexpress.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1010 |
The New Republic
newrepublic.com
|
A | 100 | 0/11 | Allows all | US |
| 1011 |
Sky News
news.sky.com
|
A | 100 | 0/11 | Allows all | UK |
| 1012 |
News18
news18.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1013 |
News247
news247.gr
|
A | 100 | 0/11 | Allows all | EU |
| 1014 |
News First
newsfirst.lk
|
A | 100 | 0/11 | Allows all | Asia |
| 1015 |
Newshub
newshub.co.nz
|
A | 100 | 0/11 | Allows all | Oceania |
| 1016 |
Newsit
newsit.gr
|
A | 100 | 0/11 | Allows all | EU |
| 1017 |
The News Letter (Belfast)
newsletter.co.uk
|
A | 100 | 0/11 | Allows all | UK |
| 1018 |
Newstalk
newstalk.com
|
A | 100 | 0/11 | Allows all | Ireland |
| 1019 |
Newsweek Polska
newsweek.pl
|
A | 100 | 0/11 | Allows all | EU |
| 1020 |
The New Times (Rwanda)
newtimes.co.rw
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1021 |
New Vision (Uganda)
newvision.co.ug
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1022 |
NFL.com
nfl.com
|
A | 100 | 0/11 | Allows all | US |
| 1023 |
Nhan Dan
nhandan.vn
|
A | 100 | 0/11 | Allows all | Asia |
| 1024 |
NHL.com
nhl.com
|
A | 100 | 0/11 | Allows all | US |
| 1025 |
Nikkei
nikkei.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1026 |
Notes from Poland
notesfrompoland.com
|
A | 100 | 0/11 | Allows all | EU |
| 1027 |
Novaya Gazeta Europe
novayagazeta.eu
|
A | 100 | 0/11 | Allows all | EU |
| 1028 |
Novinky.cz
novinky.cz
|
A | 100 | 0/11 | Allows all | EU |
| 1029 |
NV (Novoe Vremya)
nv.ua
|
A | 100 | 0/11 | Allows all | EU |
| 1030 |
New York Daily News
nydailynews.com
|
A | 100 | 0/11 | Allows all | US |
| 1031 |
Observer BD
observerbd.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1032 |
Otago Daily Times
odt.co.nz
|
A | 100 | 0/11 | Allows all | Oceania |
| 1033 |
oe24
oe24.at
|
A | 100 | 0/11 | Allows all | EU |
| 1034 |
Oman Observer
omanobserver.om
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 1035 |
Open
open.online
|
A | 100 | 0/11 | Allows all | EU |
| 1036 |
openDemocracy
opendemocracy.net
|
A | 100 | 0/11 | Allows all | UK |
| 1037 |
Origo
origo.hu
|
A | 100 | 0/11 | Allows all | EU |
| 1038 |
Orlando Sentinel
orlandosentinel.com
|
A | 100 | 0/11 | Allows all | US |
| 1039 |
Outlook India
outlookindia.com
|
A | 100 | 0/11 | Allows all | Asia |
| 1040 |
OZY
ozy.com
|
A | 100 | 0/11 | Allows all | Global |
| 1041 |
Pagina 12
pagina12.com.ar
|
A | 100 | 0/11 | Allows all | LatAm |
| 1042 |
Pakistan Today
pakistantoday.com.pk
|
A | 100 | 0/11 | Allows all | Asia |
| 1043 |
PCMag
pcmag.com
|
A | 100 | 0/11 | Allows all | US |
| 1044 |
The Press-Enterprise
pe.com
|
A | 100 | 0/11 | Allows all | US |
| 1045 |
PE Hub
pehub.com
|
A | 100 | 0/11 | Allows all | US |
| 1046 |
Perfil
perfil.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 1047 |
PerthNow
perthnow.com.au
|
A | 100 | 0/11 | Allows all | Oceania |
| 1048 |
Peru21
peru21.pe
|
A | 100 | 0/11 | Allows all | LatAm |
| 1049 |
Pew Research Center
pewresearch.org
|
A | 100 | 0/11 | Allows all | US |
| 1050 |
Pro Football Focus
pff.com
|
A | 100 | 0/11 | Allows all | US |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!