Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 751 |
Bellingcat
bellingcat.com
|
A | 100 | 0/11 | Allows all | UK |
| 752 |
Benzinga
benzinga.com
|
A | 100 | 0/11 | Allows all | US |
| 753 |
Berliner Zeitung
berliner-zeitung.de
|
A | 100 | 0/11 | Allows all | EU |
| 754 |
The Big Issue
bigissue.com
|
A | 100 | 0/11 | Allows all | UK |
| 755 |
Billings Gazette
billingsgazette.com
|
A | 100 | 0/11 | Allows all | US |
| 756 |
BioBioChile
biobiochile.cl
|
A | 100 | 0/11 | Allows all | LatAm |
| 757 |
Bismarck Tribune
bismarcktribune.com
|
A | 100 | 0/11 | Allows all | US |
| 758 |
Blick
blick.ch
|
A | 100 | 0/11 | Allows all | EU |
| 759 |
Borkena
borkena.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 760 |
Business Recorder
brecorder.com
|
A | 100 | 0/11 | Allows all | Asia |
| 761 |
Breitbart News
breitbart.com
|
A | 100 | 0/11 | Allows all | US |
| 762 |
Business Standard
business-standard.com
|
A | 100 | 0/11 | Allows all | Asia |
| 763 |
BusinessDay (Nigeria)
businessday.ng
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 764 |
BusinessKorea
businesskorea.co.kr
|
A | 100 | 0/11 | Allows all | Asia |
| 765 |
Business Post
businesspostbd.com
|
A | 100 | 0/11 | Allows all | Asia |
| 766 |
Calcalist (CTech)
calcalistech.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 767 |
Capital FM (Kenya)
capitalfm.co.ke
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 768 |
Caracol Radio
caracol.com.co
|
A | 100 | 0/11 | Allows all | LatAm |
| 769 |
Casper Star-Tribune
casperstartribune.net
|
A | 100 | 0/11 | Allows all | US |
| 770 |
Censor.NET
censor.net
|
A | 100 | 0/11 | Allows all | EU |
| 771 |
CFO Dive
cfodive.com
|
A | 100 | 0/11 | Allows all | US |
| 772 |
CGTN
cgtn.com
|
A | 100 | 0/11 | Allows all | Asia |
| 773 |
CNA (Channel NewsAsia)
channelnewsasia.com
|
A | 100 | 0/11 | Allows all | Asia |
| 774 |
Channels Television
channelstv.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 775 |
Chatelaine
chatelaine.com
|
A | 100 | 0/11 | Allows all | CA |
| 776 |
Chemistry World
chemistryworld.com
|
A | 100 | 0/11 | Allows all | US |
| 777 |
Chicago Tribune
chicagotribune.com
|
A | 100 | 0/11 | Allows all | US |
| 778 |
Chief Executive
chiefexecutive.net
|
A | 100 | 0/11 | Allows all | US |
| 779 |
China Daily
chinadaily.com.cn
|
A | 100 | 0/11 | Allows all | Asia |
| 780 |
Cinco Días
cincodias.elpais.com
|
A | 100 | 0/11 | Allows all | EU |
| 781 |
CIPER Chile
ciperchile.cl
|
A | 100 | 0/11 | Allows all | LatAm |
| 782 |
Citi Newsroom
citinewsroom.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 783 |
The Citizen (SA)
citizen.co.za
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 784 |
Citizen Digital (Kenya)
citizen.digital
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 785 |
City A.M.
cityam.com
|
A | 100 | 0/11 | Allows all | UK |
| 786 |
Columbia Journalism Review
cjr.org
|
A | 100 | 0/11 | Allows all | US |
| 787 |
Clarín
clarin.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 788 |
CNN Brasil
cnnbrasil.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 789 |
Colombo Telegraph
colombotelegraph.com
|
A | 100 | 0/11 | Allows all | Asia |
| 790 |
ComicBook.com
comicbook.com
|
A | 100 | 0/11 | Allows all | US |
| 791 |
Connacht Tribune
connachttribune.ie
|
A | 100 | 0/11 | Allows all | Ireland |
| 792 |
Construction Dive
constructiondive.com
|
A | 100 | 0/11 | Allows all | US |
| 793 |
Radio Cooperativa
cooperativa.cl
|
A | 100 | 0/11 | Allows all | LatAm |
| 794 |
Correio Braziliense
correiobraziliense.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 795 |
Corriere dello Sport
corrieredellosport.it
|
A | 100 | 0/11 | Allows all | EU |
| 796 |
Hartford Courant
courant.com
|
A | 100 | 0/11 | Allows all | US |
| 797 |
Courthouse News Service
courthousenews.com
|
A | 100 | 0/11 | Allows all | US |
| 798 |
El Cronista
cronista.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 799 |
The Daily Beast
dailybeast.com
|
A | 100 | 0/11 | Allows all | US |
| 800 |
Daily Herald (Chicago suburbs)
dailyherald.com
|
A | 100 | 0/11 | Allows all | US |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!