Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 651 |
La Patilla
lapatilla.com
|
B | 82 | 2/11 |
GPTBot
ClaudeBot
|
LatAm |
| 652 |
Law360
law360.com
|
B | 82 | 2/11 |
Google-Extended
Bytespider
|
US |
| 653 |
Newsday
newsday.com
|
B | 82 | 2/11 |
GPTBot
CCBot
|
US |
| 654 |
New Statesman
newstatesman.com
|
B | 82 | 2/11 |
GPTBot
Meta-ExternalAgent
|
UK |
| 655 |
Prospect
prospectmagazine.co.uk
|
B | 82 | 2/11 |
GPTBot
Google-Extended
|
UK |
| 656 |
SunStar
sunstar.com.ph
|
B | 82 | 2/11 |
GPTBot
ClaudeBot
|
Asia |
| 657 |
TheJournal.ie
thejournal.ie
|
B | 82 | 2/11 |
GPTBot
ChatGPT-User
|
Ireland |
| 658 |
THISDAY
thisdaylive.com
|
B | 82 | 2/11 |
Bytespider
Amazonbot
|
MENA-Africa |
| 659 |
Vanguardia
vanguardia.com.mx
|
B | 82 | 2/11 |
Bytespider
Meta-ExternalAgent
|
LatAm |
| 660 |
Washington Examiner
washingtonexaminer.com
|
B | 82 | 2/11 |
CCBot
Bytespider
|
US |
| 661 |
WION
wionews.com
|
B | 82 | 2/11 |
OAI-SearchBot
Bytespider
|
Asia |
| 662 |
Wirtualna Polska
wp.pl
|
B | 82 | 2/11 |
GPTBot
CCBot
|
EU |
| 663 |
404 Media
404media.co
|
A | 91 | 1/11 |
GPTBot
|
US |
| 664 |
Ada Derana
adaderana.lk
|
A | 91 | 1/11 |
Bytespider
|
Asia |
| 665 |
Aktualne.cz
aktualne.cz
|
A | 91 | 1/11 |
GPTBot
|
EU |
| 666 |
Antara News
antaranews.com
|
A | 91 | 1/11 |
ClaudeBot
|
Asia |
| 667 |
Aristegui Noticias
aristeguinoticias.com
|
A | 91 | 1/11 |
CCBot
|
LatAm |
| 668 |
Bangor Daily News
bangordailynews.com
|
A | 91 | 1/11 |
Applebot-Extended
|
US |
| 669 |
bdnews24
bdnews24.com
|
A | 91 | 1/11 |
GPTBot
|
Asia |
| 670 |
The Business Times
businesstimes.com.sg
|
A | 91 | 1/11 |
Google-Extended
|
Asia |
| 671 |
Calgary Herald
calgaryherald.com
|
A | 91 | 1/11 |
Meta-ExternalAgent
|
CA |
| 672 |
CBS News
cbsnews.com
|
A | 91 | 1/11 |
GPTBot
|
US |
| 673 |
CBS Sports
cbssports.com
|
A | 91 | 1/11 |
GPTBot
|
US |
| 674 |
The Chosun Ilbo
chosun.com
|
A | 91 | 1/11 |
GPTBot
|
Asia |
| 675 |
DNA India
dnaindia.com
|
A | 91 | 1/11 |
Bytespider
|
Asia |
| 676 |
VnExpress International
e.vnexpress.net
|
A | 91 | 1/11 |
Amazonbot
|
Asia |
| 677 |
The Economic Times
economictimes.indiatimes.com
|
A | 91 | 1/11 |
Meta-ExternalAgent
|
Asia |
| 678 |
Edmonton Journal
edmontonjournal.com
|
A | 91 | 1/11 |
Meta-ExternalAgent
|
CA |
| 679 |
Entrepreneur
entrepreneur.com
|
A | 91 | 1/11 |
Meta-ExternalAgent
|
US |
| 680 |
EurekAlert!
eurekalert.org
|
A | 91 | 1/11 |
Bytespider
|
US |
| 681 |
Hindustan Times
hindustantimes.com
|
A | 91 | 1/11 |
Meta-ExternalAgent
|
Asia |
| 682 |
Hospodarske noviny
ihned.cz
|
A | 91 | 1/11 |
GPTBot
|
EU |
| 683 |
Independent Australia
independentaustralia.net
|
A | 91 | 1/11 |
Applebot-Extended
|
Oceania |
| 684 |
Philippine Daily Inquirer
inquirer.net
|
A | 91 | 1/11 |
GPTBot
|
Asia |
| 685 |
Insurance Journal
insurancejournal.com
|
A | 91 | 1/11 |
Amazonbot
|
US |
| 686 |
Kleine Zeitung
kleinezeitung.at
|
A | 91 | 1/11 |
Amazonbot
|
EU |
| 687 |
Korea Bizwire
koreaboo.com
|
A | 91 | 1/11 |
Meta-ExternalAgent
|
Asia |
| 688 |
La Voz del Interior
lavoz.com.ar
|
A | 91 | 1/11 |
ChatGPT-User
|
LatAm |
| 689 |
Mint
livemint.com
|
A | 91 | 1/11 |
Meta-ExternalAgent
|
Asia |
| 690 |
Milenio
milenio.com
|
A | 91 | 1/11 |
CCBot
|
LatAm |
| 691 |
Observador
observador.pt
|
A | 91 | 1/11 |
Bytespider
|
EU |
| 692 |
Ottawa Citizen
ottawacitizen.com
|
A | 91 | 1/11 |
Meta-ExternalAgent
|
CA |
| 693 |
Las Vegas Review-Journal
reviewjournal.com
|
A | 91 | 1/11 |
Bytespider
|
US |
| 694 |
SaltWire Network
saltwire.com
|
A | 91 | 1/11 |
Meta-ExternalAgent
|
CA |
| 695 |
Stirile ProTV
stirileprotv.ro
|
A | 91 | 1/11 |
GPTBot
|
EU |
| 696 |
The Straits Times
straitstimes.com
|
A | 91 | 1/11 |
Google-Extended
|
Asia |
| 697 |
Die Tageszeitung (taz)
taz.de
|
A | 91 | 1/11 |
GPTBot
|
EU |
| 698 |
Telex
telex.hu
|
A | 91 | 1/11 |
Meta-ExternalAgent
|
EU |
| 699 |
The Dispatch
thedispatch.com
|
A | 91 | 1/11 |
Meta-ExternalAgent
|
US |
| 700 |
The Edge Malaysia
theedgemalaysia.com
|
A | 91 | 1/11 |
GPTBot
|
Asia |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!