Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 551 |
Dazed
dazeddigital.com
|
C | 55 | 5/11 |
ClaudeBot
Google-Extended
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 552 |
Dennik N
dennikn.sk
|
C | 55 | 5/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
|
EU |
| 553 |
El Heraldo
elheraldo.co
|
C | 55 | 5/11 |
GPTBot
ClaudeBot
PerplexityBot
CCBot
Bytespider
|
LatAm |
| 554 |
Elle
elle.com
|
C | 55 | 5/11 |
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
US |
| 555 |
El País
elpais.com
|
C | 55 | 5/11 |
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
|
EU |
| 556 |
ESPN
espn.com
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
Google-Extended
CCBot
Bytespider
|
US |
| 557 |
Esquire
esquire.com
|
C | 55 | 5/11 |
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
US |
| 558 |
San Antonio Express-News
expressnews.com
|
C | 55 | 5/11 |
ClaudeBot
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
US |
| 559 |
The Financial Express
financialexpress.com
|
C | 55 | 5/11 |
ClaudeBot
PerplexityBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 560 |
Graphic Online (Ghana)
graphic.com.gh
|
C | 55 | 5/11 |
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
MENA-Africa |
| 561 |
Harper's Bazaar
harpersbazaar.com
|
C | 55 | 5/11 |
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
US |
| 562 |
Houston Chronicle
houstonchronicle.com
|
C | 55 | 5/11 |
ClaudeBot
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
US |
| 563 |
The Indian Express
indianexpress.com
|
C | 55 | 5/11 |
ClaudeBot
PerplexityBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 564 |
IOL (Independent Online)
iol.co.za
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
ClaudeBot
Bytespider
Amazonbot
|
MENA-Africa |
| 565 |
Le Journal de Montréal
journaldemontreal.com
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
|
CA |
| 566 |
Le Journal de Québec
journaldequebec.com
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
|
CA |
| 567 |
LBC
lbc.co.uk
|
C | 55 | 5/11 |
ClaudeBot
Google-Extended
PerplexityBot
Bytespider
Applebot-Extended
|
UK |
| 568 |
Le Parisien
leparisien.fr
|
C | 55 | 5/11 |
GPTBot
ClaudeBot
PerplexityBot
CCBot
Amazonbot
|
EU |
| 569 |
Mamamia
mamamia.com.au
|
C | 55 | 5/11 |
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Oceania |
| 570 |
Men's Health
menshealth.com
|
C | 55 | 5/11 |
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
US |
| 571 |
Popular Mechanics
popularmechanics.com
|
C | 55 | 5/11 |
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
US |
| 572 |
Science
science.org
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
Google-Extended
CCBot
Applebot-Extended
|
US |
| 573 |
Science
sciencemag.org
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
Google-Extended
CCBot
Applebot-Extended
|
US |
| 574 |
Seeking Alpha
seekingalpha.com
|
C | 55 | 5/11 |
ChatGPT-User
ClaudeBot
CCBot
Bytespider
Meta-ExternalAgent
|
US |
| 575 |
SFGATE
sfgate.com
|
C | 55 | 5/11 |
ClaudeBot
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
US |
| 576 |
The Salt Lake Tribune
sltrib.com
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
Google-Extended
PerplexityBot
CCBot
|
US |
| 577 |
Stern
stern.de
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
Google-Extended
CCBot
Applebot-Extended
|
EU |
| 578 |
Süddeutsche Zeitung
sueddeutsche.de
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
|
EU |
| 579 |
Town & Country
townandcountrymag.com
|
C | 55 | 5/11 |
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
US |
| 580 |
Die Welt
welt.de
|
C | 55 | 5/11 |
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 581 |
Women's Health
womenshealthmag.com
|
C | 55 | 5/11 |
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
US |
| 582 |
Android Central
androidcentral.com
|
C | 64 | 4/11 |
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 583 |
Borsen
borsen.dk
|
C | 64 | 4/11 |
GPTBot
ChatGPT-User
Google-Extended
CCBot
|
EU |
| 584 |
Daily Hive
dailyhive.com
|
C | 64 | 4/11 |
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
CA |
| 585 |
El Colombiano
elcolombiano.com
|
C | 64 | 4/11 |
ClaudeBot
CCBot
Bytespider
Applebot-Extended
|
LatAm |
| 586 |
The Motley Fool
fool.com
|
C | 64 | 4/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
|
US |
| 587 |
Genetic Engineering & Biotechnology News
genengnews.com
|
C | 64 | 4/11 |
GPTBot
ChatGPT-User
Google-Extended
CCBot
|
US |
| 588 |
Il Resto del Carlino
ilrestodelcarlino.it
|
C | 64 | 4/11 |
ClaudeBot
CCBot
Bytespider
Applebot-Extended
|
EU |
| 589 |
Interfax-Ukraine
interfax.com.ua
|
C | 64 | 4/11 |
ClaudeBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
EU |
| 590 |
Khaleej Times
khaleejtimes.com
|
C | 64 | 4/11 |
ClaudeBot
Google-Extended
Amazonbot
Applebot-Extended
|
MENA-Africa |
| 591 |
Kiplinger
kiplinger.com
|
C | 64 | 4/11 |
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 592 |
Korea JoongAng Daily
koreajoongangdaily.joins.com
|
C | 64 | 4/11 |
GPTBot
ClaudeBot
Bytespider
Amazonbot
|
Asia |
| 593 |
La Nazione
lanazione.it
|
C | 64 | 4/11 |
ClaudeBot
CCBot
Bytespider
Applebot-Extended
|
EU |
| 594 |
La Vanguardia
lavanguardia.com
|
C | 64 | 4/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
|
EU |
| 595 |
Live Science
livescience.com
|
C | 64 | 4/11 |
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 596 |
Marie Claire
marieclaire.com
|
C | 64 | 4/11 |
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 597 |
Mundo Deportivo
mundodeportivo.com
|
C | 64 | 4/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
|
EU |
| 598 |
PBS
pbs.org
|
C | 64 | 4/11 |
GPTBot
PerplexityBot
CCBot
Bytespider
|
US |
| 599 |
RTÉ
rte.ie
|
C | 64 | 4/11 |
GPTBot
ChatGPT-User
Google-Extended
CCBot
|
EU |
| 600 |
Space.com
space.com
|
C | 64 | 4/11 |
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!