Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 351 |
Forbes
forbes.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 352 |
Frankfurter Rundschau
fr.de
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 353 |
GameSpot
gamespot.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 354 |
Glamour
glamour.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 355 |
Golf.com
golf.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 356 |
GQ
gq.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 357 |
Handelsblatt
handelsblatt.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 358 |
Harper's Magazine
harpers.org
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Global |
| 359 |
Health
health.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 360 |
InStyle
instyle.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 361 |
Investopedia
investopedia.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 362 |
Iraqi News
iraqinews.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 363 |
Cesky rozhlas
irozhlas.cz
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 364 |
Kjarninn
kjarninn.is
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 365 |
Kuwait Times
kuwaittimes.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 366 |
The Libya Observer
libyaobserver.ly
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 367 |
Mada Masr
madamasr.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 368 |
Marianne
marianne.net
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 369 |
Münchner Merkur
merkur.de
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 370 |
MIT Technology Review
mittechnologyreview.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Global |
| 371 |
MMA Fighting
mmafighting.com
|
D | 27 | 8/11 |
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 372 |
n-tv
n-tv.de
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 373 |
NASASpaceflight
nasaspaceflight.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 374 |
NBA.com
nba.com
|
D | 27 | 8/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Amazonbot
Applebot-Extended
|
US |
| 375 |
The New Yorker
newyorker.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 376 |
Nexo Jornal
nexojornal.com.br
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
LatAm |
| 377 |
New York Magazine
nymag.com
|
D | 27 | 8/11 |
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 378 |
New York Post
nypost.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 379 |
People
people.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 380 |
Pitchfork
pitchfork.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 381 |
Polygon
polygon.com
|
D | 27 | 8/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
US |
| 382 |
Portland Press Herald
pressherald.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 383 |
PV Magazine
pv-magazine.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 384 |
Rabble.ca
rabble.ca
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 385 |
Rai News
rainews.it
|
D | 27 | 8/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
|
EU |
| 386 |
Rest of World
restofworld.org
|
D | 27 | 8/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
|
Global |
| 387 |
RUV
ruv.is
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 388 |
SB Nation
sbnation.com
|
D | 27 | 8/11 |
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 389 |
SBS News
sbs.com.au
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Oceania |
| 390 |
Science News
sciencenews.org
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 391 |
Scoop
scoop.co.nz
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Oceania |
| 392 |
Screen Rant
screenrant.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 393 |
Self
self.com
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 394 |
Slate
slate.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 395 |
Der Spiegel
spiegel.de
|
D | 27 | 8/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 396 |
Spin
spin.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 397 |
The Sporting News
sportingnews.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 398 |
STAT News
statnews.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 399 |
The Sunday Times (Sri Lanka)
sundaytimes.lk
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 400 |
Svenska Dagbladet
svd.se
|
D | 27 | 8/11 |
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!