Free public ranking

Robots.txt Leaderboard

How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.

42%

of 1,236 ranked sites block GPTBot in robots.txt

1,236

sites graded this batch (2026-10)

63/100

average openness score (higher = more open to AI)

Share of sites blocking each AI crawler

Click a bot to filter the ranking to sites that block it.

Grade distribution

A = wide open to AI, F = walled off.

A
628 sites
B
64 sites
C
52 sites
D
189 sites
F
303 sites
# AI crawlers Region
1
Aamulehti
aamulehti.fi
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
2
Acadie Nouvelle
acadienouvelle.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
CA
3
Australian Financial Review
afr.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
Oceania
4
Asbury Park Press
app.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
5
Argus Leader
argusleader.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
6
Arkansas Democrat-Gazette
arkansasonline.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
7
The Asahi Shimbun
asahi.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
Asia
8
Nikkei Asia
asia.nikkei.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
Asia
9
The Augusta Chronicle
augustachronicle.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
10
The Arizona Republic
azcentral.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
11
BBC
bbc.co.uk
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
UK
12
BBC
bbc.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
UK
13
BFM TV
bfmtv.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
14
Brisbane Times
brisbanetimes.com.au
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
Oceania
15
Burlington Free Press
burlingtonfreepress.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
16
BuzzFeed News
buzzfeednews.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
17
Corpus Christi Caller-Times
caller.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
18
Cape Cod Times
capecodtimes.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
19
The Cincinnati Enquirer
cincinnati.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
20
The Topeka Capital-Journal
cjonline.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
21
The Clarion-Ledger
clarionledger.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
22
CNBC
cnbc.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
23
Fort Collins Coloradoan
coloradoan.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
24
The Commercial Appeal (Memphis)
commercialappeal.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
25
The Courier-Journal (Louisville)
courier-journal.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
26
Courier-Post (Cherry Hill)
courierpostonline.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
27
Daily Mail
dailymail.co.uk
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
UK
28
Los Angeles Daily News
dailynews.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
29
The News Journal (Delaware)
delawareonline.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
30
Delaware County Daily Times
delcotimes.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
31
Democrat and Chronicle (Rochester)
democratandchronicle.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
32
Denik
denik.cz
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
33
The Denver Post
denverpost.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
34
The Des Moines Register
desmoinesregister.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
35
The Detroit News
detroitnews.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
36
The Columbus Dispatch
dispatch.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
37
DR (Danmarks Radio)
dr.dk
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
38
Duluth News Tribune
duluthnewstribune.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
39
Deutsche Welle
dw.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
40
East Bay Times
eastbaytimes.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
41
El Confidencial
elconfidencial.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
42
El Paso Times
elpasotimes.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
43
El Universal (Venezuela)
eluniversal.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
LatAm
44
El Universal
eluniversal.com.mx
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
LatAm
45
Yonhap (English)
en.yna.co.kr
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
Asia
46
Expresso
expresso.pt
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
47
FAZ
faz.net
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
48
Het Financieele Dagblad
fd.nl
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
49
France Info
francetvinfo.fr
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
50
Detroit Free Press
freep.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
Showing 1 to 50 of 1236 results

Methodology

For each site we fetch its public robots.txt and grade it against 11 of the major AI crawlers , including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site “blocks” a bot when its robots.txt has a Disallow rule that applies to that crawler’s user-agent token. The openness score is the share of graded bots that are allowed, rolled up into an A–F grade.

This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall , a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.

The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine) , we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.

Latest batch: 2026-10. Reads stored scan data only , no live requests are made when you load this page.

14-days Free Trial

Take back control of your Traffic.
Try HoneyLog for Free!

If you are interested in HoneyLog, you can try it with our Free Trial!

Free Trial Free Demo
No credit card required