Free public ranking

Robots.txt Leaderboard

How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.

44%

of 1,241 ranked sites block GPTBot in robots.txt

1,241

sites graded this batch (2026-07)

62/100

average openness score (higher = more open to AI)

Share of sites blocking each AI crawler

Click a bot to filter the ranking to sites that block it.

Grade distribution

A = wide open to AI, F = walled off.

A
604 sites
B
62 sites
C
64 sites
D
215 sites
F
296 sites
# AI crawlers Region
1
Aamulehti
aamulehti.fi
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
2
Albuquerque Journal
abqjournal.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
3
Acadie Nouvelle
acadienouvelle.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
CA
4
Australian Financial Review
afr.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
Oceania
5
Asbury Park Press
app.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
6
Argus Leader
argusleader.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
7
Arkansas Democrat-Gazette
arkansasonline.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
8
The Asahi Shimbun
asahi.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
Asia
9
Nikkei Asia
asia.nikkei.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
Asia
10
The Augusta Chronicle
augustachronicle.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
11
The Arizona Republic
azcentral.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
12
BBC
bbc.co.uk
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
UK
13
BBC
bbc.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
UK
14
Bernama
bernama.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
Asia
15
BFM TV
bfmtv.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
16
Bloomberg
bloomberg.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
17
Brisbane Times
brisbanetimes.com.au
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
Oceania
18
Burlington Free Press
burlingtonfreepress.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
19
The Business Post
businesspost.ie
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
Ireland
20
BuzzFeed News
buzzfeednews.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
21
Corpus Christi Caller-Times
caller.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
22
Cape Cod Times
capecodtimes.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
23
The Cincinnati Enquirer
cincinnati.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
24
The Topeka Capital-Journal
cjonline.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
25
The Clarion-Ledger
clarionledger.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
26
CNBC
cnbc.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
27
Fort Collins Coloradoan
coloradoan.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
28
The Commercial Appeal (Memphis)
commercialappeal.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
29
The Courier-Journal (Louisville)
courier-journal.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
30
Courier-Post (Cherry Hill)
courierpostonline.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
31
Daily Mail
dailymail.co.uk
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
UK
32
Los Angeles Daily News
dailynews.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
33
Daily Sun
dailysun.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
Asia
34
The News Journal (Delaware)
delawareonline.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
35
Delaware County Daily Times
delcotimes.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
36
Democrat and Chronicle (Rochester)
democratandchronicle.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
37
Denik
denik.cz
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
38
The Denver Post
denverpost.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
39
The Des Moines Register
desmoinesregister.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
40
The Detroit News
detroitnews.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
41
The Columbus Dispatch
dispatch.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
42
DR (Danmarks Radio)
dr.dk
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
43
Duluth News Tribune
duluthnewstribune.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
44
Deutsche Welle
dw.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
45
East Bay Times
eastbaytimes.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
46
El Confidencial
elconfidencial.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
47
El Paso Times
elpasotimes.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
48
El Universal (Venezuela)
eluniversal.com
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
LatAm
49
Yonhap (English)
en.yna.co.kr
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
Asia
50
Expresso
expresso.pt
 F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
Showing 1 to 50 of 1241 results

Methodology

For each site we fetch its public robots.txt and grade it against 11 of the major AI crawlers , including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site “blocks” a bot when its robots.txt has a Disallow rule that applies to that crawler’s user-agent token. The openness score is the share of graded bots that are allowed, rolled up into an A–F grade.

This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall , a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.

The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine) , we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.

Latest batch: 2026-07. Reads stored scan data only , no live requests are made when you load this page.

14-days Free Trial

Take back control of your Traffic.
Try HoneyLog for Free!

If you are interested in HoneyLog, you can try it with our Free Trial!

Free Trial Free Demo
No credit card required