Robots.txt Leaderboard
How the web's top news & publisher sites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
45%
of 1,240 ranked sites block GPTBot in robots.txt
1,240
sites graded this batch (2026-11)
60/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 578 sites
- B
- 54 sites
- C
- 73 sites
- D
- 216 sites
- F
- 319 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 151 |
Sydney Morning Herald
smh.com.au
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Oceania |
| 152 |
Honolulu Star-Advertiser
staradvertiser.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 153 |
Statesman Journal
statesmanjournal.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 154 |
Stuff
stuff.co.nz
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Oceania |
| 155 |
Tallahassee Democrat
tallahassee.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 156 |
Treasure Coast Newspapers
tcpalm.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 157 |
MIT Technology Review
technologyreview.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 158 |
Telegram & Gazette (Worcester)
telegram.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 159 |
Tempo
tempo.co
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 160 |
The Tennessean
tennessean.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 161 |
The Review (Alliance)
the-review.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 162 |
The Age
theage.com.au
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Oceania |
| 163 |
The Canadian Press
thecanadianpress.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 164 |
The Conversation
theconversation.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Oceania |
| 165 |
The Hindu BusinessLine
thehindubusinessline.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 166 |
The Peterborough Examiner
thepeterboroughexaminer.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 167 |
The Post (Wellington)
thepost.co.nz
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Oceania |
| 168 |
Waterloo Region Record
therecord.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 169 |
The Hamilton Spectator
thespec.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 170 |
Toronto Star
thestar.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 171 |
TODAY
todayonline.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 172 |
Turun Sanomat
ts.fi
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 173 |
TV 2 Denmark
tv2.dk
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 174 |
Pioneer Press
twincities.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 175 |
New Hampshire Union Leader
unionleader.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 176 |
USA Today
usatoday.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 177 |
U.S. News & World Report
usnews.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 178 |
Waikato Times
waikatotimes.co.nz
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Oceania |
| 179 |
WAtoday
watoday.com.au
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Oceania |
| 180 |
Yonhap News Agency
yna.co.kr
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 181 |
The Japan News (Yomiuri)
yomiuri.co.jp
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 182 |
ABC
abc.es
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 183 |
Algemeen Dagblad
ad.nl
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 184 |
The Advertiser (Adelaide)
adelaidenow.com.au
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
Oceania |
| 185 |
Alabama Media Group (AL.com)
al.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
US |
| 186 |
The Bakersfield Californian
bakersfield.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 187 |
Belfast Telegraph
belfasttelegraph.co.uk
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
UK |
| 188 |
Billboard
billboard.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 189 |
The Buffalo News
buffalonews.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 190 |
Business Day (BusinessLIVE)
businesslive.co.za
|
F | 9 | 10/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 191 |
The Business Post
businesspost.ie
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Ireland |
| 192 |
The Canberra Times
canberratimes.com.au
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
Oceania |
| 193 |
Ceske noviny (CTK)
ceskenoviny.cz
|
F | 9 | 10/11 |
GPTBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 194 |
Channel 4 News
channel4.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 195 |
The Plain Dealer
cleveland.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
US |
| 196 |
CNET
cnet.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 197 |
The Courier-Mail
couriermail.com.au
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
Oceania |
| 198 |
The Daily Telegraph (Sydney)
dailytelegraph.com.au
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
Oceania |
| 199 |
Dayton Daily News
daytondailynews.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
| 200 |
Deadline
deadline.com
|
F | 9 | 10/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
US |
All 1,240 graded sites from the 2026-11 batch, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Latest batch: 2026-11. Reads stored scan data only: no live requests are made when you load this page.
Take back control of your Traffic.
Try
HoneyLog for Free!
Try HoneyLog for Free!
If you are interested in HoneyLog, you can try it with our Free Trial!