Free public ranking

Business Robots.txt Leaderboard

How the web's top corporate & business websites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.

1%

of 201 ranked sites block GPTBot in robots.txt

201

sites graded, last checked Nov 11, 2026

99/100

average openness score (higher = more open to AI)

Share of sites blocking each AI crawler

Click a bot to filter the ranking to sites that block it.

Grade distribution

A = wide open to AI, F = walled off.

A
197 sites
B
2 sites
C
0 sites
D
2 sites
F
0 sites
# AI crawlers Region
151
RBC Royal Bank
rbc.com
A 100 0/11 Allows all CA
152
Reckitt
reckitt.com
A 100 0/11 Allows all UK
153
Reliance Industries
ril.com
A 100 0/11 Allows all Asia
154
Rio Tinto
riotinto.com
A 100 0/11 Allows all UK
155
Roche
roche.com
A 100 0/11 Allows all EU
156
Rolls-Royce
rolls-royce.com
A 100 0/11 Allows all UK
157
RTX
rtx.com
A 100 0/11 Allows all US
158
Ryanair
ryanair.com
A 100 0/11 Allows all Ireland
159
SABIC
sabic.com
A 100 0/11 Allows all MENA-Africa
160
Safaricom
safaricom.co.ke
A 100 0/11 Allows all MENA-Africa
161
Samsung
samsung.com
A 100 0/11 Allows all Asia
162
Sanofi
sanofi.com
A 100 0/11 Allows all EU
163
Santander
santander.com
A 100 0/11 Allows all EU
164
Sasol
sasol.com
A 100 0/11 Allows all MENA-Africa
165
Scotiabank
scotiabank.com
A 100 0/11 Allows all CA
166
Schneider Electric
se.com
A 100 0/11 Allows all EU
167
Shell
shell.com
A 100 0/11 Allows all UK
168
Siemens
siemens.com
A 100 0/11 Allows all EU
169
Singapore Airlines
singaporeair.com
A 100 0/11 Allows all Asia
170
Singtel
singtel.com
A 100 0/11 Allows all Asia
171
SK hynix
skhynix.com
A 100 0/11 Allows all Asia
172
Sony
sony.com
A 100 0/11 Allows all Asia
173
Standard Bank
standardbank.com
A 100 0/11 Allows all MENA-Africa
174
Starbucks
starbucks.com
A 100 0/11 Allows all US
175
State Farm
statefarm.com
A 100 0/11 Allows all US
176
Stellantis
stellantis.com
A 100 0/11 Allows all EU
177
Swiss Re
swissre.com
A 100 0/11 Allows all EU
178
T-Mobile
t-mobile.com
A 100 0/11 Allows all US
179
Takeda Pharmaceutical
takeda.com
A 100 0/11 Allows all Asia
180
Tata Group
tata.com
A 100 0/11 Allows all Asia
181
Tata Consultancy Services
tcs.com
A 100 0/11 Allows all Asia
182
TD Bank Group
td.com
A 100 0/11 Allows all CA
183
Telefónica
telefonica.com
A 100 0/11 Allows all EU
184
Deutsche Telekom
telekom.com
A 100 0/11 Allows all EU
185
Teva Pharmaceuticals
tevapharm.com
A 100 0/11 Allows all MENA-Africa
186
Heineken
theheinekencompany.com
A 100 0/11 Allows all EU
187
TotalEnergies
totalenergies.com
A 100 0/11 Allows all EU
188
TSMC
tsmc.com
A 100 0/11 Allows all Asia
189
Unilever
unilever.com
A 100 0/11 Allows all UK
190
UnitedHealth Group
unitedhealthgroup.com
A 100 0/11 Allows all US
191
UPS
ups.com
A 100 0/11 Allows all US
192
Vale
vale.com
A 100 0/11 Allows all LatAm
193
Verizon
verizon.com
A 100 0/11 Allows all US
194
Visa
visa.com
A 100 0/11 Allows all US
195
Vodafone
vodafone.com
A 100 0/11 Allows all UK
196
Volkswagen Group
volkswagen-group.com
A 100 0/11 Allows all EU
197
Volvo Group
volvogroup.com
A 100 0/11 Allows all EU
198
Wells Fargo
wellsfargo.com
A 100 0/11 Allows all US
199
Westpac
westpac.com.au
A 100 0/11 Allows all Oceania
200
Wipro
wipro.com
A 100 0/11 Allows all Asia
Showing 151 to 200 of 201 results
Download full leaderboard (CSV)

All 201 graded sites from the latest weekly check, regardless of the filters above.

Methodology

For each site we fetch its public robots.txt and grade it against 11 of the major AI crawlers, including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site “blocks” a bot when its robots.txt has a Disallow rule that applies to that crawler’s user-agent token. The openness score is the share of graded bots that are allowed, rolled up into an A–F grade.

This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.

The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.

Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.

14-day Free Trial

Your Logs already store the data you need,
you just need a way to use it!

If you are interested in HoneyLog, you can try it with our Free Trial!

No credit card required