Business Robots.txt Leaderboard
How the web's top corporate & business websites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
1%
of 201 ranked sites block GPTBot in robots.txt
201
sites graded, last checked Nov 11, 2026
99/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 197 sites
- B
- 2 sites
- C
- 0 sites
- D
- 2 sites
- F
- 0 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 151 |
RBC Royal Bank
rbc.com
|
A | 100 | 0/11 | Allows all | CA |
| 152 |
Reckitt
reckitt.com
|
A | 100 | 0/11 | Allows all | UK |
| 153 |
Reliance Industries
ril.com
|
A | 100 | 0/11 | Allows all | Asia |
| 154 |
Rio Tinto
riotinto.com
|
A | 100 | 0/11 | Allows all | UK |
| 155 |
Roche
roche.com
|
A | 100 | 0/11 | Allows all | EU |
| 156 |
Rolls-Royce
rolls-royce.com
|
A | 100 | 0/11 | Allows all | UK |
| 157 |
RTX
rtx.com
|
A | 100 | 0/11 | Allows all | US |
| 158 |
Ryanair
ryanair.com
|
A | 100 | 0/11 | Allows all | Ireland |
| 159 |
SABIC
sabic.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 160 |
Safaricom
safaricom.co.ke
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 161 |
Samsung
samsung.com
|
A | 100 | 0/11 | Allows all | Asia |
| 162 |
Sanofi
sanofi.com
|
A | 100 | 0/11 | Allows all | EU |
| 163 |
Santander
santander.com
|
A | 100 | 0/11 | Allows all | EU |
| 164 |
Sasol
sasol.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 165 |
Scotiabank
scotiabank.com
|
A | 100 | 0/11 | Allows all | CA |
| 166 |
Schneider Electric
se.com
|
A | 100 | 0/11 | Allows all | EU |
| 167 |
Shell
shell.com
|
A | 100 | 0/11 | Allows all | UK |
| 168 |
Siemens
siemens.com
|
A | 100 | 0/11 | Allows all | EU |
| 169 |
Singapore Airlines
singaporeair.com
|
A | 100 | 0/11 | Allows all | Asia |
| 170 |
Singtel
singtel.com
|
A | 100 | 0/11 | Allows all | Asia |
| 171 |
SK hynix
skhynix.com
|
A | 100 | 0/11 | Allows all | Asia |
| 172 |
Sony
sony.com
|
A | 100 | 0/11 | Allows all | Asia |
| 173 |
Standard Bank
standardbank.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 174 |
Starbucks
starbucks.com
|
A | 100 | 0/11 | Allows all | US |
| 175 |
State Farm
statefarm.com
|
A | 100 | 0/11 | Allows all | US |
| 176 |
Stellantis
stellantis.com
|
A | 100 | 0/11 | Allows all | EU |
| 177 |
Swiss Re
swissre.com
|
A | 100 | 0/11 | Allows all | EU |
| 178 |
T-Mobile
t-mobile.com
|
A | 100 | 0/11 | Allows all | US |
| 179 |
Takeda Pharmaceutical
takeda.com
|
A | 100 | 0/11 | Allows all | Asia |
| 180 |
Tata Group
tata.com
|
A | 100 | 0/11 | Allows all | Asia |
| 181 |
Tata Consultancy Services
tcs.com
|
A | 100 | 0/11 | Allows all | Asia |
| 182 |
TD Bank Group
td.com
|
A | 100 | 0/11 | Allows all | CA |
| 183 |
Telefónica
telefonica.com
|
A | 100 | 0/11 | Allows all | EU |
| 184 |
Deutsche Telekom
telekom.com
|
A | 100 | 0/11 | Allows all | EU |
| 185 |
Teva Pharmaceuticals
tevapharm.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 186 |
Heineken
theheinekencompany.com
|
A | 100 | 0/11 | Allows all | EU |
| 187 |
TotalEnergies
totalenergies.com
|
A | 100 | 0/11 | Allows all | EU |
| 188 |
TSMC
tsmc.com
|
A | 100 | 0/11 | Allows all | Asia |
| 189 |
Unilever
unilever.com
|
A | 100 | 0/11 | Allows all | UK |
| 190 |
UnitedHealth Group
unitedhealthgroup.com
|
A | 100 | 0/11 | Allows all | US |
| 191 |
UPS
ups.com
|
A | 100 | 0/11 | Allows all | US |
| 192 |
Vale
vale.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 193 |
Verizon
verizon.com
|
A | 100 | 0/11 | Allows all | US |
| 194 |
Visa
visa.com
|
A | 100 | 0/11 | Allows all | US |
| 195 |
Vodafone
vodafone.com
|
A | 100 | 0/11 | Allows all | UK |
| 196 |
Volkswagen Group
volkswagen-group.com
|
A | 100 | 0/11 | Allows all | EU |
| 197 |
Volvo Group
volvogroup.com
|
A | 100 | 0/11 | Allows all | EU |
| 198 |
Wells Fargo
wellsfargo.com
|
A | 100 | 0/11 | Allows all | US |
| 199 |
Westpac
westpac.com.au
|
A | 100 | 0/11 | Allows all | Oceania |
| 200 |
Wipro
wipro.com
|
A | 100 | 0/11 | Allows all | Asia |
All 201 graded sites from the latest weekly check, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.
Your Logs already store the data you need,
you just need a way to use it!
If you are interested in HoneyLog, you can try it with our Free Trial!