Business Robots.txt Leaderboard
How the web's top corporate & business websites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
1%
of 201 ranked sites block GPTBot in robots.txt
201
sites graded, last checked Nov 11, 2026
99/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 197 sites
- B
- 2 sites
- C
- 0 sites
- D
- 2 sites
- F
- 0 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 1 |
Carlsberg Group
carlsberggroup.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 2 |
Kerry Group
kerry.com
|
D | 45 | 6/11 |
GPTBot
ClaudeBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
Ireland |
| 3 |
SAP
sap.com
|
B | 73 | 3/11 |
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 4 |
Telstra
telstra.com.au
|
B | 73 | 3/11 |
CCBot
Bytespider
Amazonbot
|
Oceania |
| 5 |
Airbus
airbus.com
|
A | 91 | 1/11 |
Bytespider
|
EU |
| 6 |
3M
3m.com
|
A | 100 | 0/11 | Allows all | US |
| 7 |
American Airlines
aa.com
|
A | 100 | 0/11 | Allows all | US |
| 8 |
AB InBev
ab-inbev.com
|
A | 100 | 0/11 | Allows all | EU |
| 9 |
AbbVie
abbvie.com
|
A | 100 | 0/11 | Allows all | US |
| 10 |
Accenture
accenture.com
|
A | 100 | 0/11 | Allows all | Ireland |
| 11 |
ADNOC
adnoc.ae
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 12 |
Air Canada
aircanada.com
|
A | 100 | 0/11 | Allows all | CA |
| 13 |
Air New Zealand
airnewzealand.com
|
A | 100 | 0/11 | Allows all | Oceania |
| 14 |
Alibaba Group
alibabagroup.com
|
A | 100 | 0/11 | Allows all | Asia |
| 15 |
Allianz
allianz.com
|
A | 100 | 0/11 | Allows all | EU |
| 16 |
Ambev
ambev.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 17 |
América Móvil
americamovil.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 18 |
American Express
americanexpress.com
|
A | 100 | 0/11 | Allows all | US |
| 19 |
Amgen
amgen.com
|
A | 100 | 0/11 | Allows all | US |
| 20 |
ANZ
anz.com
|
A | 100 | 0/11 | Allows all | Oceania |
| 21 |
Saudi Aramco
aramco.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 22 |
ASML
asml.com
|
A | 100 | 0/11 | Allows all | EU |
| 23 |
AT&T
att.com
|
A | 100 | 0/11 | Allows all | US |
| 24 |
Aviva
aviva.com
|
A | 100 | 0/11 | Allows all | UK |
| 25 |
AXA
axa.com
|
A | 100 | 0/11 | Allows all | EU |
| 26 |
BAE Systems
baesystems.com
|
A | 100 | 0/11 | Allows all | UK |
| 27 |
Bain & Company
bain.com
|
A | 100 | 0/11 | Allows all | US |
| 28 |
Baker McKenzie
bakermckenzie.com
|
A | 100 | 0/11 | Allows all | US |
| 29 |
Bank of America
bankofamerica.com
|
A | 100 | 0/11 | Allows all | US |
| 30 |
BASF
basf.com
|
A | 100 | 0/11 | Allows all | EU |
| 31 |
Bayer
bayer.com
|
A | 100 | 0/11 | Allows all | EU |
| 32 |
BBVA
bbva.com
|
A | 100 | 0/11 | Allows all | EU |
| 33 |
Boston Consulting Group
bcg.com
|
A | 100 | 0/11 | Allows all | US |
| 34 |
BHP
bhp.com
|
A | 100 | 0/11 | Allows all | Oceania |
| 35 |
BlackRock
blackrock.com
|
A | 100 | 0/11 | Allows all | US |
| 36 |
Bristol Myers Squibb
bms.com
|
A | 100 | 0/11 | Allows all | US |
| 37 |
BMW Group
bmwgroup.com
|
A | 100 | 0/11 | Allows all | EU |
| 38 |
BNP Paribas
bnpparibas.com
|
A | 100 | 0/11 | Allows all | EU |
| 39 |
Boeing
boeing.com
|
A | 100 | 0/11 | Allows all | US |
| 40 |
Booz Allen Hamilton
boozallen.com
|
A | 100 | 0/11 | Allows all | US |
| 41 |
Bosch
bosch.com
|
A | 100 | 0/11 | Allows all | EU |
| 42 |
BP
bp.com
|
A | 100 | 0/11 | Allows all | UK |
| 43 |
Bridgestone
bridgestone.com
|
A | 100 | 0/11 | Allows all | Asia |
| 44 |
BT Group
bt.com
|
A | 100 | 0/11 | Allows all | UK |
| 45 |
BYD
byd.com
|
A | 100 | 0/11 | Allows all | Asia |
| 46 |
Capgemini
capgemini.com
|
A | 100 | 0/11 | Allows all | EU |
| 47 |
Cargill
cargill.com
|
A | 100 | 0/11 | Allows all | US |
| 48 |
CBRE
cbre.com
|
A | 100 | 0/11 | Allows all | US |
| 49 |
Cemex
cemex.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 50 |
CGI
cgi.com
|
A | 100 | 0/11 | Allows all | CA |
All 201 graded sites from the latest weekly check, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.
Your Logs already store the data you need,
you just need a way to use it!
If you are interested in HoneyLog, you can try it with our Free Trial!