Business Robots.txt Leaderboard
How the web's top corporate & business websites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
1%
of 201 ranked sites block GPTBot in robots.txt
201
sites graded, last checked Nov 11, 2026
99/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 197 sites
- B
- 2 sites
- C
- 0 sites
- D
- 2 sites
- F
- 0 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 101 |
Itaú Unibanco
itau.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 102 |
Jardine Matheson
jardines.com
|
A | 100 | 0/11 | Allows all | Asia |
| 103 |
Johnson & Johnson
jnj.com
|
A | 100 | 0/11 | Allows all | US |
| 104 |
JPMorgan Chase
jpmorganchase.com
|
A | 100 | 0/11 | Allows all | US |
| 105 |
Kirkland & Ellis
kirkland.com
|
A | 100 | 0/11 | Allows all | US |
| 106 |
KPMG
kpmg.com
|
A | 100 | 0/11 | Allows all | UK |
| 107 |
LATAM Airlines
latamairlines.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 108 |
Eli Lilly
lilly.com
|
A | 100 | 0/11 | Allows all | US |
| 109 |
Lloyds Banking Group
lloydsbankinggroup.com
|
A | 100 | 0/11 | Allows all | UK |
| 110 |
Lockheed Martin
lockheedmartin.com
|
A | 100 | 0/11 | Allows all | US |
| 111 |
L'Oréal
loreal.com
|
A | 100 | 0/11 | Allows all | EU |
| 112 |
Lufthansa
lufthansa.com
|
A | 100 | 0/11 | Allows all | EU |
| 113 |
LVMH
lvmh.com
|
A | 100 | 0/11 | Allows all | EU |
| 114 |
Latham & Watkins
lw.com
|
A | 100 | 0/11 | Allows all | US |
| 115 |
Macquarie Group
macquarie.com
|
A | 100 | 0/11 | Allows all | Oceania |
| 116 |
Maersk
maersk.com
|
A | 100 | 0/11 | Allows all | EU |
| 117 |
Magna International
magna.com
|
A | 100 | 0/11 | Allows all | CA |
| 118 |
Manulife
manulife.com
|
A | 100 | 0/11 | Allows all | CA |
| 119 |
Marriott
marriott.com
|
A | 100 | 0/11 | Allows all | US |
| 120 |
Mastercard
mastercard.com
|
A | 100 | 0/11 | Allows all | US |
| 121 |
McDonald's
mcdonalds.com
|
A | 100 | 0/11 | Allows all | US |
| 122 |
Mercedes-Benz
mercedes-benz.com
|
A | 100 | 0/11 | Allows all | EU |
| 123 |
Merck & Co.
merck.com
|
A | 100 | 0/11 | Allows all | US |
| 124 |
MetLife
metlife.com
|
A | 100 | 0/11 | Allows all | US |
| 125 |
Michelin
michelin.com
|
A | 100 | 0/11 | Allows all | EU |
| 126 |
Microsoft
microsoft.com
|
A | 100 | 0/11 | Allows all | US |
| 127 |
Mondelez International
mondelezinternational.com
|
A | 100 | 0/11 | Allows all | US |
| 128 |
MTN Group
mtn.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 129 |
Munich Re
munichre.com
|
A | 100 | 0/11 | Allows all | EU |
| 130 |
National Grid
nationalgrid.com
|
A | 100 | 0/11 | Allows all | UK |
| 131 |
NatWest Group
natwestgroup.com
|
A | 100 | 0/11 | Allows all | UK |
| 132 |
Nestlé
nestle.com
|
A | 100 | 0/11 | Allows all | EU |
| 133 |
NextEra Energy
nexteraenergy.com
|
A | 100 | 0/11 | Allows all | US |
| 134 |
Nissan
nissan-global.com
|
A | 100 | 0/11 | Allows all | Asia |
| 135 |
Nokia
nokia.com
|
A | 100 | 0/11 | Allows all | EU |
| 136 |
Novartis
novartis.com
|
A | 100 | 0/11 | Allows all | EU |
| 137 |
Novo Nordisk
novonordisk.com
|
A | 100 | 0/11 | Allows all | EU |
| 138 |
NTT DATA
nttdata.com
|
A | 100 | 0/11 | Allows all | Asia |
| 139 |
NVIDIA
nvidia.com
|
A | 100 | 0/11 | Allows all | US |
| 140 |
Oracle
oracle.com
|
A | 100 | 0/11 | Allows all | US |
| 141 |
Orange
orange.com
|
A | 100 | 0/11 | Allows all | EU |
| 142 |
Panasonic
panasonic.com
|
A | 100 | 0/11 | Allows all | Asia |
| 143 |
PepsiCo
pepsico.com
|
A | 100 | 0/11 | Allows all | US |
| 144 |
Petrobras
petrobras.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 145 |
Pfizer
pfizer.com
|
A | 100 | 0/11 | Allows all | US |
| 146 |
Procter & Gamble
pg.com
|
A | 100 | 0/11 | Allows all | US |
| 147 |
Philip Morris International
pmi.com
|
A | 100 | 0/11 | Allows all | US |
| 148 |
PwC
pwc.com
|
A | 100 | 0/11 | Allows all | UK |
| 149 |
Qatar Airways
qatarairways.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 150 |
Qualcomm
qualcomm.com
|
A | 100 | 0/11 | Allows all | US |
All 201 graded sites from the latest weekly check, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.
Your Logs already store the data you need,
you just need a way to use it!
If you are interested in HoneyLog, you can try it with our Free Trial!