Free public ranking

Business Robots.txt Leaderboard

How the web's top corporate & business websites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.

1%

of 201 ranked sites block GPTBot in robots.txt

201

sites graded, last checked Nov 11, 2026

99/100

average openness score (higher = more open to AI)

Share of sites blocking each AI crawler

Click a bot to filter the ranking to sites that block it.

Grade distribution

A = wide open to AI, F = walled off.

A
197 sites
B
2 sites
C
0 sites
D
2 sites
F
0 sites
# AI crawlers Region
101
Itaú Unibanco
itau.com.br
A 100 0/11 Allows all LatAm
102
Jardine Matheson
jardines.com
A 100 0/11 Allows all Asia
103
Johnson & Johnson
jnj.com
A 100 0/11 Allows all US
104
JPMorgan Chase
jpmorganchase.com
A 100 0/11 Allows all US
105
Kirkland & Ellis
kirkland.com
A 100 0/11 Allows all US
106
KPMG
kpmg.com
A 100 0/11 Allows all UK
107
LATAM Airlines
latamairlines.com
A 100 0/11 Allows all LatAm
108
Eli Lilly
lilly.com
A 100 0/11 Allows all US
109
Lloyds Banking Group
lloydsbankinggroup.com
A 100 0/11 Allows all UK
110
Lockheed Martin
lockheedmartin.com
A 100 0/11 Allows all US
111
L'Oréal
loreal.com
A 100 0/11 Allows all EU
112
Lufthansa
lufthansa.com
A 100 0/11 Allows all EU
113
LVMH
lvmh.com
A 100 0/11 Allows all EU
114
Latham & Watkins
lw.com
A 100 0/11 Allows all US
115
Macquarie Group
macquarie.com
A 100 0/11 Allows all Oceania
116
Maersk
maersk.com
A 100 0/11 Allows all EU
117
Magna International
magna.com
A 100 0/11 Allows all CA
118
Manulife
manulife.com
A 100 0/11 Allows all CA
119
Marriott
marriott.com
A 100 0/11 Allows all US
120
Mastercard
mastercard.com
A 100 0/11 Allows all US
121
McDonald's
mcdonalds.com
A 100 0/11 Allows all US
122
Mercedes-Benz
mercedes-benz.com
A 100 0/11 Allows all EU
123
Merck & Co.
merck.com
A 100 0/11 Allows all US
124
MetLife
metlife.com
A 100 0/11 Allows all US
125
Michelin
michelin.com
A 100 0/11 Allows all EU
126
Microsoft
microsoft.com
A 100 0/11 Allows all US
127
Mondelez International
mondelezinternational.com
A 100 0/11 Allows all US
128
MTN Group
mtn.com
A 100 0/11 Allows all MENA-Africa
129
Munich Re
munichre.com
A 100 0/11 Allows all EU
130
National Grid
nationalgrid.com
A 100 0/11 Allows all UK
131
NatWest Group
natwestgroup.com
A 100 0/11 Allows all UK
132
Nestlé
nestle.com
A 100 0/11 Allows all EU
133
NextEra Energy
nexteraenergy.com
A 100 0/11 Allows all US
134
Nissan
nissan-global.com
A 100 0/11 Allows all Asia
135
Nokia
nokia.com
A 100 0/11 Allows all EU
136
Novartis
novartis.com
A 100 0/11 Allows all EU
137
Novo Nordisk
novonordisk.com
A 100 0/11 Allows all EU
138
NTT DATA
nttdata.com
A 100 0/11 Allows all Asia
139
NVIDIA
nvidia.com
A 100 0/11 Allows all US
140
Oracle
oracle.com
A 100 0/11 Allows all US
141
Orange
orange.com
A 100 0/11 Allows all EU
142
Panasonic
panasonic.com
A 100 0/11 Allows all Asia
143
PepsiCo
pepsico.com
A 100 0/11 Allows all US
144
Petrobras
petrobras.com.br
A 100 0/11 Allows all LatAm
145
Pfizer
pfizer.com
A 100 0/11 Allows all US
146
Procter & Gamble
pg.com
A 100 0/11 Allows all US
147
Philip Morris International
pmi.com
A 100 0/11 Allows all US
148
PwC
pwc.com
A 100 0/11 Allows all UK
149
Qatar Airways
qatarairways.com
A 100 0/11 Allows all MENA-Africa
150
Qualcomm
qualcomm.com
A 100 0/11 Allows all US
Showing 101 to 150 of 201 results
Download full leaderboard (CSV)

All 201 graded sites from the latest weekly check, regardless of the filters above.

Methodology

For each site we fetch its public robots.txt and grade it against 11 of the major AI crawlers, including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site “blocks” a bot when its robots.txt has a Disallow rule that applies to that crawler’s user-agent token. The openness score is the share of graded bots that are allowed, rolled up into an A–F grade.

This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.

The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.

Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.

14-day Free Trial

Your Logs already store the data you need,
you just need a way to use it!

If you are interested in HoneyLog, you can try it with our Free Trial!

No credit card required