Free public ranking

Business Robots.txt Leaderboard

How the web's top corporate & business websites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.

1%

of 201 ranked sites block GPTBot in robots.txt

201

sites graded, last checked Nov 11, 2026

99/100

average openness score (higher = more open to AI)

Share of sites blocking each AI crawler

Click a bot to filter the ranking to sites that block it.

Grade distribution

A = wide open to AI, F = walled off.

A
197 sites
B
2 sites
C
0 sites
D
2 sites
F
0 sites
# AI crawlers Region
51
Chevron
chevron.com
A 100 0/11 Allows all US
52
Citigroup
citigroup.com
A 100 0/11 Allows all US
53
Clifford Chance
cliffordchance.com
A 100 0/11 Allows all UK
54
CN
cn.ca
A 100 0/11 Allows all CA
55
The Coca-Cola Company
coca-colacompany.com
A 100 0/11 Allows all US
56
Codelco
codelco.com
A 100 0/11 Allows all LatAm
57
Commonwealth Bank
commbank.com.au
A 100 0/11 Allows all Oceania
58
ConocoPhillips
conocophillips.com
A 100 0/11 Allows all US
59
CRH
crh.com
A 100 0/11 Allows all Ireland
60
CSL
csl.com
A 100 0/11 Allows all Oceania
61
Danone
danone.com
A 100 0/11 Allows all EU
62
Deutsche Bank
db.com
A 100 0/11 Allows all EU
63
DBS Bank
dbs.com
A 100 0/11 Allows all Asia
64
John Deere
deere.com
A 100 0/11 Allows all US
65
Deloitte
deloitte.com
A 100 0/11 Allows all UK
66
Diageo
diageo.com
A 100 0/11 Allows all UK
67
DNB
dnb.no
A 100 0/11 Allows all EU
68
Dow
dow.com
A 100 0/11 Allows all US
69
DP World
dpworld.com
A 100 0/11 Allows all MENA-Africa
70
Embraer
embraer.com
A 100 0/11 Allows all LatAm
71
Emerson
emerson.com
A 100 0/11 Allows all US
72
Enbridge
enbridge.com
A 100 0/11 Allows all CA
73
Enel
enel.com
A 100 0/11 Allows all EU
74
Engie
engie.com
A 100 0/11 Allows all EU
75
Eni
eni.com
A 100 0/11 Allows all EU
76
ExxonMobil
exxonmobil.com
A 100 0/11 Allows all US
77
EY
ey.com
A 100 0/11 Allows all UK
78
FedEx
fedex.com
A 100 0/11 Allows all US
79
FEMSA
femsa.com
A 100 0/11 Allows all LatAm
80
Ford
ford.com
A 100 0/11 Allows all US
81
Foxconn
foxconn.com
A 100 0/11 Allows all Asia
82
GE Aerospace
geaerospace.com
A 100 0/11 Allows all US
83
Generali
generali.com
A 100 0/11 Allows all EU
84
General Mills
generalmills.com
A 100 0/11 Allows all US
85
Toyota
global.toyota
A 100 0/11 Allows all Asia
86
General Motors
gm.com
A 100 0/11 Allows all US
87
Goldman Sachs
goldmansachs.com
A 100 0/11 Allows all US
88
Grupo Bimbo
grupobimbo.com
A 100 0/11 Allows all LatAm
89
GSK
gsk.com
A 100 0/11 Allows all UK
90
HDFC Bank
hdfcbank.com
A 100 0/11 Allows all Asia
91
Hilton
hilton.com
A 100 0/11 Allows all US
92
Honeywell
honeywell.com
A 100 0/11 Allows all US
93
Hyundai
hyundai.com
A 100 0/11 Allows all Asia
94
Iberdrola
iberdrola.com
A 100 0/11 Allows all EU
95
IBM
ibm.com
A 100 0/11 Allows all US
96
IHG Hotels & Resorts
ihg.com
A 100 0/11 Allows all UK
97
Infosys
infosys.com
A 100 0/11 Allows all Asia
98
ING
ing.com
A 100 0/11 Allows all EU
99
Intel
intel.com
A 100 0/11 Allows all US
100
Intesa Sanpaolo
intesasanpaolo.com
A 100 0/11 Allows all EU
Showing 51 to 100 of 201 results
Download full leaderboard (CSV)

All 201 graded sites from the latest weekly check, regardless of the filters above.

Methodology

For each site we fetch its public robots.txt and grade it against 11 of the major AI crawlers, including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site “blocks” a bot when its robots.txt has a Disallow rule that applies to that crawler’s user-agent token. The openness score is the share of graded bots that are allowed, rolled up into an A–F grade.

This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.

The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.

Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.

14-day Free Trial

Your Logs already store the data you need,
you just need a way to use it!

If you are interested in HoneyLog, you can try it with our Free Trial!

No credit card required