Business Robots.txt Leaderboard
How the web's top corporate & business websites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
1%
of 201 ranked sites block GPTBot in robots.txt
201
sites graded, last checked Nov 11, 2026
99/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 197 sites
- B
- 2 sites
- C
- 0 sites
- D
- 2 sites
- F
- 0 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 51 |
Chevron
chevron.com
|
A | 100 | 0/11 | Allows all | US |
| 52 |
Citigroup
citigroup.com
|
A | 100 | 0/11 | Allows all | US |
| 53 |
Clifford Chance
cliffordchance.com
|
A | 100 | 0/11 | Allows all | UK |
| 54 |
CN
cn.ca
|
A | 100 | 0/11 | Allows all | CA |
| 55 |
The Coca-Cola Company
coca-colacompany.com
|
A | 100 | 0/11 | Allows all | US |
| 56 |
Codelco
codelco.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 57 |
Commonwealth Bank
commbank.com.au
|
A | 100 | 0/11 | Allows all | Oceania |
| 58 |
ConocoPhillips
conocophillips.com
|
A | 100 | 0/11 | Allows all | US |
| 59 |
CRH
crh.com
|
A | 100 | 0/11 | Allows all | Ireland |
| 60 |
CSL
csl.com
|
A | 100 | 0/11 | Allows all | Oceania |
| 61 |
Danone
danone.com
|
A | 100 | 0/11 | Allows all | EU |
| 62 |
Deutsche Bank
db.com
|
A | 100 | 0/11 | Allows all | EU |
| 63 |
DBS Bank
dbs.com
|
A | 100 | 0/11 | Allows all | Asia |
| 64 |
John Deere
deere.com
|
A | 100 | 0/11 | Allows all | US |
| 65 |
Deloitte
deloitte.com
|
A | 100 | 0/11 | Allows all | UK |
| 66 |
Diageo
diageo.com
|
A | 100 | 0/11 | Allows all | UK |
| 67 |
DNB
dnb.no
|
A | 100 | 0/11 | Allows all | EU |
| 68 |
Dow
dow.com
|
A | 100 | 0/11 | Allows all | US |
| 69 |
DP World
dpworld.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 70 |
Embraer
embraer.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 71 |
Emerson
emerson.com
|
A | 100 | 0/11 | Allows all | US |
| 72 |
Enbridge
enbridge.com
|
A | 100 | 0/11 | Allows all | CA |
| 73 |
Enel
enel.com
|
A | 100 | 0/11 | Allows all | EU |
| 74 |
Engie
engie.com
|
A | 100 | 0/11 | Allows all | EU |
| 75 |
Eni
eni.com
|
A | 100 | 0/11 | Allows all | EU |
| 76 |
ExxonMobil
exxonmobil.com
|
A | 100 | 0/11 | Allows all | US |
| 77 |
EY
ey.com
|
A | 100 | 0/11 | Allows all | UK |
| 78 |
FedEx
fedex.com
|
A | 100 | 0/11 | Allows all | US |
| 79 |
FEMSA
femsa.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 80 |
Ford
ford.com
|
A | 100 | 0/11 | Allows all | US |
| 81 |
Foxconn
foxconn.com
|
A | 100 | 0/11 | Allows all | Asia |
| 82 |
GE Aerospace
geaerospace.com
|
A | 100 | 0/11 | Allows all | US |
| 83 |
Generali
generali.com
|
A | 100 | 0/11 | Allows all | EU |
| 84 |
General Mills
generalmills.com
|
A | 100 | 0/11 | Allows all | US |
| 85 |
Toyota
global.toyota
|
A | 100 | 0/11 | Allows all | Asia |
| 86 |
General Motors
gm.com
|
A | 100 | 0/11 | Allows all | US |
| 87 |
Goldman Sachs
goldmansachs.com
|
A | 100 | 0/11 | Allows all | US |
| 88 |
Grupo Bimbo
grupobimbo.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 89 |
GSK
gsk.com
|
A | 100 | 0/11 | Allows all | UK |
| 90 |
HDFC Bank
hdfcbank.com
|
A | 100 | 0/11 | Allows all | Asia |
| 91 |
Hilton
hilton.com
|
A | 100 | 0/11 | Allows all | US |
| 92 |
Honeywell
honeywell.com
|
A | 100 | 0/11 | Allows all | US |
| 93 |
Hyundai
hyundai.com
|
A | 100 | 0/11 | Allows all | Asia |
| 94 |
Iberdrola
iberdrola.com
|
A | 100 | 0/11 | Allows all | EU |
| 95 |
IBM
ibm.com
|
A | 100 | 0/11 | Allows all | US |
| 96 |
IHG Hotels & Resorts
ihg.com
|
A | 100 | 0/11 | Allows all | UK |
| 97 |
Infosys
infosys.com
|
A | 100 | 0/11 | Allows all | Asia |
| 98 |
ING
ing.com
|
A | 100 | 0/11 | Allows all | EU |
| 99 |
Intel
intel.com
|
A | 100 | 0/11 | Allows all | US |
| 100 |
Intesa Sanpaolo
intesasanpaolo.com
|
A | 100 | 0/11 | Allows all | EU |
All 201 graded sites from the latest weekly check, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.
Your Logs already store the data you need,
you just need a way to use it!
If you are interested in HoneyLog, you can try it with our Free Trial!