Free public ranking

Business Robots.txt Leaderboard

How the web's top corporate & business websites compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.

1%

of 201 ranked sites block GPTBot in robots.txt

201

sites graded, last checked Nov 11, 2026

99/100

average openness score (higher = more open to AI)

Share of sites blocking each AI crawler

Click a bot to filter the ranking to sites that block it.

Grade distribution

A = wide open to AI, F = walled off.

A
197 sites
B
2 sites
C
0 sites
D
2 sites
F
0 sites
# AI crawlers Region
1
Carlsberg Group
carlsberggroup.com
D 27 8/11
GPTBot ClaudeBot Google-Extended CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
2
Kerry Group
kerry.com
D 45 6/11
GPTBot ClaudeBot CCBot Bytespider Applebot-Extended Meta-ExternalAgent
Ireland
3
SAP
sap.com
B 73 3/11
Bytespider Applebot-Extended Meta-ExternalAgent
EU
4
Telstra
telstra.com.au
B 73 3/11
CCBot Bytespider Amazonbot
Oceania
5
Airbus
airbus.com
A 91 1/11
Bytespider
EU
6
3M
3m.com
A 100 0/11 Allows all US
7
American Airlines
aa.com
A 100 0/11 Allows all US
8
AB InBev
ab-inbev.com
A 100 0/11 Allows all EU
9
AbbVie
abbvie.com
A 100 0/11 Allows all US
10
Accenture
accenture.com
A 100 0/11 Allows all Ireland
11
ADNOC
adnoc.ae
A 100 0/11 Allows all MENA-Africa
12
Air Canada
aircanada.com
A 100 0/11 Allows all CA
13
Air New Zealand
airnewzealand.com
A 100 0/11 Allows all Oceania
14
Alibaba Group
alibabagroup.com
A 100 0/11 Allows all Asia
15
Allianz
allianz.com
A 100 0/11 Allows all EU
16
Ambev
ambev.com.br
A 100 0/11 Allows all LatAm
17
América Móvil
americamovil.com
A 100 0/11 Allows all LatAm
18
American Express
americanexpress.com
A 100 0/11 Allows all US
19
Amgen
amgen.com
A 100 0/11 Allows all US
20
ANZ
anz.com
A 100 0/11 Allows all Oceania
21
Saudi Aramco
aramco.com
A 100 0/11 Allows all MENA-Africa
22
ASML
asml.com
A 100 0/11 Allows all EU
23
AT&T
att.com
A 100 0/11 Allows all US
24
Aviva
aviva.com
A 100 0/11 Allows all UK
25
AXA
axa.com
A 100 0/11 Allows all EU
26
BAE Systems
baesystems.com
A 100 0/11 Allows all UK
27
Bain & Company
bain.com
A 100 0/11 Allows all US
28
Baker McKenzie
bakermckenzie.com
A 100 0/11 Allows all US
29
Bank of America
bankofamerica.com
A 100 0/11 Allows all US
30
BASF
basf.com
A 100 0/11 Allows all EU
31
Bayer
bayer.com
A 100 0/11 Allows all EU
32
BBVA
bbva.com
A 100 0/11 Allows all EU
33
Boston Consulting Group
bcg.com
A 100 0/11 Allows all US
34
BHP
bhp.com
A 100 0/11 Allows all Oceania
35
BlackRock
blackrock.com
A 100 0/11 Allows all US
36
Bristol Myers Squibb
bms.com
A 100 0/11 Allows all US
37
BMW Group
bmwgroup.com
A 100 0/11 Allows all EU
38
BNP Paribas
bnpparibas.com
A 100 0/11 Allows all EU
39
Boeing
boeing.com
A 100 0/11 Allows all US
40
Booz Allen Hamilton
boozallen.com
A 100 0/11 Allows all US
41
Bosch
bosch.com
A 100 0/11 Allows all EU
42
BP
bp.com
A 100 0/11 Allows all UK
43
Bridgestone
bridgestone.com
A 100 0/11 Allows all Asia
44
BT Group
bt.com
A 100 0/11 Allows all UK
45
BYD
byd.com
A 100 0/11 Allows all Asia
46
Capgemini
capgemini.com
A 100 0/11 Allows all EU
47
Cargill
cargill.com
A 100 0/11 Allows all US
48
CBRE
cbre.com
A 100 0/11 Allows all US
49
Cemex
cemex.com
A 100 0/11 Allows all LatAm
50
CGI
cgi.com
A 100 0/11 Allows all CA
Showing 1 to 50 of 201 results
Download full leaderboard (CSV)

All 201 graded sites from the latest weekly check, regardless of the filters above.

Methodology

For each site we fetch its public robots.txt and grade it against 11 of the major AI crawlers, including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site “blocks” a bot when its robots.txt has a Disallow rule that applies to that crawler’s user-agent token. The openness score is the share of graded bots that are allowed, rolled up into an A–F grade.

This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.

The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.

Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.

14-day Free Trial

Your Logs already store the data you need,
you just need a way to use it!

If you are interested in HoneyLog, you can try it with our Free Trial!

No credit card required