Free public ranking

Ecommerce Robots.txt Leaderboard

How the web's top online stores & retailers compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.

7%

of 199 ranked sites block GPTBot in robots.txt

199

sites graded, last checked Nov 11, 2026

95/100

average openness score (higher = more open to AI)

Share of sites blocking each AI crawler

Click a bot to filter the ranking to sites that block it.

Grade distribution

A = wide open to AI, F = walled off.

A
183 sites
B
3 sites
C
4 sites
D
5 sites
F
4 sites
# AI crawlers Region
51
Brown Thomas
brownthomas.com
A 100 0/11 Allows all Ireland
52
Bunnings
bunnings.com.au
A 100 0/11 Allows all Oceania
53
Canadian Tire
canadiantire.ca
A 100 0/11 Allows all CA
54
Carrefour
carrefour.fr
A 100 0/11 Allows all EU
55
Casas Bahia
casasbahia.com.br
A 100 0/11 Allows all LatAm
56
Cdiscount
cdiscount.com
A 100 0/11 Allows all EU
57
Chemist Warehouse
chemistwarehouse.com.au
A 100 0/11 Allows all Oceania
58
Conrad
conrad.de
A 100 0/11 Allows all EU
59
Coolblue
coolblue.nl
A 100 0/11 Allows all EU
60
Coppel
coppel.com
A 100 0/11 Allows all LatAm
61
Costco
costco.com
A 100 0/11 Allows all US
62
Coupang
coupang.com
A 100 0/11 Allows all Asia
63
Crate & Barrel
crateandbarrel.com
A 100 0/11 Allows all US
64
Currys
currys.co.uk
A 100 0/11 Allows all UK
65
CVS
cvs.com
A 100 0/11 Allows all US
66
Daraz
daraz.pk
A 100 0/11 Allows all Asia
67
Darty
darty.com
A 100 0/11 Allows all EU
68
Decathlon
decathlon.fr
A 100 0/11 Allows all EU
69
DHgate
dhgate.com
A 100 0/11 Allows all Asia
70
Dick's Sporting Goods
dickssportinggoods.com
A 100 0/11 Allows all US
71
Dillard's
dillards.com
A 100 0/11 Allows all US
72
B&Q
diy.com
A 100 0/11 Allows all UK
73
dm
dm.de
A 100 0/11 Allows all EU
74
Douglas
douglas.de
A 100 0/11 Allows all EU
75
Dunnes Stores
dunnesstores.com
A 100 0/11 Allows all Ireland
76
El Corte Inglés
elcorteingles.es
A 100 0/11 Allows all EU
77
Elgiganten
elgiganten.se
A 100 0/11 Allows all EU
78
eMAG
emag.ro
A 100 0/11 Allows all EU
79
Empik
empik.com
A 100 0/11 Allows all EU
80
Etsy
etsy.com
A 100 0/11 Allows all US
81
Éxito
exito.com
A 100 0/11 Allows all LatAm
82
eXtra
extra.com
A 100 0/11 Allows all MENA-Africa
83
Falabella
falabella.com
A 100 0/11 Allows all LatAm
84
Farfetch
farfetch.com
A 100 0/11 Allows all UK
85
Fashion Nova
fashionnova.com
A 100 0/11 Allows all US
86
Fnac
fnac.com
A 100 0/11 Allows all EU
87
Foot Locker
footlocker.com
A 100 0/11 Allows all US
88
GameStop
gamestop.com
A 100 0/11 Allows all US
89
Gap
gap.com
A 100 0/11 Allows all US
90
Gymshark
gymshark.com
A 100 0/11 Allows all UK
91
Harbor Freight
harborfreight.com
A 100 0/11 Allows all US
92
Harvey Norman
harveynorman.com.au
A 100 0/11 Allows all Oceania
93
Hepsiburada
hepsiburada.com
A 100 0/11 Allows all MENA-Africa
94
HKTVmall
hktvmall.com
A 100 0/11 Allows all Asia
95
H&M
hm.com
A 100 0/11 Allows all EU
96
The Home Depot
homedepot.com
A 100 0/11 Allows all US
97
iHerb
iherb.com
A 100 0/11 Allows all US
98
Indigo
indigo.ca
A 100 0/11 Allows all CA
99
JB Hi-Fi
jbhifi.com.au
A 100 0/11 Allows all Oceania
100
JCPenney
jcpenney.com
A 100 0/11 Allows all US
Showing 51 to 100 of 199 results
Download full leaderboard (CSV)

All 199 graded sites from the latest weekly check, regardless of the filters above.

Methodology

For each site we fetch its public robots.txt and grade it against 11 of the major AI crawlers, including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site “blocks” a bot when its robots.txt has a Disallow rule that applies to that crawler’s user-agent token. The openness score is the share of graded bots that are allowed, rolled up into an A–F grade.

This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.

The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.

Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.

14-day Free Trial

Your Logs already store the data you need,
you just need a way to use it!

If you are interested in HoneyLog, you can try it with our Free Trial!

No credit card required