Free public ranking

Ecommerce Robots.txt Leaderboard

How the web's top online stores & retailers compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.

7%

of 199 ranked sites block GPTBot in robots.txt

199

sites graded, last checked Nov 11, 2026

95/100

average openness score (higher = more open to AI)

Share of sites blocking each AI crawler

Click a bot to filter the ranking to sites that block it.

Grade distribution

A = wide open to AI, F = walled off.

A
183 sites
B
3 sites
C
4 sites
D
5 sites
F
4 sites
# AI crawlers Region
151
REI
rei.com
A 100 0/11 Allows all US
152
Saks Fifth Avenue
saksfifthavenue.com
A 100 0/11 Allows all US
153
Sam's Club
samsclub.com
A 100 0/11 Allows all US
154
Screwfix
screwfix.com
A 100 0/11 Allows all UK
155
Selfridges
selfridges.com
A 100 0/11 Allows all UK
156
Sephora
sephora.com
A 100 0/11 Allows all US
157
SHEIN
shein.com
A 100 0/11 Allows all Global
158
Shopee Indonesia
shopee.co.id
A 100 0/11 Allows all Asia
159
Shopee Brasil
shopee.com.br
A 100 0/11 Allows all LatAm
160
Shopee Singapore
shopee.sg
A 100 0/11 Allows all Asia
161
Shoppers Drug Mart
shoppersdrugmart.ca
A 100 0/11 Allows all CA
162
Simons
simons.ca
A 100 0/11 Allows all CA
163
SSENSE
ssense.com
A 100 0/11 Allows all CA
164
Staples
staples.com
A 100 0/11 Allows all US
165
StockX
stockx.com
A 100 0/11 Allows all US
166
Takealot
takealot.com
A 100 0/11 Allows all MENA-Africa
167
Taobao
taobao.com
A 100 0/11 Allows all Asia
168
Target
target.com
A 100 0/11 Allows all US
169
Temu
temu.com
A 100 0/11 Allows all Global
170
Tesco
tesco.com
A 100 0/11 Allows all UK
171
THE ICONIC
theiconic.com.au
A 100 0/11 Allows all Oceania
172
The RealReal
therealreal.com
A 100 0/11 Allows all US
173
The Warehouse
thewarehouse.co.nz
A 100 0/11 Allows all Oceania
174
ThriftBooks
thriftbooks.com
A 100 0/11 Allows all US
175
Ticketmaster
ticketmaster.com
A 100 0/11 Allows all US
176
Tmall
tmall.com
A 100 0/11 Allows all Asia
177
Tokopedia
tokopedia.com
A 100 0/11 Allows all Asia
178
Trendyol
trendyol.com
A 100 0/11 Allows all MENA-Africa
179
Ulta Beauty
ulta.com
A 100 0/11 Allows all US
180
Unieuro
unieuro.it
A 100 0/11 Allows all EU
181
Urban Outfitters
urbanoutfitters.com
A 100 0/11 Allows all US
182
Veepee
veepee.fr
A 100 0/11 Allows all EU
183
Verkkokauppa.com
verkkokauppa.com
A 100 0/11 Allows all EU
184
Very
very.co.uk
A 100 0/11 Allows all UK
185
Victoria's Secret
victoriassecret.com
A 100 0/11 Allows all US
186
Walgreens
walgreens.com
A 100 0/11 Allows all US
187
Walmart Canada
walmart.ca
A 100 0/11 Allows all CA
188
Walmart
walmart.com
A 100 0/11 Allows all US
189
Walmart México
walmart.com.mx
A 100 0/11 Allows all LatAm
190
Warby Parker
warbyparker.com
A 100 0/11 Allows all US
191
Wayfair
wayfair.com
A 100 0/11 Allows all US
192
Williams Sonoma
williams-sonoma.com
A 100 0/11 Allows all US
193
Woolworths
woolworths.com.au
A 100 0/11 Allows all Oceania
194
Worten
worten.pt
A 100 0/11 Allows all EU
195
YOOX
yoox.com
A 100 0/11 Allows all EU
196
Zalando
zalando.de
A 100 0/11 Allows all EU
197
Zappos
zappos.com
A 100 0/11 Allows all US
198
Zara
zara.com
A 100 0/11 Allows all EU
199
zooplus
zooplus.de
A 100 0/11 Allows all EU
Showing 151 to 199 of 199 results
Download full leaderboard (CSV)

All 199 graded sites from the latest weekly check, regardless of the filters above.

Methodology

For each site we fetch its public robots.txt and grade it against 11 of the major AI crawlers, including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site “blocks” a bot when its robots.txt has a Disallow rule that applies to that crawler’s user-agent token. The openness score is the share of graded bots that are allowed, rolled up into an A–F grade.

This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.

The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.

Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.

14-day Free Trial

Your Logs already store the data you need,
you just need a way to use it!

If you are interested in HoneyLog, you can try it with our Free Trial!

No credit card required