Free public ranking

Ecommerce Robots.txt Leaderboard

How the web's top online stores & retailers compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.

7%

of 199 ranked sites block GPTBot in robots.txt

199

sites graded, last checked Nov 11, 2026

95/100

average openness score (higher = more open to AI)

Share of sites blocking each AI crawler

Click a bot to filter the ranking to sites that block it.

Grade distribution

A = wide open to AI, F = walled off.

A
183 sites
B
3 sites
C
4 sites
D
5 sites
F
4 sites
# AI crawlers Region
101
J.Crew
jcrew.com
A 100 0/11 Allows all US
102
JD.com
jd.com
A 100 0/11 Allows all Asia
103
JD Sports
jdsports.co.uk
A 100 0/11 Allows all UK
104
John Lewis
johnlewis.com
A 100 0/11 Allows all UK
105
KaBuM!
kabum.com.br
A 100 0/11 Allows all LatAm
106
Kaufland
kaufland.de
A 100 0/11 Allows all EU
107
Kmart Australia
kmart.com.au
A 100 0/11 Allows all Oceania
108
Kogan
kogan.com
A 100 0/11 Allows all Oceania
109
Kohl's
kohls.com
A 100 0/11 Allows all US
110
Konga
konga.com
A 100 0/11 Allows all MENA-Africa
111
La Redoute
laredoute.fr
A 100 0/11 Allows all EU
112
Lazada Philippines
lazada.com.ph
A 100 0/11 Allows all Asia
113
Lazada Singapore
lazada.sg
A 100 0/11 Allows all Asia
114
LEGO
lego.com
A 100 0/11 Allows all EU
115
Lenskart
lenskart.com
A 100 0/11 Allows all Asia
116
Leroy Merlin
leroymerlin.fr
A 100 0/11 Allows all EU
117
Littlewoods Ireland
littlewoodsireland.ie
A 100 0/11 Allows all Ireland
118
Liverpool
liverpool.com.mx
A 100 0/11 Allows all LatAm
119
L.L.Bean
llbean.com
A 100 0/11 Allows all US
120
Lowe's
lowes.com
A 100 0/11 Allows all US
121
Macy's
macys.com
A 100 0/11 Allows all US
122
Magazine Luiza
magazineluiza.com.br
A 100 0/11 Allows all LatAm
123
Mango
mango.com
A 100 0/11 Allows all EU
124
ManoMano
manomano.fr
A 100 0/11 Allows all EU
125
Marks & Spencer
marksandspencer.com
A 100 0/11 Allows all UK
126
Meesho
meesho.com
A 100 0/11 Allows all Asia
127
Mercari
mercari.com
A 100 0/11 Allows all US
128
Michaels
michaels.com
A 100 0/11 Allows all US
129
Micro Center
microcenter.com
A 100 0/11 Allows all US
130
momo
momoshop.com.tw
A 100 0/11 Allows all Asia
131
Myer
myer.com.au
A 100 0/11 Allows all Oceania
132
Mytheresa
mytheresa.com
A 100 0/11 Allows all EU
133
Namshi
namshi.com
A 100 0/11 Allows all MENA-Africa
134
NET-A-PORTER
net-a-porter.com
A 100 0/11 Allows all UK
135
Newegg
newegg.com
A 100 0/11 Allows all US
136
Next
next.co.uk
A 100 0/11 Allows all UK
137
Nike
nike.com
A 100 0/11 Allows all US
138
noon
noon.com
A 100 0/11 Allows all MENA-Africa
139
Nordstrom
nordstrom.com
A 100 0/11 Allows all US
140
Notino
notino.cz
A 100 0/11 Allows all EU
141
Nykaa
nykaa.com
A 100 0/11 Allows all Asia
142
Ocado
ocado.com
A 100 0/11 Allows all UK
143
OTTO
otto.de
A 100 0/11 Allows all EU
144
Patagonia
patagonia.com
A 100 0/11 Allows all US
145
PcComponentes
pccomponentes.com
A 100 0/11 Allows all EU
146
Petco
petco.com
A 100 0/11 Allows all US
147
Poshmark
poshmark.com
A 100 0/11 Allows all US
148
QVC
qvc.com
A 100 0/11 Allows all US
149
Rakuten Ichiba
rakuten.co.jp
A 100 0/11 Allows all Asia
150
Ralph Lauren
ralphlauren.com
A 100 0/11 Allows all US
Showing 101 to 150 of 199 results
Download full leaderboard (CSV)

All 199 graded sites from the latest weekly check, regardless of the filters above.

Methodology

For each site we fetch its public robots.txt and grade it against 11 of the major AI crawlers, including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site “blocks” a bot when its robots.txt has a Disallow rule that applies to that crawler’s user-agent token. The openness score is the share of graded bots that are allowed, rolled up into an A–F grade.

This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.

The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.

Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.

14-day Free Trial

Your Logs already store the data you need,
you just need a way to use it!

If you are interested in HoneyLog, you can try it with our Free Trial!

No credit card required