Ecommerce Robots.txt Leaderboard
How the web's top online stores & retailers compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
7%
of 199 ranked sites block GPTBot in robots.txt
199
sites graded, last checked Nov 11, 2026
95/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 183 sites
- B
- 3 sites
- C
- 4 sites
- D
- 5 sites
- F
- 4 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 51 |
Brown Thomas
brownthomas.com
|
A | 100 | 0/11 | Allows all | Ireland |
| 52 |
Bunnings
bunnings.com.au
|
A | 100 | 0/11 | Allows all | Oceania |
| 53 |
Canadian Tire
canadiantire.ca
|
A | 100 | 0/11 | Allows all | CA |
| 54 |
Carrefour
carrefour.fr
|
A | 100 | 0/11 | Allows all | EU |
| 55 |
Casas Bahia
casasbahia.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 56 |
Cdiscount
cdiscount.com
|
A | 100 | 0/11 | Allows all | EU |
| 57 |
Chemist Warehouse
chemistwarehouse.com.au
|
A | 100 | 0/11 | Allows all | Oceania |
| 58 |
Conrad
conrad.de
|
A | 100 | 0/11 | Allows all | EU |
| 59 |
Coolblue
coolblue.nl
|
A | 100 | 0/11 | Allows all | EU |
| 60 |
Coppel
coppel.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 61 |
Costco
costco.com
|
A | 100 | 0/11 | Allows all | US |
| 62 |
Coupang
coupang.com
|
A | 100 | 0/11 | Allows all | Asia |
| 63 |
Crate & Barrel
crateandbarrel.com
|
A | 100 | 0/11 | Allows all | US |
| 64 |
Currys
currys.co.uk
|
A | 100 | 0/11 | Allows all | UK |
| 65 |
CVS
cvs.com
|
A | 100 | 0/11 | Allows all | US |
| 66 |
Daraz
daraz.pk
|
A | 100 | 0/11 | Allows all | Asia |
| 67 |
Darty
darty.com
|
A | 100 | 0/11 | Allows all | EU |
| 68 |
Decathlon
decathlon.fr
|
A | 100 | 0/11 | Allows all | EU |
| 69 |
DHgate
dhgate.com
|
A | 100 | 0/11 | Allows all | Asia |
| 70 |
Dick's Sporting Goods
dickssportinggoods.com
|
A | 100 | 0/11 | Allows all | US |
| 71 |
Dillard's
dillards.com
|
A | 100 | 0/11 | Allows all | US |
| 72 |
B&Q
diy.com
|
A | 100 | 0/11 | Allows all | UK |
| 73 |
dm
dm.de
|
A | 100 | 0/11 | Allows all | EU |
| 74 |
Douglas
douglas.de
|
A | 100 | 0/11 | Allows all | EU |
| 75 |
Dunnes Stores
dunnesstores.com
|
A | 100 | 0/11 | Allows all | Ireland |
| 76 |
El Corte Inglés
elcorteingles.es
|
A | 100 | 0/11 | Allows all | EU |
| 77 |
Elgiganten
elgiganten.se
|
A | 100 | 0/11 | Allows all | EU |
| 78 |
eMAG
emag.ro
|
A | 100 | 0/11 | Allows all | EU |
| 79 |
Empik
empik.com
|
A | 100 | 0/11 | Allows all | EU |
| 80 |
Etsy
etsy.com
|
A | 100 | 0/11 | Allows all | US |
| 81 |
Éxito
exito.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 82 |
eXtra
extra.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 83 |
Falabella
falabella.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 84 |
Farfetch
farfetch.com
|
A | 100 | 0/11 | Allows all | UK |
| 85 |
Fashion Nova
fashionnova.com
|
A | 100 | 0/11 | Allows all | US |
| 86 |
Fnac
fnac.com
|
A | 100 | 0/11 | Allows all | EU |
| 87 |
Foot Locker
footlocker.com
|
A | 100 | 0/11 | Allows all | US |
| 88 |
GameStop
gamestop.com
|
A | 100 | 0/11 | Allows all | US |
| 89 |
Gap
gap.com
|
A | 100 | 0/11 | Allows all | US |
| 90 |
Gymshark
gymshark.com
|
A | 100 | 0/11 | Allows all | UK |
| 91 |
Harbor Freight
harborfreight.com
|
A | 100 | 0/11 | Allows all | US |
| 92 |
Harvey Norman
harveynorman.com.au
|
A | 100 | 0/11 | Allows all | Oceania |
| 93 |
Hepsiburada
hepsiburada.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 94 |
HKTVmall
hktvmall.com
|
A | 100 | 0/11 | Allows all | Asia |
| 95 |
H&M
hm.com
|
A | 100 | 0/11 | Allows all | EU |
| 96 |
The Home Depot
homedepot.com
|
A | 100 | 0/11 | Allows all | US |
| 97 |
iHerb
iherb.com
|
A | 100 | 0/11 | Allows all | US |
| 98 |
Indigo
indigo.ca
|
A | 100 | 0/11 | Allows all | CA |
| 99 |
JB Hi-Fi
jbhifi.com.au
|
A | 100 | 0/11 | Allows all | Oceania |
| 100 |
JCPenney
jcpenney.com
|
A | 100 | 0/11 | Allows all | US |
All 199 graded sites from the latest weekly check, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.
Your Logs already store the data you need,
you just need a way to use it!
If you are interested in HoneyLog, you can try it with our Free Trial!