Ecommerce Robots.txt Leaderboard
How the web's top online stores & retailers compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
7%
of 199 ranked sites block GPTBot in robots.txt
199
sites graded, last checked Nov 11, 2026
95/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 183 sites
- B
- 3 sites
- C
- 4 sites
- D
- 5 sites
- F
- 4 sites
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 1 |
Gmarket
gmarket.co.kr
|
F | 9 | 10/11 |
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Asia |
| 2 |
Amazon UK
amazon.co.uk
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Meta-ExternalAgent
|
UK |
| 3 |
Amazon
amazon.com
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Meta-ExternalAgent
|
US |
| 4 |
Amazon Germany
amazon.de
|
F | 18 | 9/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Meta-ExternalAgent
|
EU |
| 5 |
Asda
asda.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 6 |
eBay UK
ebay.co.uk
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
UK |
| 7 |
eBay
ebay.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 8 |
eBay Germany
ebay.de
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
EU |
| 9 |
AliExpress
aliexpress.com
|
D | 45 | 6/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
|
Global |
| 10 |
Mercado Libre Argentina
mercadolibre.com.ar
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
ClaudeBot
PerplexityBot
Amazonbot
|
LatAm |
| 11 |
Mercado Libre México
mercadolibre.com.mx
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
ClaudeBot
PerplexityBot
Amazonbot
|
LatAm |
| 12 |
Mercado Livre
mercadolivre.com.br
|
C | 55 | 5/11 |
GPTBot
ChatGPT-User
ClaudeBot
PerplexityBot
Amazonbot
|
LatAm |
| 13 |
Vinted
vinted.fr
|
C | 55 | 5/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Amazonbot
|
EU |
| 14 |
Americanas
americanas.com.br
|
B | 73 | 3/11 |
GPTBot
Google-Extended
CCBot
|
LatAm |
| 15 |
Jarir Bookstore
jarir.com
|
B | 82 | 2/11 |
CCBot
Bytespider
|
MENA-Africa |
| 16 |
MediaMarkt
mediamarkt.de
|
B | 82 | 2/11 |
CCBot
Amazonbot
|
EU |
| 17 |
Chewy
chewy.com
|
A | 91 | 1/11 |
Amazonbot
|
US |
| 18 |
Fanatics
fanatics.com
|
A | 91 | 1/11 |
CCBot
|
US |
| 19 |
Flipkart
flipkart.com
|
A | 91 | 1/11 |
Amazonbot
|
Asia |
| 20 |
IKEA
ikea.com
|
A | 91 | 1/11 |
Amazonbot
|
EU |
| 21 |
Jumia Egypt
jumia.com.eg
|
A | 91 | 1/11 |
CCBot
|
MENA-Africa |
| 22 |
Jumia Nigeria
jumia.com.ng
|
A | 91 | 1/11 |
CCBot
|
MENA-Africa |
| 23 |
MUSINSA
musinsa.com
|
A | 91 | 1/11 |
Bytespider
|
Asia |
| 24 |
Thomann
thomann.de
|
A | 91 | 1/11 |
GPTBot
|
EU |
| 25 |
1-800-Flowers.com
1800flowers.com
|
A | 100 | 0/11 | Allows all | US |
| 26 |
Abercrombie & Fitch
abercrombie.com
|
A | 100 | 0/11 | Allows all | US |
| 27 |
ABOUT YOU
aboutyou.de
|
A | 100 | 0/11 | Allows all | EU |
| 28 |
Academy Sports + Outdoors
academy.com
|
A | 100 | 0/11 | Allows all | US |
| 29 |
American Eagle
ae.com
|
A | 100 | 0/11 | Allows all | US |
| 30 |
Albert Heijn
ah.nl
|
A | 100 | 0/11 | Allows all | EU |
| 31 |
AJIO
ajio.com
|
A | 100 | 0/11 | Allows all | Asia |
| 32 |
Allegro
allegro.pl
|
A | 100 | 0/11 | Allows all | EU |
| 33 |
Alza
alza.cz
|
A | 100 | 0/11 | Allows all | EU |
| 34 |
Anthropologie
anthropologie.com
|
A | 100 | 0/11 | Allows all | US |
| 35 |
Argos
argos.co.uk
|
A | 100 | 0/11 | Allows all | UK |
| 36 |
Aritzia
aritzia.com
|
A | 100 | 0/11 | Allows all | CA |
| 37 |
AutoZone
autozone.com
|
A | 100 | 0/11 | Allows all | US |
| 38 |
Barnes & Noble
barnesandnoble.com
|
A | 100 | 0/11 | Allows all | US |
| 39 |
Bath & Body Works
bathandbodyworks.com
|
A | 100 | 0/11 | Allows all | US |
| 40 |
Best Buy Canada
bestbuy.ca
|
A | 100 | 0/11 | Allows all | CA |
| 41 |
Best Buy
bestbuy.com
|
A | 100 | 0/11 | Allows all | US |
| 42 |
B&H Photo Video
bhphotovideo.com
|
A | 100 | 0/11 | Allows all | US |
| 43 |
BigBasket
bigbasket.com
|
A | 100 | 0/11 | Allows all | Asia |
| 44 |
BJ's Wholesale Club
bjs.com
|
A | 100 | 0/11 | Allows all | US |
| 45 |
Bloomingdale's
bloomingdales.com
|
A | 100 | 0/11 | Allows all | US |
| 46 |
Blue Nile
bluenile.com
|
A | 100 | 0/11 | Allows all | US |
| 47 |
bol
bol.com
|
A | 100 | 0/11 | Allows all | EU |
| 48 |
boohoo
boohoo.com
|
A | 100 | 0/11 | Allows all | UK |
| 49 |
Boots
boots.com
|
A | 100 | 0/11 | Allows all | UK |
| 50 |
Boozt
boozt.com
|
A | 100 | 0/11 | Allows all | EU |
All 199 graded sites from the latest weekly check, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.
Your Logs already store the data you need,
you just need a way to use it!
If you are interested in HoneyLog, you can try it with our Free Trial!