Free public ranking

Ecommerce Robots.txt Leaderboard

How the web's top online stores & retailers compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.

7%

of 199 ranked sites block GPTBot in robots.txt

199

sites graded, last checked Nov 11, 2026

95/100

average openness score (higher = more open to AI)

Share of sites blocking each AI crawler

Click a bot to filter the ranking to sites that block it.

Grade distribution

A = wide open to AI, F = walled off.

A
183 sites
B
3 sites
C
4 sites
D
5 sites
F
4 sites
# AI crawlers Region
1
Gmarket
gmarket.co.kr
F 9 10/11
OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
Asia
2
Amazon UK
amazon.co.uk
F 18 9/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Meta-ExternalAgent
UK
3
Amazon
amazon.com
F 18 9/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Meta-ExternalAgent
US
4
Amazon Germany
amazon.de
F 18 9/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Meta-ExternalAgent
EU
5
Asda
asda.com
D 27 8/11
GPTBot ClaudeBot Google-Extended CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
UK
6
eBay UK
ebay.co.uk
D 27 8/11
GPTBot ClaudeBot PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
UK
7
eBay
ebay.com
D 27 8/11
GPTBot ClaudeBot PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
8
eBay Germany
ebay.de
D 27 8/11
GPTBot ClaudeBot PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
EU
9
AliExpress
aliexpress.com
D 45 6/11
GPTBot ClaudeBot Google-Extended CCBot Bytespider Amazonbot
Global
10
Mercado Libre Argentina
mercadolibre.com.ar
C 55 5/11
GPTBot ChatGPT-User ClaudeBot PerplexityBot Amazonbot
LatAm
11
Mercado Libre México
mercadolibre.com.mx
C 55 5/11
GPTBot ChatGPT-User ClaudeBot PerplexityBot Amazonbot
LatAm
12
Mercado Livre
mercadolivre.com.br
C 55 5/11
GPTBot ChatGPT-User ClaudeBot PerplexityBot Amazonbot
LatAm
13
Vinted
vinted.fr
C 55 5/11
GPTBot ClaudeBot Google-Extended CCBot Amazonbot
EU
14
Americanas
americanas.com.br
B 73 3/11
GPTBot Google-Extended CCBot
LatAm
15
Jarir Bookstore
jarir.com
B 82 2/11
CCBot Bytespider
MENA-Africa
16
MediaMarkt
mediamarkt.de
B 82 2/11
CCBot Amazonbot
EU
17
Chewy
chewy.com
A 91 1/11
Amazonbot
US
18
Fanatics
fanatics.com
A 91 1/11
CCBot
US
19
Flipkart
flipkart.com
A 91 1/11
Amazonbot
Asia
20
IKEA
ikea.com
A 91 1/11
Amazonbot
EU
21
Jumia Egypt
jumia.com.eg
A 91 1/11
CCBot
MENA-Africa
22
Jumia Nigeria
jumia.com.ng
A 91 1/11
CCBot
MENA-Africa
23
MUSINSA
musinsa.com
A 91 1/11
Bytespider
Asia
24
Thomann
thomann.de
A 91 1/11
GPTBot
EU
25
1-800-Flowers.com
1800flowers.com
A 100 0/11 Allows all US
26
Abercrombie & Fitch
abercrombie.com
A 100 0/11 Allows all US
27
ABOUT YOU
aboutyou.de
A 100 0/11 Allows all EU
28
Academy Sports + Outdoors
academy.com
A 100 0/11 Allows all US
29
American Eagle
ae.com
A 100 0/11 Allows all US
30
Albert Heijn
ah.nl
A 100 0/11 Allows all EU
31
AJIO
ajio.com
A 100 0/11 Allows all Asia
32
Allegro
allegro.pl
A 100 0/11 Allows all EU
33
Alza
alza.cz
A 100 0/11 Allows all EU
34
Anthropologie
anthropologie.com
A 100 0/11 Allows all US
35
Argos
argos.co.uk
A 100 0/11 Allows all UK
36
Aritzia
aritzia.com
A 100 0/11 Allows all CA
37
AutoZone
autozone.com
A 100 0/11 Allows all US
38
Barnes & Noble
barnesandnoble.com
A 100 0/11 Allows all US
39
Bath & Body Works
bathandbodyworks.com
A 100 0/11 Allows all US
40
Best Buy Canada
bestbuy.ca
A 100 0/11 Allows all CA
41
Best Buy
bestbuy.com
A 100 0/11 Allows all US
42
B&H Photo Video
bhphotovideo.com
A 100 0/11 Allows all US
43
BigBasket
bigbasket.com
A 100 0/11 Allows all Asia
44
BJ's Wholesale Club
bjs.com
A 100 0/11 Allows all US
45
Bloomingdale's
bloomingdales.com
A 100 0/11 Allows all US
46
Blue Nile
bluenile.com
A 100 0/11 Allows all US
47
bol
bol.com
A 100 0/11 Allows all EU
48
boohoo
boohoo.com
A 100 0/11 Allows all UK
49
Boots
boots.com
A 100 0/11 Allows all UK
50
Boozt
boozt.com
A 100 0/11 Allows all EU
Showing 1 to 50 of 199 results
Download full leaderboard (CSV)

All 199 graded sites from the latest weekly check, regardless of the filters above.

Methodology

For each site we fetch its public robots.txt and grade it against 11 of the major AI crawlers, including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site “blocks” a bot when its robots.txt has a Disallow rule that applies to that crawler’s user-agent token. The openness score is the share of graded bots that are allowed, rolled up into an A–F grade.

This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.

The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.

Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.

14-day Free Trial

Your Logs already store the data you need,
you just need a way to use it!

If you are interested in HoneyLog, you can try it with our Free Trial!

No credit card required