Free public ranking

Marketplace & Directory Robots.txt Leaderboard

How the web's top marketplaces, classifieds & directories compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.

10%

of 209 ranked sites block GPTBot in robots.txt

209

sites graded, last checked Nov 11, 2026

94/100

average openness score (higher = more open to AI)

Share of sites blocking each AI crawler

Click a bot to filter the ranking to sites that block it.

Grade distribution

A = wide open to AI, F = walled off.

A
183 sites
B
14 sites
C
2 sites
D
9 sites
F
1 site
# AI crawlers Region
1
Yelp
yelp.com
F 0 11/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
2
AutoTrader.ca
autotrader.ca
D 27 8/11
GPTBot ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Applebot-Extended Meta-ExternalAgent
CA
3
Glassdoor
glassdoor.com
D 27 8/11
GPTBot ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended
US
4
Instacart
instacart.com
D 27 8/11
GPTBot ClaudeBot Google-Extended PerplexityBot CCBot Bytespider Amazonbot Meta-ExternalAgent
US
5
realestate.com.au
realestate.com.au
D 27 8/11
GPTBot ClaudeBot PerplexityBot CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
Oceania
6
REALTOR.ca
realtor.ca
D 27 8/11
GPTBot ClaudeBot Google-Extended CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
CA
7
Tripadvisor
tripadvisor.com
D 27 8/11
GPTBot ClaudeBot Google-Extended CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
US
8
Wuzzuf
wuzzuf.net
D 27 8/11
GPTBot ClaudeBot Google-Extended CCBot Bytespider Amazonbot Applebot-Extended Meta-ExternalAgent
MENA-Africa
9
Buscapé
buscape.com.br
D 36 7/11
GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Google-Extended PerplexityBot Amazonbot
LatAm
10
Milanuncios
milanuncios.com
D 45 6/11
GPTBot ClaudeBot Google-Extended CCBot Bytespider Amazonbot
EU
11
AutoScout24
autoscout24.de
C 55 5/11
GPTBot ClaudeBot Google-Extended CCBot Applebot-Extended
EU
12
Edmunds
edmunds.com
C 64 4/11
GPTBot ClaudeBot Bytespider Meta-ExternalAgent
US
13
Airbnb
airbnb.com
B 73 3/11
GPTBot ClaudeBot Applebot-Extended
US
14
Cars.co.za
cars.co.za
B 73 3/11
CCBot Bytespider Amazonbot
MENA-Africa
15
Kelkoo
kelkoo.com
B 73 3/11
GPTBot Amazonbot Meta-ExternalAgent
EU
16
MyHome.ie
myhome.ie
B 73 3/11
GPTBot Google-Extended Bytespider
Ireland
17
Trustpilot
trustpilot.com
B 73 3/11
Bytespider Amazonbot Applebot-Extended
EU
18
Wyzant
wyzant.com
B 73 3/11
GPTBot CCBot Applebot-Extended
US
19
Zillow
zillow.com
B 73 3/11
Amazonbot Applebot-Extended Meta-ExternalAgent
US
20
Kakaku.com
kakaku.com
B 82 2/11
Bytespider Amazonbot
Asia
21
PagesJaunes
pagesjaunes.fr
B 82 2/11
CCBot Amazonbot
EU
22
Rightmove
rightmove.co.uk
B 82 2/11
GPTBot CCBot
UK
23
Tabelog
tabelog.com
B 82 2/11
GPTBot Google-Extended
Asia
24
Webjet
webjet.com.au
B 82 2/11
GPTBot ClaudeBot
Oceania
25
ZipRecruiter
ziprecruiter.com
B 82 2/11
Bytespider Amazonbot
US
26
Zumper
zumper.com
B 82 2/11
CCBot Bytespider
US
27
Autotrader
autotrader.com
A 91 1/11
Amazonbot
US
28
Avvo
avvo.com
A 91 1/11
GPTBot
US
29
Capterra
capterra.com
A 91 1/11
Bytespider
US
30
Carsales
carsales.com.au
A 91 1/11
CCBot
Oceania
31
Carwow
carwow.co.uk
A 91 1/11
Meta-ExternalAgent
UK
32
Checkatrade
checkatrade.com
A 91 1/11
Bytespider
UK
33
Eventbrite
eventbrite.com
A 91 1/11
CCBot
US
34
G2
g2.com
A 91 1/11
Bytespider
US
35
Geizhals
geizhals.at
A 91 1/11
Meta-ExternalAgent
EU
36
Gumtree Australia
gumtree.com.au
A 91 1/11
Bytespider
Oceania
37
HomeAdvisor
homeadvisor.com
A 91 1/11
GPTBot
US
38
Justdial
justdial.com
A 91 1/11
ClaudeBot
Asia
39
Kelley Blue Book
kbb.com
A 91 1/11
Amazonbot
US
40
La Centrale
lacentrale.fr
A 91 1/11
CCBot
EU
41
leboncoin
leboncoin.fr
A 91 1/11
Bytespider
EU
42
Manta
manta.com
A 91 1/11
CCBot
US
43
OnTheMarket
onthemarket.com
A 91 1/11
ClaudeBot
UK
44
ProductReview.com.au
productreview.com.au
A 91 1/11
Applebot-Extended
Oceania
45
Realtor.com
realtor.com
A 91 1/11
Meta-ExternalAgent
US
46
Skyscanner
skyscanner.net
A 91 1/11
Meta-ExternalAgent
UK
47
SpareRoom
spareroom.co.uk
A 91 1/11
Bytespider
UK
48
Totaljobs
totaljobs.com
A 91 1/11
Amazonbot
UK
49
Trade Me
trademe.co.nz
A 91 1/11
Bytespider
Oceania
50
Wallapop
wallapop.com
A 91 1/11
CCBot
EU
Showing 1 to 50 of 209 results
Download full leaderboard (CSV)

All 209 graded sites from the latest weekly check, regardless of the filters above.

Methodology

For each site we fetch its public robots.txt and grade it against 11 of the major AI crawlers, including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site “blocks” a bot when its robots.txt has a Disallow rule that applies to that crawler’s user-agent token. The openness score is the share of graded bots that are allowed, rolled up into an A–F grade.

This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.

The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.

Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.

14-day Free Trial

Your Logs already store the data you need,
you just need a way to use it!

If you are interested in HoneyLog, you can try it with our Free Trial!

No credit card required