Marketplace & Directory Robots.txt Leaderboard
How the web's top marketplaces, classifieds & directories compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
10%
of 209 ranked sites block GPTBot in robots.txt
209
sites graded, last checked Nov 11, 2026
94/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 183 sites
- B
- 14 sites
- C
- 2 sites
- D
- 9 sites
- F
- 1 site
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 1 |
Yelp
yelp.com
|
F | 0 | 11/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 2 |
AutoTrader.ca
autotrader.ca
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 3 |
Glassdoor
glassdoor.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
|
US |
| 4 |
Instacart
instacart.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
PerplexityBot
CCBot
Bytespider
Amazonbot
Meta-ExternalAgent
|
US |
| 5 |
realestate.com.au
realestate.com.au
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
PerplexityBot
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
Oceania |
| 6 |
REALTOR.ca
realtor.ca
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
CA |
| 7 |
Tripadvisor
tripadvisor.com
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 8 |
Wuzzuf
wuzzuf.net
|
D | 27 | 8/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
MENA-Africa |
| 9 |
Buscapé
buscape.com.br
|
D | 36 | 7/11 |
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Google-Extended
PerplexityBot
Amazonbot
|
LatAm |
| 10 |
Milanuncios
milanuncios.com
|
D | 45 | 6/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Bytespider
Amazonbot
|
EU |
| 11 |
AutoScout24
autoscout24.de
|
C | 55 | 5/11 |
GPTBot
ClaudeBot
Google-Extended
CCBot
Applebot-Extended
|
EU |
| 12 |
Edmunds
edmunds.com
|
C | 64 | 4/11 |
GPTBot
ClaudeBot
Bytespider
Meta-ExternalAgent
|
US |
| 13 |
Airbnb
airbnb.com
|
B | 73 | 3/11 |
GPTBot
ClaudeBot
Applebot-Extended
|
US |
| 14 |
Cars.co.za
cars.co.za
|
B | 73 | 3/11 |
CCBot
Bytespider
Amazonbot
|
MENA-Africa |
| 15 |
Kelkoo
kelkoo.com
|
B | 73 | 3/11 |
GPTBot
Amazonbot
Meta-ExternalAgent
|
EU |
| 16 |
MyHome.ie
myhome.ie
|
B | 73 | 3/11 |
GPTBot
Google-Extended
Bytespider
|
Ireland |
| 17 |
Trustpilot
trustpilot.com
|
B | 73 | 3/11 |
Bytespider
Amazonbot
Applebot-Extended
|
EU |
| 18 |
Wyzant
wyzant.com
|
B | 73 | 3/11 |
GPTBot
CCBot
Applebot-Extended
|
US |
| 19 |
Zillow
zillow.com
|
B | 73 | 3/11 |
Amazonbot
Applebot-Extended
Meta-ExternalAgent
|
US |
| 20 |
Kakaku.com
kakaku.com
|
B | 82 | 2/11 |
Bytespider
Amazonbot
|
Asia |
| 21 |
PagesJaunes
pagesjaunes.fr
|
B | 82 | 2/11 |
CCBot
Amazonbot
|
EU |
| 22 |
Rightmove
rightmove.co.uk
|
B | 82 | 2/11 |
GPTBot
CCBot
|
UK |
| 23 |
Tabelog
tabelog.com
|
B | 82 | 2/11 |
GPTBot
Google-Extended
|
Asia |
| 24 |
Webjet
webjet.com.au
|
B | 82 | 2/11 |
GPTBot
ClaudeBot
|
Oceania |
| 25 |
ZipRecruiter
ziprecruiter.com
|
B | 82 | 2/11 |
Bytespider
Amazonbot
|
US |
| 26 |
Zumper
zumper.com
|
B | 82 | 2/11 |
CCBot
Bytespider
|
US |
| 27 |
Autotrader
autotrader.com
|
A | 91 | 1/11 |
Amazonbot
|
US |
| 28 |
Avvo
avvo.com
|
A | 91 | 1/11 |
GPTBot
|
US |
| 29 |
Capterra
capterra.com
|
A | 91 | 1/11 |
Bytespider
|
US |
| 30 |
Carsales
carsales.com.au
|
A | 91 | 1/11 |
CCBot
|
Oceania |
| 31 |
Carwow
carwow.co.uk
|
A | 91 | 1/11 |
Meta-ExternalAgent
|
UK |
| 32 |
Checkatrade
checkatrade.com
|
A | 91 | 1/11 |
Bytespider
|
UK |
| 33 |
Eventbrite
eventbrite.com
|
A | 91 | 1/11 |
CCBot
|
US |
| 34 |
G2
g2.com
|
A | 91 | 1/11 |
Bytespider
|
US |
| 35 |
Geizhals
geizhals.at
|
A | 91 | 1/11 |
Meta-ExternalAgent
|
EU |
| 36 |
Gumtree Australia
gumtree.com.au
|
A | 91 | 1/11 |
Bytespider
|
Oceania |
| 37 |
HomeAdvisor
homeadvisor.com
|
A | 91 | 1/11 |
GPTBot
|
US |
| 38 |
Justdial
justdial.com
|
A | 91 | 1/11 |
ClaudeBot
|
Asia |
| 39 |
Kelley Blue Book
kbb.com
|
A | 91 | 1/11 |
Amazonbot
|
US |
| 40 |
La Centrale
lacentrale.fr
|
A | 91 | 1/11 |
CCBot
|
EU |
| 41 |
leboncoin
leboncoin.fr
|
A | 91 | 1/11 |
Bytespider
|
EU |
| 42 |
Manta
manta.com
|
A | 91 | 1/11 |
CCBot
|
US |
| 43 |
OnTheMarket
onthemarket.com
|
A | 91 | 1/11 |
ClaudeBot
|
UK |
| 44 |
ProductReview.com.au
productreview.com.au
|
A | 91 | 1/11 |
Applebot-Extended
|
Oceania |
| 45 |
Realtor.com
realtor.com
|
A | 91 | 1/11 |
Meta-ExternalAgent
|
US |
| 46 |
Skyscanner
skyscanner.net
|
A | 91 | 1/11 |
Meta-ExternalAgent
|
UK |
| 47 |
SpareRoom
spareroom.co.uk
|
A | 91 | 1/11 |
Bytespider
|
UK |
| 48 |
Totaljobs
totaljobs.com
|
A | 91 | 1/11 |
Amazonbot
|
UK |
| 49 |
Trade Me
trademe.co.nz
|
A | 91 | 1/11 |
Bytespider
|
Oceania |
| 50 |
Wallapop
wallapop.com
|
A | 91 | 1/11 |
CCBot
|
EU |
All 209 graded sites from the latest weekly check, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.
Your Logs already store the data you need,
you just need a way to use it!
If you are interested in HoneyLog, you can try it with our Free Trial!