Marketplace & Directory Robots.txt Leaderboard
How the web's top marketplaces, classifieds & directories compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
10%
of 209 ranked sites block GPTBot in robots.txt
209
sites graded, last checked Nov 11, 2026
94/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 183 sites
- B
- 14 sites
- C
- 2 sites
- D
- 9 sites
- F
- 1 site
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 51 |
2dehands
2dehands.be
|
A | 100 | 0/11 | Allows all | EU |
| 52 |
58.com
58.com
|
A | 100 | 0/11 | Allows all | Asia |
| 53 |
591
591.com.tw
|
A | 100 | 0/11 | Allows all | Asia |
| 54 |
99acres
99acres.com
|
A | 100 | 0/11 | Allows all | Asia |
| 55 |
99designs
99designs.com
|
A | 100 | 0/11 | Allows all | Global |
| 56 |
Agoda
agoda.com
|
A | 100 | 0/11 | Allows all | Asia |
| 57 |
Airtasker
airtasker.com
|
A | 100 | 0/11 | Allows all | Oceania |
| 58 |
Angi
angi.com
|
A | 100 | 0/11 | Allows all | US |
| 59 |
Apartments.com
apartments.com
|
A | 100 | 0/11 | Allows all | US |
| 60 |
Auto Trader
autotrader.co.uk
|
A | 100 | 0/11 | Allows all | UK |
| 61 |
Avito
avito.ru
|
A | 100 | 0/11 | Allows all | EU |
| 62 |
Bark
bark.com
|
A | 100 | 0/11 | Allows all | UK |
| 63 |
Bayt.com
bayt.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 64 |
Bayut
bayut.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 65 |
Better Business Bureau
bbb.org
|
A | 100 | 0/11 | Allows all | US |
| 66 |
BlaBlaCar
blablacar.com
|
A | 100 | 0/11 | Allows all | EU |
| 67 |
Blocket
blocket.se
|
A | 100 | 0/11 | Allows all | EU |
| 68 |
Booking.com
booking.com
|
A | 100 | 0/11 | Allows all | EU |
| 69 |
Bring a Trailer
bringatrailer.com
|
A | 100 | 0/11 | Allows all | US |
| 70 |
CarDekho
cardekho.com
|
A | 100 | 0/11 | Allows all | Asia |
| 71 |
Care.com
care.com
|
A | 100 | 0/11 | Allows all | US |
| 72 |
CareerBuilder
careerbuilder.com
|
A | 100 | 0/11 | Allows all | US |
| 73 |
CarGurus
cargurus.com
|
A | 100 | 0/11 | Allows all | US |
| 74 |
Carousell
carousell.sg
|
A | 100 | 0/11 | Allows all | Asia |
| 75 |
Cars.com
cars.com
|
A | 100 | 0/11 | Allows all | US |
| 76 |
Catho
catho.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 77 |
Centris
centris.ca
|
A | 100 | 0/11 | Allows all | CA |
| 78 |
Check24
check24.de
|
A | 100 | 0/11 | Allows all | EU |
| 79 |
Clutch
clutch.co
|
A | 100 | 0/11 | Allows all | US |
| 80 |
Compare the Market
comparethemarket.com
|
A | 100 | 0/11 | Allows all | UK |
| 81 |
Computrabajo
computrabajo.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 82 |
ConsumerAffairs
consumeraffairs.com
|
A | 100 | 0/11 | Allows all | US |
| 83 |
Craigslist
craigslist.org
|
A | 100 | 0/11 | Allows all | US |
| 84 |
CV-Library
cv-library.co.uk
|
A | 100 | 0/11 | Allows all | UK |
| 85 |
Daft.ie
daft.ie
|
A | 100 | 0/11 | Allows all | Ireland |
| 86 |
Deliveroo
deliveroo.co.uk
|
A | 100 | 0/11 | Allows all | UK |
| 87 |
Despegar
despegar.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 88 |
Dice
dice.com
|
A | 100 | 0/11 | Allows all | US |
| 89 |
Doctolib
doctolib.fr
|
A | 100 | 0/11 | Allows all | EU |
| 90 |
Domain
domain.com.au
|
A | 100 | 0/11 | Allows all | Oceania |
| 91 |
DoneDeal
donedeal.ie
|
A | 100 | 0/11 | Allows all | Ireland |
| 92 |
DoorDash
doordash.com
|
A | 100 | 0/11 | Allows all | US |
| 93 |
dubizzle
dubizzle.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 94 |
eDreams
edreams.com
|
A | 100 | 0/11 | Allows all | EU |
| 95 |
Expedia
expedia.com
|
A | 100 | 0/11 | Allows all | US |
| 96 |
FINN
finn.no
|
A | 100 | 0/11 | Allows all | EU |
| 97 |
Fiverr
fiverr.com
|
A | 100 | 0/11 | Allows all | Global |
| 98 |
Freelancer
freelancer.com
|
A | 100 | 0/11 | Allows all | Oceania |
| 99 |
Funda
funda.nl
|
A | 100 | 0/11 | Allows all | EU |
| 100 |
Gelbe Seiten
gelbeseiten.de
|
A | 100 | 0/11 | Allows all | EU |
All 209 graded sites from the latest weekly check, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.
Your Logs already store the data you need,
you just need a way to use it!
If you are interested in HoneyLog, you can try it with our Free Trial!