Free public ranking

Marketplace & Directory Robots.txt Leaderboard

How the web's top marketplaces, classifieds & directories compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.

10%

of 209 ranked sites block GPTBot in robots.txt

209

sites graded, last checked Nov 11, 2026

94/100

average openness score (higher = more open to AI)

Share of sites blocking each AI crawler

Click a bot to filter the ranking to sites that block it.

Grade distribution

A = wide open to AI, F = walled off.

A
183 sites
B
14 sites
C
2 sites
D
9 sites
F
1 site
# AI crawlers Region
51
2dehands
2dehands.be
A 100 0/11 Allows all EU
52
58.com
58.com
A 100 0/11 Allows all Asia
53
591
591.com.tw
A 100 0/11 Allows all Asia
54
99acres
99acres.com
A 100 0/11 Allows all Asia
55
99designs
99designs.com
A 100 0/11 Allows all Global
56
Agoda
agoda.com
A 100 0/11 Allows all Asia
57
Airtasker
airtasker.com
A 100 0/11 Allows all Oceania
58
Angi
angi.com
A 100 0/11 Allows all US
59
Apartments.com
apartments.com
A 100 0/11 Allows all US
60
Auto Trader
autotrader.co.uk
A 100 0/11 Allows all UK
61
Avito
avito.ru
A 100 0/11 Allows all EU
62
Bark
bark.com
A 100 0/11 Allows all UK
63
Bayt.com
bayt.com
A 100 0/11 Allows all MENA-Africa
64
Bayut
bayut.com
A 100 0/11 Allows all MENA-Africa
65
Better Business Bureau
bbb.org
A 100 0/11 Allows all US
66
BlaBlaCar
blablacar.com
A 100 0/11 Allows all EU
67
Blocket
blocket.se
A 100 0/11 Allows all EU
68
Booking.com
booking.com
A 100 0/11 Allows all EU
69
Bring a Trailer
bringatrailer.com
A 100 0/11 Allows all US
70
CarDekho
cardekho.com
A 100 0/11 Allows all Asia
71
Care.com
care.com
A 100 0/11 Allows all US
72
CareerBuilder
careerbuilder.com
A 100 0/11 Allows all US
73
CarGurus
cargurus.com
A 100 0/11 Allows all US
74
Carousell
carousell.sg
A 100 0/11 Allows all Asia
75
Cars.com
cars.com
A 100 0/11 Allows all US
76
Catho
catho.com.br
A 100 0/11 Allows all LatAm
77
Centris
centris.ca
A 100 0/11 Allows all CA
78
Check24
check24.de
A 100 0/11 Allows all EU
79
Clutch
clutch.co
A 100 0/11 Allows all US
80
Compare the Market
comparethemarket.com
A 100 0/11 Allows all UK
81
Computrabajo
computrabajo.com
A 100 0/11 Allows all LatAm
82
ConsumerAffairs
consumeraffairs.com
A 100 0/11 Allows all US
83
Craigslist
craigslist.org
A 100 0/11 Allows all US
84
CV-Library
cv-library.co.uk
A 100 0/11 Allows all UK
85
Daft.ie
daft.ie
A 100 0/11 Allows all Ireland
86
Deliveroo
deliveroo.co.uk
A 100 0/11 Allows all UK
87
Despegar
despegar.com
A 100 0/11 Allows all LatAm
88
Dice
dice.com
A 100 0/11 Allows all US
89
Doctolib
doctolib.fr
A 100 0/11 Allows all EU
90
Domain
domain.com.au
A 100 0/11 Allows all Oceania
91
DoneDeal
donedeal.ie
A 100 0/11 Allows all Ireland
92
DoorDash
doordash.com
A 100 0/11 Allows all US
93
dubizzle
dubizzle.com
A 100 0/11 Allows all MENA-Africa
94
eDreams
edreams.com
A 100 0/11 Allows all EU
95
Expedia
expedia.com
A 100 0/11 Allows all US
96
FINN
finn.no
A 100 0/11 Allows all EU
97
Fiverr
fiverr.com
A 100 0/11 Allows all Global
98
Freelancer
freelancer.com
A 100 0/11 Allows all Oceania
99
Funda
funda.nl
A 100 0/11 Allows all EU
100
Gelbe Seiten
gelbeseiten.de
A 100 0/11 Allows all EU
Showing 51 to 100 of 209 results
Download full leaderboard (CSV)

All 209 graded sites from the latest weekly check, regardless of the filters above.

Methodology

For each site we fetch its public robots.txt and grade it against 11 of the major AI crawlers, including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site “blocks” a bot when its robots.txt has a Disallow rule that applies to that crawler’s user-agent token. The openness score is the share of graded bots that are allowed, rolled up into an A–F grade.

This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.

The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.

Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.

14-day Free Trial

Your Logs already store the data you need,
you just need a way to use it!

If you are interested in HoneyLog, you can try it with our Free Trial!

No credit card required