Free public ranking

Marketplace & Directory Robots.txt Leaderboard

How the web's top marketplaces, classifieds & directories compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.

10%

of 209 ranked sites block GPTBot in robots.txt

209

sites graded, last checked Nov 11, 2026

94/100

average openness score (higher = more open to AI)

Share of sites blocking each AI crawler

Click a bot to filter the ranking to sites that block it.

Grade distribution

A = wide open to AI, F = walled off.

A
183 sites
B
14 sites
C
2 sites
D
9 sites
F
1 site
# AI crawlers Region
151
Portal Inmobiliario
portalinmobiliario.com
A 100 0/11 Allows all LatAm
152
Practo
practo.com
A 100 0/11 Allows all Asia
153
Pracuj.pl
pracuj.pl
A 100 0/11 Allows all EU
154
Priceline
priceline.com
A 100 0/11 Allows all US
155
PriceRunner
pricerunner.com
A 100 0/11 Allows all EU
156
Property24
property24.com
A 100 0/11 Allows all MENA-Africa
157
Property Finder
propertyfinder.ae
A 100 0/11 Allows all MENA-Africa
158
PropertyGuru
propertyguru.com.sg
A 100 0/11 Allows all Asia
159
Rappi
rappi.com
A 100 0/11 Allows all LatAm
160
RateMDs
ratemds.com
A 100 0/11 Allows all CA
161
Reclame Aqui
reclameaqui.com.br
A 100 0/11 Allows all LatAm
162
Redfin
redfin.com
A 100 0/11 Allows all US
163
Reed
reed.co.uk
A 100 0/11 Allows all UK
164
Resy
resy.com
A 100 0/11 Allows all US
165
Rover
rover.com
A 100 0/11 Allows all US
166
sahibinden.com
sahibinden.com
A 100 0/11 Allows all MENA-Africa
167
SeatGeek
seatgeek.com
A 100 0/11 Allows all US
168
SEEK
seek.com.au
A 100 0/11 Allows all Oceania
169
SeLoger
seloger.com
A 100 0/11 Allows all EU
170
SkipTheDishes
skipthedishes.com
A 100 0/11 Allows all CA
171
StepStone
stepstone.de
A 100 0/11 Allows all EU
172
StubHub
stubhub.com
A 100 0/11 Allows all US
173
Subito
subito.it
A 100 0/11 Allows all EU
174
SUUMO
suumo.jp
A 100 0/11 Allows all Asia
175
Swiggy
swiggy.com
A 100 0/11 Allows all Asia
176
talabat
talabat.com
A 100 0/11 Allows all MENA-Africa
177
Taskrabbit
taskrabbit.com
A 100 0/11 Allows all US
178
TheFork
thefork.com
A 100 0/11 Allows all EU
179
Thumbtack
thumbtack.com
A 100 0/11 Allows all US
180
Toptal
toptal.com
A 100 0/11 Allows all Global
181
Trainline
trainline.com
A 100 0/11 Allows all UK
182
Traveloka
traveloka.com
A 100 0/11 Allows all Asia
183
Treatwell
treatwell.co.uk
A 100 0/11 Allows all UK
184
Trip.com
trip.com
A 100 0/11 Allows all Asia
185
trivago
trivago.com
A 100 0/11 Allows all EU
186
Trovaprezzi
trovaprezzi.it
A 100 0/11 Allows all EU
187
TrueCar
truecar.com
A 100 0/11 Allows all US
188
Trulia
trulia.com
A 100 0/11 Allows all US
189
Turo
turo.com
A 100 0/11 Allows all US
190
Uber Eats
ubereats.com
A 100 0/11 Allows all US
191
Upwork
upwork.com
A 100 0/11 Allows all US
192
Urban Company
urbancompany.com
A 100 0/11 Allows all Asia
193
Viator
viator.com
A 100 0/11 Allows all US
194
Vivid Seats
vividseats.com
A 100 0/11 Allows all US
195
Vrbo
vrbo.com
A 100 0/11 Allows all US
196
Webmotors
webmotors.com.br
A 100 0/11 Allows all LatAm
197
Welcome to the Jungle
welcometothejungle.com
A 100 0/11 Allows all EU
198
We Work Remotely
weworkremotely.com
A 100 0/11 Allows all Global
199
willhaben
willhaben.at
A 100 0/11 Allows all EU
200
Wolt
wolt.com
A 100 0/11 Allows all EU
Showing 151 to 200 of 209 results
Download full leaderboard (CSV)

All 209 graded sites from the latest weekly check, regardless of the filters above.

Methodology

For each site we fetch its public robots.txt and grade it against 11 of the major AI crawlers, including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site “blocks” a bot when its robots.txt has a Disallow rule that applies to that crawler’s user-agent token. The openness score is the share of graded bots that are allowed, rolled up into an A–F grade.

This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.

The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.

Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.

14-day Free Trial

Your Logs already store the data you need,
you just need a way to use it!

If you are interested in HoneyLog, you can try it with our Free Trial!

No credit card required