Free public ranking

Marketplace & Directory Robots.txt Leaderboard

How the web's top marketplaces, classifieds & directories compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.

10%

of 209 ranked sites block GPTBot in robots.txt

209

sites graded, last checked Nov 11, 2026

94/100

average openness score (higher = more open to AI)

Share of sites blocking each AI crawler

Click a bot to filter the ranking to sites that block it.

Grade distribution

A = wide open to AI, F = walled off.

A
183 sites
B
14 sites
C
2 sites
D
9 sites
F
1 site
# AI crawlers Region
101
GetYourGuide
getyourguide.com
A 100 0/11 Allows all EU
102
Glovo
glovoapp.com
A 100 0/11 Allows all EU
103
Grubhub
grubhub.com
A 100 0/11 Allows all US
104
Gumtree
gumtree.com
A 100 0/11 Allows all UK
105
Haraj
haraj.com.sa
A 100 0/11 Allows all MENA-Africa
106
Healthgrades
healthgrades.com
A 100 0/11 Allows all US
107
Hemnet
hemnet.se
A 100 0/11 Allows all EU
108
Heureka
heureka.cz
A 100 0/11 Allows all EU
109
hipages
hipages.com.au
A 100 0/11 Allows all Oceania
110
Homes.com
homes.com
A 100 0/11 Allows all US
111
HomeStars
homestars.com
A 100 0/11 Allows all CA
112
Hopper
hopper.com
A 100 0/11 Allows all CA
113
Hostelworld
hostelworld.com
A 100 0/11 Allows all Global
114
HotPads
hotpads.com
A 100 0/11 Allows all US
115
Houzz
houzz.com
A 100 0/11 Allows all US
116
idealista
idealista.com
A 100 0/11 Allows all EU
117
idealo
idealo.de
A 100 0/11 Allows all EU
118
iFood
ifood.com.br
A 100 0/11 Allows all LatAm
119
Immobiliare.it
immobiliare.it
A 100 0/11 Allows all EU
120
ImmobilienScout24
immobilienscout24.de
A 100 0/11 Allows all EU
121
Immoweb
immoweb.be
A 100 0/11 Allows all EU
122
Indeed
indeed.com
A 100 0/11 Allows all US
123
InfoJobs
infojobs.net
A 100 0/11 Allows all EU
124
Inmuebles24
inmuebles24.com
A 100 0/11 Allows all LatAm
125
Jiji
jiji.ng
A 100 0/11 Allows all MENA-Africa
126
Just Eat
just-eat.co.uk
A 100 0/11 Allows all UK
127
Kayak
kayak.com
A 100 0/11 Allows all US
128
Kijiji
kijiji.ca
A 100 0/11 Allows all CA
129
Kleinanzeigen
kleinanzeigen.de
A 100 0/11 Allows all EU
130
Lieferando
lieferando.de
A 100 0/11 Allows all EU
131
LoopNet
loopnet.com
A 100 0/11 Allows all US
132
Magicbricks
magicbricks.com
A 100 0/11 Allows all Asia
133
Malt
malt.fr
A 100 0/11 Allows all EU
134
Marktplaats
marktplaats.nl
A 100 0/11 Allows all EU
135
mobile.de
mobile.de
A 100 0/11 Allows all EU
136
momondo
momondo.com
A 100 0/11 Allows all EU
137
MoneySuperMarket
moneysupermarket.com
A 100 0/11 Allows all UK
138
Monster
monster.com
A 100 0/11 Allows all US
139
Mudah.my
mudah.my
A 100 0/11 Allows all Asia
140
MyBuilder
mybuilder.com
A 100 0/11 Allows all UK
141
Naukri
naukri.com
A 100 0/11 Allows all Asia
142
OfferUp
offerup.com
A 100 0/11 Allows all US
143
OLX Brasil
olx.com.br
A 100 0/11 Allows all LatAm
144
OLX
olx.pl
A 100 0/11 Allows all EU
145
OpenSooq
opensooq.com
A 100 0/11 Allows all MENA-Africa
146
Orbitz
orbitz.com
A 100 0/11 Allows all US
147
Otodom
otodom.pl
A 100 0/11 Allows all EU
148
PagineGialle
paginegialle.it
A 100 0/11 Allows all EU
149
PedidosYa
pedidosya.com
A 100 0/11 Allows all LatAm
150
PeoplePerHour
peopleperhour.com
A 100 0/11 Allows all UK
Showing 101 to 150 of 209 results
Download full leaderboard (CSV)

All 209 graded sites from the latest weekly check, regardless of the filters above.

Methodology

For each site we fetch its public robots.txt and grade it against 11 of the major AI crawlers, including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site “blocks” a bot when its robots.txt has a Disallow rule that applies to that crawler’s user-agent token. The openness score is the share of graded bots that are allowed, rolled up into an A–F grade.

This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.

The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.

Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.

14-day Free Trial

Your Logs already store the data you need,
you just need a way to use it!

If you are interested in HoneyLog, you can try it with our Free Trial!

No credit card required