Marketplace & Directory Robots.txt Leaderboard
How the web's top marketplaces, classifieds & directories compare on openness to AI crawlers. Graded straight from each site's robots.txt against the major AI bots (GPTBot, ClaudeBot, Google-Extended and more). Use this page to see how your site compares, and to check if your robots.txt is blocking or allowing the AI bots you want to allow or block.
10%
of 209 ranked sites block GPTBot in robots.txt
209
sites graded, last checked Nov 11, 2026
94/100
average openness score (higher = more open to AI)
Share of sites blocking each AI crawler
Click a bot to filter the ranking to sites that block it.
Grade distribution
A = wide open to AI, F = walled off.
- A
- 183 sites
- B
- 14 sites
- C
- 2 sites
- D
- 9 sites
- F
- 1 site
| # | AI crawlers | Region | ||||
|---|---|---|---|---|---|---|
| 101 |
GetYourGuide
getyourguide.com
|
A | 100 | 0/11 | Allows all | EU |
| 102 |
Glovo
glovoapp.com
|
A | 100 | 0/11 | Allows all | EU |
| 103 |
Grubhub
grubhub.com
|
A | 100 | 0/11 | Allows all | US |
| 104 |
Gumtree
gumtree.com
|
A | 100 | 0/11 | Allows all | UK |
| 105 |
Haraj
haraj.com.sa
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 106 |
Healthgrades
healthgrades.com
|
A | 100 | 0/11 | Allows all | US |
| 107 |
Hemnet
hemnet.se
|
A | 100 | 0/11 | Allows all | EU |
| 108 |
Heureka
heureka.cz
|
A | 100 | 0/11 | Allows all | EU |
| 109 |
hipages
hipages.com.au
|
A | 100 | 0/11 | Allows all | Oceania |
| 110 |
Homes.com
homes.com
|
A | 100 | 0/11 | Allows all | US |
| 111 |
HomeStars
homestars.com
|
A | 100 | 0/11 | Allows all | CA |
| 112 |
Hopper
hopper.com
|
A | 100 | 0/11 | Allows all | CA |
| 113 |
Hostelworld
hostelworld.com
|
A | 100 | 0/11 | Allows all | Global |
| 114 |
HotPads
hotpads.com
|
A | 100 | 0/11 | Allows all | US |
| 115 |
Houzz
houzz.com
|
A | 100 | 0/11 | Allows all | US |
| 116 |
idealista
idealista.com
|
A | 100 | 0/11 | Allows all | EU |
| 117 |
idealo
idealo.de
|
A | 100 | 0/11 | Allows all | EU |
| 118 |
iFood
ifood.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 119 |
Immobiliare.it
immobiliare.it
|
A | 100 | 0/11 | Allows all | EU |
| 120 |
ImmobilienScout24
immobilienscout24.de
|
A | 100 | 0/11 | Allows all | EU |
| 121 |
Immoweb
immoweb.be
|
A | 100 | 0/11 | Allows all | EU |
| 122 |
Indeed
indeed.com
|
A | 100 | 0/11 | Allows all | US |
| 123 |
InfoJobs
infojobs.net
|
A | 100 | 0/11 | Allows all | EU |
| 124 |
Inmuebles24
inmuebles24.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 125 |
Jiji
jiji.ng
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 126 |
Just Eat
just-eat.co.uk
|
A | 100 | 0/11 | Allows all | UK |
| 127 |
Kayak
kayak.com
|
A | 100 | 0/11 | Allows all | US |
| 128 |
Kijiji
kijiji.ca
|
A | 100 | 0/11 | Allows all | CA |
| 129 |
Kleinanzeigen
kleinanzeigen.de
|
A | 100 | 0/11 | Allows all | EU |
| 130 |
Lieferando
lieferando.de
|
A | 100 | 0/11 | Allows all | EU |
| 131 |
LoopNet
loopnet.com
|
A | 100 | 0/11 | Allows all | US |
| 132 |
Magicbricks
magicbricks.com
|
A | 100 | 0/11 | Allows all | Asia |
| 133 |
Malt
malt.fr
|
A | 100 | 0/11 | Allows all | EU |
| 134 |
Marktplaats
marktplaats.nl
|
A | 100 | 0/11 | Allows all | EU |
| 135 |
mobile.de
mobile.de
|
A | 100 | 0/11 | Allows all | EU |
| 136 |
momondo
momondo.com
|
A | 100 | 0/11 | Allows all | EU |
| 137 |
MoneySuperMarket
moneysupermarket.com
|
A | 100 | 0/11 | Allows all | UK |
| 138 |
Monster
monster.com
|
A | 100 | 0/11 | Allows all | US |
| 139 |
Mudah.my
mudah.my
|
A | 100 | 0/11 | Allows all | Asia |
| 140 |
MyBuilder
mybuilder.com
|
A | 100 | 0/11 | Allows all | UK |
| 141 |
Naukri
naukri.com
|
A | 100 | 0/11 | Allows all | Asia |
| 142 |
OfferUp
offerup.com
|
A | 100 | 0/11 | Allows all | US |
| 143 |
OLX Brasil
olx.com.br
|
A | 100 | 0/11 | Allows all | LatAm |
| 144 |
OLX
olx.pl
|
A | 100 | 0/11 | Allows all | EU |
| 145 |
OpenSooq
opensooq.com
|
A | 100 | 0/11 | Allows all | MENA-Africa |
| 146 |
Orbitz
orbitz.com
|
A | 100 | 0/11 | Allows all | US |
| 147 |
Otodom
otodom.pl
|
A | 100 | 0/11 | Allows all | EU |
| 148 |
PagineGialle
paginegialle.it
|
A | 100 | 0/11 | Allows all | EU |
| 149 |
PedidosYa
pedidosya.com
|
A | 100 | 0/11 | Allows all | LatAm |
| 150 |
PeoplePerHour
peopleperhour.com
|
A | 100 | 0/11 | Allows all | UK |
All 209 graded sites from the latest weekly check, regardless of the filters above.
Methodology
For each site we fetch its public robots.txt and grade it against
11 of the major AI crawlers,
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot. A site
“blocks” a bot when its robots.txt has a Disallow rule that applies
to that crawler’s user-agent token. The openness score is the share of
graded bots that are allowed, rolled up into an A–F grade.
This measures declared access in robots.txt, not enforcement. robots.txt is a request, not a firewall: a well-behaved crawler obeys it, but it does not technically stop anything. A site can also block AI bots at the edge (WAF, rate limits, IP blocks) in ways this ranking cannot see. Treat the grades as a signal of intent, not a guarantee.
The trend combines two sources: current grades come from a live scan of each site’s robots.txt, re-run every week, while historical snapshots are reconstructed from the Internet Archive (Wayback Machine): we grade the nearest archived robots.txt at or before each quarter. Historical points therefore reflect the best available capture, not necessarily the exact first of the period.
Last checked: Nov 11, 2026, refreshed every week. Reads stored scan data only: no live requests are made when you load this page.
Your Logs already store the data you need,
you just need a way to use it!
If you are interested in HoneyLog, you can try it with our Free Trial!