Robots.txt best practices for marketplaces and directories
How marketplaces and directories keep filter, sort and search URLs out of the crawl while listings, categories and location pages stay crawlable.
The strategy
Listings and category or location pages are what you rank on, and crawl budget is your scarcest resource: filter, sort and radius combinations create an almost infinite URL space. Disciplined parameter blocking matters more here than anywhere else. Expired listings get a 404/410 or a redirect, never a robots.txt rule.
- Listings and category and location pages are crawlable, pagination included.
- Filter, sort and radius combinations are blocked.
- Posting flows, dashboards and messages stay private, behind login.
- Sitemaps list live listings with honest lastmod dates.
Keep crawlable
What you rank on: never let a rule touch it.
-
/listing/ · /item/Your inventory. -
/cars/ · /rome/apartments/Category and location landing pages rank for the head searches. -
?page=2 (pagination)How crawlers reach deeper listings. -
/seller/ storefrontsOften worth indexing: check before blocking.
Keep out of the crawl
URLs with no search value that waste crawling.
-
/*?*sort= · /*?*view=The same results in another order or layout. -
/*?*price_min= · /*?*radius=Filter combinations: the near-infinite URL space Google warns about. -
/search/ · /*?q=Internal search results. -
/post-ad/ · /submit/Posting flows. -
/account/ · /messages/Private areas (they also need a login). -
/api/ · /ajax/Endpoints, unless your public pages need them to render.
AI crawlers
Marketplaces gain from AI visibility: people ask assistants where to find a product, a service or a place. Keep AI search crawlers and user fetchers allowed. Whether to block training crawlers depends on whether your listing data is the asset that sets you apart.
- Keep OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed.
- If your data is your moat, block the training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent) and add the shared __usage_ai-training token.
- Heavy scraping comes from bots that ignore robots.txt: rate limiting and firewall rules are the real defense.
Example robots.txt
A starting point, not a drop-in file: check every path against your own URLs before publishing it.
# Marketplace / Directory robots.txt starting point for www.example.com # Review every path against your real URL structure before publishing. # Listings, category/location pages and their pagination stay crawlable. # Expired listings: use 404/410 or redirects, never robots.txt. User-agent: * # User areas, posting flows and messages Disallow: /account/ Disallow: /dashboard/ Disallow: /messages/ Disallow: /post-ad/ Disallow: /submit/ # Internal search Disallow: /search/ Disallow: /*?q= # Filter / sort combinations (adapt to your parameters) Disallow: /*?*sort= Disallow: /*?*view= Disallow: /*?*price_min= Disallow: /*?*price_max= Disallow: /*?*radius= # AJAX / API endpoints Disallow: /api/ Disallow: /ajax/ # Tracking parameters Disallow: /*?*utm_ Sitemap: https://www.example.com/sitemap_index.xml
Common mistakes
- Using robots.txt for expired listings: return 404/410 or redirect, so they leave the crawl queue faster.
- Blocking category pagination.
- Disallowing location pages because their URLs "look like parameters".
- Letting bots crawl millions of filter combinations, and answering empty ones with 200 instead of 404.
Before you publish it
Some rules hold whatever your type of website: how crawlers match groups and paths, what status codes do to your file, and which crawlers you can actually block.
See what crawlers really do with your robots.txt
A robots.txt is a request, not a wall. HoneyLog reads your server logs, so you see which bots fetch your robots.txt, every version it went through, and which crawlers request pages it disallows for them.
How the top publishers configure robots.txt: the Robots.txt LeaderboardTry HoneyLog 14-day Free Trial
If you are interested in HoneyLog, you can try it with our Free Trial!