HoneyLog

Robots.txt best practices for ecommerce websites

How an online store sets up robots.txt: block faceted navigation, cart and checkout, keep products and categories crawlable, and welcome AI shopping assistants.

The strategy

Product and category pages, pagination included, are your revenue pages and must stay crawlable. The job is taming faceted navigation: filters and sort orders turn every category into thousands of URLs. Google's guidance is direct: if you don't need filtered URLs in search results, disallow them in robots.txt and keep the unfiltered listing crawlable.

  • Every product and category page is crawlable, pagination included.
  • Filter and sort parameters are blocked, unless a filtered page is a landing page you want to rank.
  • Cart, checkout, account and action URLs (add-to-cart, wishlist) are never crawled.
  • Product images stay crawlable for Google Images and the Search features that show them.
  • AI shopping and search assistants can read your catalog.

Keep crawlable

What you rank on: never let a rule touch it.

  • /product/ · /collections/ Your money pages.
  • ?page=2 (pagination) Don't block every parameter blindly: pagination must stay crawlable.
  • /*?products=all$ Google's own faceted-navigation example: allow the unfiltered "view all" listing.
  • Product images Googlebot-Image powers Google Images, Discover and every Search feature that shows images.
  • /blog/ · buying guides They rank for long-tail and comparison searches.

Keep out of the crawl

URLs with no search value that waste crawling.

  • /cart/ · /checkout/ No search value: every visit is wasted crawling.
  • /account/ Private, session-bound pages.
  • /*?*add-to-cart= Action URLs: crawling them adds items, not pages.
  • /*?*color= · /*?*size= Facets multiply every category into thousands of near-duplicate URLs.
  • /*?*sort= · /*?*price= Sorted and price-filtered copies of the same list.
  • /search · /*?q= Internal search results.

AI crawlers

For a store, AI assistants are a discovery channel: ChatGPT search, Claude, Perplexity and Amazon's assistants recommend products to buyers. Keep their search crawlers and user fetchers allowed. Block training crawlers only if your editorial content, not your product data, is the asset to protect.

  • Keep OAI-SearchBot, Claude-SearchBot, PerplexityBot and Amzn-SearchBot allowed: they feed AI and assistant search results. Amzn-SearchBot is not used for AI training.
  • A bot named in its own group ignores your "User-agent: *" rules, so repeat the cart and filter rules in that group.
  • Google's AdsBot ignores the "User-agent: *" group. If you run Google Ads, check you don't block your landing pages for it by name.
  • Whether your products appear in AI answers depends on crawlable product pages far more than on robots.txt tweaks.

Example robots.txt

A starting point, not a drop-in file: check every path against your own URLs before publishing it.

# Ecommerce robots.txt starting point for www.example.com
# Review every path against your real URL structure before publishing.
# Product pages, categories and their pagination stay crawlable.

User-agent: *
# Purchase funnel and account
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
# Action parameters (add-to-cart, wishlist...)
Disallow: /*?*add-to-cart=
Disallow: /*?*wishlist=
# Faceted navigation and sorting (adapt to your parameters)
Disallow: /*?*color=
Disallow: /*?*size=
Disallow: /*?*price=
Disallow: /*?*sort=
# Keep the unfiltered "view all" listing crawlable, if you have one
Allow: /*?products=all$
# Internal search
Disallow: /search
Disallow: /*?q=
# Tracking parameters
Disallow: /*?*utm_

# AI shopping assistants (OAI-SearchBot, Claude-SearchBot,
# PerplexityBot, Amzn-SearchBot) stay allowed on purpose:
# they recommend your products to buyers.

Sitemap: https://www.example.com/sitemap.xml

Common mistakes

  • Blocking every parameter (/*?), which also blocks pagination.
  • Blocking filtered URLs that already rank or have backlinks. Check them in Search Console first.
  • Answering empty filter combinations with a 200 page. Google recommends a 404 when a filter combination returns no products.
  • Using robots.txt for discontinued products: return 404/410, or redirect to the closest replacement.
  • Blocking theme CSS and JavaScript, or the product image folder.

Before you publish it

Some rules hold whatever your type of website: how crawlers match groups and paths, what status codes do to your file, and which crawlers you can actually block.

See what crawlers really do with your robots.txt

A robots.txt is a request, not a wall. HoneyLog reads your server logs, so you see which bots fetch your robots.txt, every version it went through, and which crawlers request pages it disallows for them.

How the top publishers configure robots.txt: the Robots.txt Leaderboard
14-day Free Trial

Try HoneyLog 14-day Free Trial

If you are interested in HoneyLog, you can try it with our Free Trial!

14-day Free Trial Contact Us
No credit card required