HoneyLog

Robots.txt best practices for any website

A safe default robots.txt for any kind of website: what to block, what never to block, how to treat AI crawlers, and the mistakes that cost search traffic.

The strategy

Default-open. Block only what has no value on any site: admin and login, cart and checkout if you have them, internal search, API endpoints and tracking parameters. An empty robots.txt is valid, and safer than a wrong one.

  • Everything public is crawlable by default.
  • Admin, login and internal search stay out.
  • CSS, JavaScript and images are never blocked.
  • noindex for keeping pages out of results, login for privacy.

Keep crawlable

What you rank on: never let a rule touch it.

  • / (default open) An empty robots.txt, or none at all, means everything may be crawled.
  • CSS / JS / images Google needs them to render and understand your pages.
  • Pagination How crawlers reach deeper content.

Keep out of the crawl

URLs with no search value that waste crawling.

  • /admin/ · /login/ Back office and auth.
  • /cart/ · /checkout/ · /account/ Commerce funnels, if you have them.
  • /search/ · /*?s= · /*?q= Internal search: the most common crawl sink.
  • /api/ Endpoints, unless public pages need them to render.
  • /*?*utm_ Tracking duplicates.

AI crawlers

Keep AI search crawlers and user fetchers allowed: they are how AI assistants find and cite you. Training crawlers are a policy choice with no effect on your rankings.

  • AI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot. User fetchers: ChatGPT-User, Claude-User, Perplexity-User.
  • Training only: GPTBot, ClaudeBot, CCBot, meta-externalagent, plus the Google-Extended and Applebot-Extended opt-out tokens and the shared __usage_ai-training token.
  • Put your path rules in "User-agent: *": a bot with its own group ignores the * rules.

Example robots.txt

A starting point, not a drop-in file: check every path against your own URLs before publishing it.

# Generic robots.txt starting point for www.example.com
# Review every path against your real URL structure before publishing.
# Default-open: block only clearly worthless areas. An empty robots.txt
# is valid too, and safer than a wrong one.

User-agent: *
# Back office and login
Disallow: /admin/
Disallow: /login/
# Commerce funnels (remove if you have none)
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
# API endpoints
Disallow: /api/
# Internal search
Disallow: /search/
Disallow: /*?s=
Disallow: /*?q=
# Tracking parameters
Disallow: /*?*utm_

Sitemap: https://www.example.com/sitemap.xml

Common mistakes

  • Copying another site's robots.txt.
  • Blocking CSS or JavaScript "for cleanliness".
  • Using robots.txt to remove pages from Google or to protect private ones.
  • A stray "Disallow: /" left over from staging.

Before you publish it

Some rules hold whatever your type of website: how crawlers match groups and paths, what status codes do to your file, and which crawlers you can actually block.

See what crawlers really do with your robots.txt

A robots.txt is a request, not a wall. HoneyLog reads your server logs, so you see which bots fetch your robots.txt, every version it went through, and which crawlers request pages it disallows for them.

How the top publishers configure robots.txt: the Robots.txt Leaderboard
14-day Free Trial

Try HoneyLog 14-day Free Trial

If you are interested in HoneyLog, you can try it with our Free Trial!

14-day Free Trial Contact Us
No credit card required