HoneyLog
Free guide

Robots.txt best practices for every type of website

What to block, what to keep crawlable and how to treat AI crawlers, for your kind of website. Pick your type below: each one comes with the reasoning, the common mistakes and an example robots.txt to adapt.

Pick your type of website

Know your crawlers

The user-agent tokens that matter in 2026, grouped by what blocking them actually does.

Search and social: never block

Blocking them removes you from search results or breaks link previews.

  • Googlebot Google Search, including AI Overviews and AI Mode.
  • Bingbot Bing search.
  • Applebot Siri, Spotlight and Safari search. Follows your Googlebot rules when it isn't named.
  • DuckDuckBot DuckDuckGo.
  • facebookexternalhit Link previews on Meta apps.

AI training: your choice

Blocking them keeps your content out of future AI models, with no effect on Google or Bing rankings.

  • GPTBot OpenAI model training.
  • ClaudeBot Anthropic model training. Honours Crawl-delay.
  • CCBot Common Crawl's open web archive, widely used to train AI models.
  • meta-externalagent Meta, including AI model training.
  • Google-Extended Opt-out token, never crawls: Gemini training. No effect on Google Search.
  • Applebot-Extended Opt-out token, never crawls: Apple model training.
  • __usage_ai-training Shared opt-out token: Bing from February 2027, Google from early 2027.

AI search: usually keep allowed

They decide whether AI answers can cite and link you. Blocking them doesn't keep you out of training.

  • OAI-SearchBot ChatGPT search. Changes take about 24 hours to apply.
  • Claude-SearchBot Search results in Claude.
  • PerplexityBot Perplexity search. Not used for AI model training.
  • meta-webindexer Meta AI search and citations.
  • Amzn-SearchBot Search in Amazon products. Not used for AI training.

On a user's request: robots.txt may not apply

They fetch a page because someone asked about it. Their operators say they may not follow robots.txt.

  • ChatGPT-User OpenAI: robots.txt rules may not apply.
  • Perplexity-User Generally ignores robots.txt.
  • meta-externalfetcher May bypass robots.txt.
  • Google-Agent Google's agents acting for a user.
  • Amzn-User May not follow all directives.
  • Claude-User Claude fetching a page a user asked about.

SEO tools and scrapers: block if they bother you

Heavy crawlers feeding SEO and competitor-research tools. No ranking impact either way.

  • AhrefsBot Ahrefs.
  • SemrushBot Semrush.
  • MJ12bot Majestic.
  • DotBot Moz.
  • DataForSeoBot DataForSEO.

Rules for every website

How crawlers read the file, whatever your type of site.

One file per host

robots.txt applies only to the protocol, host and port that serve it. www.example.com, example.com, shop.example.com and app.example.com each need their own file at /robots.txt.

Google reads four fields

user-agent, allow, disallow and sitemap. Google ignores crawl-delay, and hasn't supported noindex lines in robots.txt since 2019: put noindex in a meta tag or an X-Robots-Tag header.

The longest rule wins

When an Allow and a Disallow both match a URL, the rule with the longest path wins. On a tie, Google picks the least restrictive one.

A named bot ignores the * group

A crawler follows only the most specific group that names it, and groups are never combined. Add a Googlebot group and Googlebot stops reading your "User-agent: *" rules, so repeat them there.

Wildcards, case and slashes

* matches any sequence of characters and $ marks the end of the URL. Paths are case-sensitive and rules match prefixes: /Admin/ doesn't block /admin/, and /folder/ doesn't block /folder.

Blocking is not removing

A disallowed URL can still appear in search results, without a description. To keep a page out, let it be crawled and use noindex, or return 404/410. Never disallow a page that carries noindex: Google will never see the tag.

robots.txt is not security

Anyone can read the file, and it lists the paths you name. Protect private areas with a login.

Status codes matter

A 4xx response (except 429) counts as "no robots.txt": everything may be crawled. A 5xx makes Google pause crawling for 12 hours, then use its last cached copy for up to 30 days. Google caches the file for up to 24 hours.

500 KiB limit

Google ignores everything after the first 500 KiB of the file.

Sitemap lines

Use full URLs, https:// included. List as many sitemaps as you need; they can live on another host.

Slow crawlers with 429 or 503

To reduce crawling for a while, return 429, 500 or 503. A 403 or 404 tells Google the content is gone.

llms.txt is not robots.txt

Google Search ignores llms.txt and similar AI files: robots.txt and page-level tags remain the controls that crawlers honour.

Sources

Every tip was checked against the documentation of Google, Bing and the AI crawler operators. The main references:

See what crawlers really do with your robots.txt

A robots.txt is a request, not a wall. HoneyLog reads your server logs, so you see which bots fetch your robots.txt, every version it went through, and which crawlers request pages it disallows for them.

How the top publishers configure robots.txt: the Robots.txt Leaderboard
14-day Free Trial

Try HoneyLog 14-day Free Trial

If you are interested in HoneyLog, you can try it with our Free Trial!

14-day Free Trial Contact Us
No credit card required