HoneyLog

Robots.txt best practices for business websites

A short, safe robots.txt for corporate and business websites: what to block, what never to block, noindex for thank-you pages and how to treat AI assistants.

The strategy

Default-open. Your service, case-study, team and contact pages are what you rank and get trusted on. Keep the file short: block admin, login and internal search, and never ship a staging "Disallow: /" to production. Google's crawl-budget advice targets sites with 10,000+ pages that change daily or a million+ pages, so most business sites have nothing to optimize there.

  • Every public page is crawlable.
  • Thank-you and confirmation pages stay out of search results with a noindex tag.
  • Admin, login and internal search stay out of the crawl.
  • No leftovers from staging or development.

Keep crawlable

What you rank on: never let a rule touch it.

  • / (default open) An empty robots.txt, or none at all, is valid: everything may be crawled.
  • /services/ · /case-studies/ The pages people find you through.
  • /about/ · /team/ Who you are: brand searches land here.
  • CSS / JS Especially on JavaScript-rendered sites, Google needs them to see the page.

Keep out of the crawl

URLs with no search value that waste crawling.

  • /admin/ · /wp-admin/ Back office. On WordPress keep /wp-admin/admin-ajax.php allowed.
  • /login/ Nothing to rank on.
  • /search/ · /*?s= Internal search results.
  • /*?*utm_ Campaign-tagged duplicates of your pages.

AI crawlers

For a business, AI assistants are free distribution: people ask them for the best agency, consultant or supplier for a need. Keep AI search crawlers and user fetchers allowed. Training crawlers are a policy choice with no effect on your rankings.

  • Keep OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed: they decide whether AI answers can cite you.
  • Blocking training crawlers changes nothing in Google or Bing. Lead-generation sites usually gain more from being known to AI models than from opting out.
  • Google-Extended controls Gemini training only: it doesn't affect Google Search.

Example robots.txt

A starting point, not a drop-in file: check every path against your own URLs before publishing it.

# Business / Corporate robots.txt starting point for www.example.com
# Review every path against your real URL structure before publishing.

User-agent: *
# Back office and login
Disallow: /admin/
Disallow: /login/
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# Internal search
Disallow: /search/
Disallow: /*?s=
# Tracking parameters
Disallow: /*?*utm_
# Thank-you pages: give them a noindex tag instead of a
# Disallow line, so search engines can see the tag.

Sitemap: https://www.example.com/sitemap.xml

Common mistakes

  • Shipping the staging site's "Disallow: /" to production.
  • "Hiding" private PDFs with robots.txt. The file is public and lists the paths, and blocked URLs can still appear in results. Use authentication.
  • Blocking the JavaScript bundles of a JavaScript-rendered site.
  • Disallowing a page that carries a noindex tag: Google can't see the tag, so the page can stay in results.

Before you publish it

Some rules hold whatever your type of website: how crawlers match groups and paths, what status codes do to your file, and which crawlers you can actually block.

See what crawlers really do with your robots.txt

A robots.txt is a request, not a wall. HoneyLog reads your server logs, so you see which bots fetch your robots.txt, every version it went through, and which crawlers request pages it disallows for them.

How the top publishers configure robots.txt: the Robots.txt Leaderboard
14-day Free Trial

Try HoneyLog 14-day Free Trial

If you are interested in HoneyLog, you can try it with our Free Trial!

14-day Free Trial Contact Us
No credit card required