HoneyLog

Robots.txt best practices for SaaS websites and web apps

How SaaS companies set up robots.txt: an open marketing site and docs, the app closed on its own host, login flows out of the crawl, docs open to AI assistants.

The strategy

Split by host. The marketing site (home, features, pricing, docs, blog) is fully crawlable: docs especially, since they answer long-tail searches and get cited by AI assistants. The app itself stays out. A robots.txt applies only to the host serving it, so if the app lives on app.www.example.com, that host needs its own file with "Disallow: /". Real protection comes from login, not robots.txt.

  • Marketing pages, pricing, docs, changelog and blog are crawlable.
  • The app is never crawled: its own robots.txt when it runs on a subdomain.
  • One-off auth URLs (password reset, invites, verification) stay out of the crawl.
  • Docs are open to AI search crawlers.

Keep crawlable

What you rank on: never let a rule touch it.

  • /pricing/ · /features/ The pages buyers compare.
  • /docs/ · help center Never block them: they answer the long tail and get you cited.
  • /blog/ · /customers/ Content and proof.
  • /integrations/ People search for "your product + the tool they use".
  • /signup Keep it crawlable: people search for it.

Keep out of the crawl

URLs with no search value that waste crawling.

  • /app/ · /dashboard/ The application, if it runs on the marketing host.
  • /login/ · /reset-password/ One-off auth states.
  • /api/ Endpoints, unless public pages need them to render.
  • /billing/ · /settings/ Account areas.
  • /*?q= · /*?*utm_ Internal search and tracking duplicates.

AI crawlers

AI assistants answer your future customers' questions. Keep AI search crawlers and user fetchers allowed, and your docs become citations. Most SaaS companies leave training crawlers open too: it is a data-usage decision, not a ranking one.

  • Keep docs and marketing pages open to OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user fetchers.
  • App protection is login plus noindex (or X-Robots-Tag) on anything public by mistake, never robots.txt alone.
  • Blocking training crawlers has no ranking effect: decide on data use alone.

Example robots.txt

A starting point, not a drop-in file: check every path against your own URLs before publishing it.

# SaaS robots.txt starting point for the MARKETING host (www.example.com)
# Review every path against your real URL structure before publishing.
# If the app runs on its own subdomain (e.g. app.www.example.com), that host
# needs its OWN robots.txt containing just:
#   User-agent: *
#   Disallow: /

User-agent: *
# The application (if served from this same host)
Disallow: /app/
Disallow: /dashboard/
# One-off auth states (keep /signup crawlable)
Disallow: /login/
Disallow: /reset-password/
Disallow: /invite/
Disallow: /verify/
# Account, billing and onboarding
Disallow: /billing/
Disallow: /settings/
Disallow: /onboarding/
# API endpoints
Disallow: /api/
# Internal search and tracking
Disallow: /*?q=
Disallow: /*?*utm_

# /docs/ and the marketing pages stay open on purpose: AI
# assistants citing your documentation is free acquisition.

Sitemap: https://www.example.com/sitemap.xml

Common mistakes

  • Blocking /docs/ or the help center.
  • Relying on robots.txt to protect logged-in URLs: the file is public and lists the paths.
  • Shipping the app's "Disallow: /" to the marketing host, or forgetting the app subdomain's own file.
  • Blocking scripts or API endpoints that public pages need to render.

Before you publish it

Some rules hold whatever your type of website: how crawlers match groups and paths, what status codes do to your file, and which crawlers you can actually block.

See what crawlers really do with your robots.txt

A robots.txt is a request, not a wall. HoneyLog reads your server logs, so you see which bots fetch your robots.txt, every version it went through, and which crawlers request pages it disallows for them.

How the top publishers configure robots.txt: the Robots.txt Leaderboard
14-day Free Trial

Try HoneyLog 14-day Free Trial

If you are interested in HoneyLog, you can try it with our Free Trial!

14-day Free Trial Contact Us
No credit card required