HoneyLog

Robots.txt best practices for forums and online communities

How forums and communities keep threads and their pagination crawlable, block profiles, posting actions and duplicates, and decide on AI training crawlers.

The strategy

Threads are the asset: user-generated discussions rank for countless specific questions, so topics, boards and their pagination stay crawlable. Block the noise: profiles that add nothing, login, posting actions, and sort or print variants. Forums are prime AI training material, so take an explicit position on training crawlers.

  • Threads and boards are crawlable, deep pagination included.
  • Login, settings and action URLs (post, reply, quote) stay out.
  • Sort, print and session variants of threads are blocked.
  • An explicit policy on AI training crawlers.

Keep crawlable

What you rank on: never let a rule touch it.

  • /t/ · /topic/ · /thread/ The threads themselves.
  • Board and category indexes How crawlers find threads.
  • Thread pagination Long threads continue on page 2 and beyond.
  • Curated /tags/ hubs Topic hubs can rank on their own.

Keep out of the crawl

URLs with no search value that waste crawling.

  • /u/ · /user/ · /member/ Check first: profiles that collect expert answers may rank.
  • /login/ · /register/ · /settings/ Auth and settings.
  • /messages/ · /notifications/ Private areas (they also need a login).
  • /post/ · /reply/ · /*?do= Action URLs.
  • /search/ · /*?q= Internal search results.
  • /*?sort= · /*?view=print · /*?sid= Sorted, print and session duplicates of threads.

AI crawlers

Forum content is prime training material. Blocking training crawlers protects it, and any licensing leverage you have, at no ranking cost; keeping AI search crawlers allowed keeps your threads cited in AI answers.

  • A common setup: block GPTBot, ClaudeBot, CCBot and meta-externalagent plus the shared __usage_ai-training token; keep AI search crawlers and user fetchers.
  • Some bots don't follow robots.txt at all: enforce a hard block at your CDN or firewall.
  • Never rely on robots.txt for privacy: private messages and settings need a login.

Example robots.txt

A starting point, not a drop-in file: check every path against your own URLs before publishing it.

# Community / Forum robots.txt starting point for www.example.com
# Review every path against your real URL structure before publishing.
# Threads, boards and their pagination stay crawlable.

User-agent: *
# User profiles (check first: keep them if they rank)
Disallow: /u/
Disallow: /user/
Disallow: /member/
# Auth and private areas (these must also be behind a login)
Disallow: /login/
Disallow: /register/
Disallow: /settings/
Disallow: /notifications/
Disallow: /messages/
# Posting / reply action URLs
Disallow: /post/
Disallow: /reply/
Disallow: /*?do=
# Internal search
Disallow: /search/
Disallow: /*?q=
# Sort / print / session duplicates of threads
Disallow: /*?sort=
Disallow: /*?view=print
Disallow: /*?sid=

# ---- AI training ------------------------------------------------
# Keeps your content out of AI model training. It does NOT affect
# your Google or Bing rankings, and AI search crawlers
# (OAI-SearchBot, Claude-SearchBot, PerplexityBot) stay allowed
# under "User-agent: *", so AI answers can still cite you.
# __usage_ai-training is the shared opt-out token (Bing from
# February 2027, Google from early 2027).
# Delete this block if you prefer to allow AI training.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: __usage_ai-training
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Common mistakes

  • Blocking thread pagination.
  • Blocking /u/ without checking whether profiles rank.
  • Trying to remove a page from Google with robots.txt: use noindex or delete it, then the removal tools.
  • Leaving ?sid= session variants crawlable.

Before you publish it

Some rules hold whatever your type of website: how crawlers match groups and paths, what status codes do to your file, and which crawlers you can actually block.

See what crawlers really do with your robots.txt

A robots.txt is a request, not a wall. HoneyLog reads your server logs, so you see which bots fetch your robots.txt, every version it went through, and which crawlers request pages it disallows for them.

How the top publishers configure robots.txt: the Robots.txt Leaderboard
14-day Free Trial

Try HoneyLog 14-day Free Trial

If you are interested in HoneyLog, you can try it with our Free Trial!

14-day Free Trial Contact Us
No credit card required