HoneyLog

Robots.txt best practices for news and media publishers

What news and media publishers should block in robots.txt, what to keep crawlable, and how to treat AI training and AI search crawlers, with an example file.

The strategy

Your articles are the product, so keep robots.txt wide open for search engines: articles, section and author pages, and the pagination that leads to older stories. The decision that matters is AI. AI training crawlers and AI search crawlers are different bots, so you can keep your content out of model training without disappearing from AI answers.

  • Every article, section and author page is crawlable and listed in your sitemaps.
  • Section and archive pagination stays open: it is how crawlers reach older articles.
  • Internal search, print, preview and tracking URLs don't eat your crawl budget.
  • A deliberate policy for AI training crawlers, separate from AI search.

Keep crawlable

What you rank on: never let a rule touch it.

  • / (articles) Article URLs are what you rank on. Never let a broad Disallow pattern touch them.
  • /category/…/page/2/ Paginated archives are how crawlers discover older articles.
  • /author/ Author pages show who writes your journalism.
  • CSS / JS / images Google renders pages to understand them. Blocking the files a page needs changes what Google sees.
  • Paywalled articles Don't block them. Let Googlebot crawl them and mark the paywall with paywalled-content structured data.

Keep out of the crawl

URLs with no search value that waste crawling.

  • /wp-admin/ The WordPress admin area. Keep /wp-admin/admin-ajax.php allowed, as WordPress's own default file does.
  • /search/ · /*?s= Internal search results: an endless supply of thin URLs.
  • /print/ · /*?print= Print versions duplicate the article.
  • /*?preview= Draft previews.
  • /*?*utm_ · /*?*replytocom= Tracking and comment-reply duplicates of the same article.

AI crawlers

Blocking AI training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent) and the Google-Extended and Applebot-Extended opt-out tokens doesn't change your Google or Bing rankings. AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are a different matter: they are how AI answers cite you, so blocking them costs visibility. Among the top 100 US and UK news sites, 75% block CCBot and 62% block GPTBot (BuzzStream).

  • A common setup: block the training crawlers, keep AI search crawlers and the fetchers that act on a user's request.
  • Google-Extended and Applebot-Extended are opt-out tokens, not crawlers: they never visit your site, and Google-Extended doesn't affect your inclusion in Google Search.
  • New in October 2026: the shared __usage_ai-training token opts you out of AI training with every operator that supports it. Bing honours it from 1 February 2027, Google aims for early 2027. Keep the named bots as well until then.
  • AI platforms request far more pages than they send visitors back (Cloudflare's crawl-to-refer data). Weigh what each bot gives you in return.
  • Fetchers acting on a user's request (ChatGPT-User, Perplexity-User, meta-externalfetcher) may not follow robots.txt. For a hard block, use your CDN or firewall.

Source of the figures: BuzzStream: AI crawler blocking on the top 100 US and UK news sites

Example robots.txt

A starting point, not a drop-in file: check every path against your own URLs before publishing it.

# Publisher / Media robots.txt starting point for www.example.com
# Review every path against your real URL structure before publishing.

User-agent: *
# Admin area (WordPress): keep admin-ajax.php reachable
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# Internal search results
Disallow: /search/
Disallow: /*?s=
# Print, preview and comment-reply duplicates
Disallow: /print/
Disallow: /*?print=
Disallow: /*?preview=
Disallow: /*?*replytocom=
# Tracking parameters
Disallow: /*?*utm_

# ---- AI training ------------------------------------------------
# Keeps your content out of AI model training. It does NOT affect
# your Google or Bing rankings, and AI search crawlers
# (OAI-SearchBot, Claude-SearchBot, PerplexityBot) stay allowed
# under "User-agent: *", so AI answers can still cite you.
# __usage_ai-training is the shared opt-out token (Bing from
# February 2027, Google from early 2027).
# Delete this block if you prefer to allow AI training.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: __usage_ai-training
Disallow: /

Sitemap: https://www.example.com/sitemap_index.xml
# Google News sitemap (articles from the last two days), if you have one
Sitemap: https://www.example.com/news-sitemap.xml

Common mistakes

  • Blocking /wp-content/ or /wp-includes/ wholesale: themes, images and scripts live there.
  • Blocking date or category archives that carry pagination.
  • Disallowing an article to get it out of Google. A blocked URL can still show up in results: use noindex and keep the page crawlable, or return 404/410.
  • Blocking Googlebot to leave Google News: that removes you from Google Search too. Block Googlebot-News instead.
  • Copying an AI blocklist found online: many also block AI search crawlers and cost you AI citations.

Before you publish it

Some rules hold whatever your type of website: how crawlers match groups and paths, what status codes do to your file, and which crawlers you can actually block.

See what crawlers really do with your robots.txt

A robots.txt is a request, not a wall. HoneyLog reads your server logs, so you see which bots fetch your robots.txt, every version it went through, and which crawlers request pages it disallows for them.

How the top publishers configure robots.txt: the Robots.txt Leaderboard
14-day Free Trial

Try HoneyLog 14-day Free Trial

If you are interested in HoneyLog, you can try it with our Free Trial!

14-day Free Trial Contact Us
No credit card required