HoneyLog

Robots.txt best practices for education and course websites

How course platforms and schools keep their catalog crawlable and the learning environment out of the crawl, and why robots.txt is never a paywall.

The strategy

Sell publicly, teach privately. The course catalog, course landing pages, instructor pages and free previews are crawlable. The learning environment (players, quizzes, grades, student areas) is blocked. Paid content sits behind a login, because robots.txt is not a paywall.

  • The catalog and course landing pages are crawlable and in the sitemap.
  • Player, quiz and grade URLs stay out: they are session-bound and endless.
  • Student areas stay private.
  • Paid content is protected by login, not by robots.txt.

Keep crawlable

What you rank on: never let a rule touch it.

  • /courses/ · /course/{slug}/ The pages you sell from.
  • /instructors/ Who teaches: people search for them.
  • Free previews / syllabus Let search engines see what the course covers.
  • /blog/ · /resources/ Content that brings new learners.

Keep out of the crawl

URLs with no search value that waste crawling.

  • /learn/ · /lesson/ · /player/ The learning environment.
  • /my/ · /grades/ · /profile/ Student areas.
  • /login/ · /enrol/ Auth and enrollment flows.
  • /cart/ · /checkout/ The purchase funnel.
  • /search/ · /*?q= Internal search results.
  • /certificate/*?id= Per-student certificates.

AI crawlers

People ask AI assistants which course to take, so keep the public catalog open to AI search crawlers. If your course materials are the paid asset, block the training crawlers too, and above all keep the materials behind a login.

  • Keep AI search crawlers and user fetchers allowed on the public catalog.
  • Consider blocking GPTBot, ClaudeBot, CCBot and meta-externalagent, plus the shared __usage_ai-training token.
  • robots.txt doesn't stop anyone downloading paid videos or PDFs: it is crawl hygiene, not protection.

Example robots.txt

A starting point, not a drop-in file: check every path against your own URLs before publishing it.

# Education / Courses robots.txt starting point for www.example.com
# Review every path against your real URL structure before publishing.
# The catalog and course landing pages stay crawlable; the learning
# environment is blocked. Paid content must ALSO sit behind a login:
# robots.txt is not a paywall.

User-agent: *
# Learning environment
Disallow: /learn/
Disallow: /lesson/
Disallow: /player/
Disallow: /grades/
# Student areas and auth
Disallow: /my/
Disallow: /profile/
Disallow: /login/
Disallow: /enrol/
# Purchase funnel
Disallow: /cart/
Disallow: /checkout/
# Internal search and tracking
Disallow: /search/
Disallow: /*?q=
Disallow: /*?*utm_
# Per-student certificates
Disallow: /certificate/*?id=

Sitemap: https://www.example.com/sitemap.xml

Common mistakes

  • Leaving quiz and attempt URLs crawlable: endless URLs, and answers can leak.
  • Blocking course landing pages while trying to protect the content.
  • Using robots.txt as a paywall.
  • Listing gated lesson URLs in the sitemap while disallowing them.

Before you publish it

Some rules hold whatever your type of website: how crawlers match groups and paths, what status codes do to your file, and which crawlers you can actually block.

See what crawlers really do with your robots.txt

A robots.txt is a request, not a wall. HoneyLog reads your server logs, so you see which bots fetch your robots.txt, every version it went through, and which crawlers request pages it disallows for them.

How the top publishers configure robots.txt: the Robots.txt Leaderboard
14-day Free Trial

Try HoneyLog 14-day Free Trial

If you are interested in HoneyLog, you can try it with our Free Trial!

14-day Free Trial Contact Us
No credit card required