Crawler vs. Scraper vs. Agent: A Field Guide to AI Bots
Short answer: A crawler systematically fetches pages to index them or build a dataset. A scraper targets and extracts specific content, often without announcing itself. An agent fetches pages in real time on behalf of a user, and often looks just like a person. They're all bots, but they want different things, behave differently, and call for different responses. Lumping them together as "AI traffic" is how publishers end up applying the wrong policy to the wrong bot.
This is the reference we point back to whenever these words come up. Bookmark it; the terms get used loosely everywhere else.
First, the umbrella term: "bot"
A bot is any automated software that makes requests to your site without a human directly driving each individual click. Every crawler, scraper, and agent is a bot. The word is useful as a category and useless as a description. Saying "we're getting a lot of bot traffic" tells you roughly as much as "we're getting a lot of vehicles." The whole point of this guide is the next level of detail.
Crawler
A crawler (also called a spider) is an automated program that fetches pages systematically, typically by following links from one page to the next, in order to build an index or a dataset. This is the oldest category: search engines have used crawlers for decades to index the web. The defining traits are that it's systematic (broad, repeated, link-following) and, for the well-behaved ones, self-identifying (it announces itself with a named user-agent and usually reads robots.txt).
In the AI era, crawlers split into two kinds that matter enormously to publishers:
- Training crawlers harvest content in bulk to build datasets that train AI models. They take and return nothing, resulting in pure extraction. Examples include GPTBot, ClaudeBot, CCBot, and Meta's training crawlers.
- Search / retrieval crawlers (also called answer crawlers) fetch pages to surface or cite them in live AI answers. These can send readers back to you when they link, which makes them a potential channel rather than a pure cost. Examples include OAI-SearchBot and PerplexityBot.
That single distinction (training versus search) is the difference between a crawler you might block and one you might want to keep. Treating both as "AI crawlers" erases the most important fact about them.
Scraper
A scraper is software that extracts specific content or data from pages. Where a crawler is broad and systematic, a scraper tends to be targeted, meaning it goes after particular content (an article archive, pricing, listings) rather than indexing everything. And where a well-behaved crawler identifies itself, a scraper frequently doesn't: it may send no user-agent, or spoof an ordinary browser's, precisely to avoid the controls that catch crawlers.
The line between crawler and scraper is partly behavioral and partly about intent. A polite, self-identifying program systematically building an index reads as a crawler; a stealthy, targeted program extracting content while disguised as a browser reads as a scraper. The practical upshot: crawlers are mostly a naming problem (you have to know which to allow or block), while scrapers are a detection problem (you have to spot them at all).
Agent
An agent (AI agent, or user-triggered fetcher) is software that fetches pages in real time on behalf of a specific user, such as an AI assistant browsing the web to answer a question, or an agentic browser carrying out a task. This is the newest category and the fastest-growing.
Agents break the old mental model in three ways. They're triggered by a user action rather than running on an indexing schedule. They often render pages like a real browser, executing JavaScript and behaving like a person. And they frequently ignore robots.txt by design, because the company running them treats a user-triggered fetch as a person browsing, not a crawler indexing. The combined effect is that an agent can be nearly indistinguishable from genuine human traffic, which makes it the hardest of the three to identify and the easiest to overlook.
The three at a glance
| Crawler | Scraper | Agent | |
|---|---|---|---|
| What it does | Systematically fetches pages to index or build datasets | Extracts targeted content or data | Fetches pages in real time for a specific user |
| Triggered by | An automated schedule or link-following | An operator's extraction task | A user action, in the moment |
| Identifies itself? | Usually, with a named user-agent | Often not; may spoof a browser | Sometimes; often looks like a user |
| Respects robots.txt? | The compliant ones do | Frequently not | Frequently not (treated as a user proxy) |
| Value to you | Training: a cost. Search: possible referrals | Usually pure extraction | Possible referrals; hard to tell |
| Your hardest problem | Keeping the name list current | Detecting it at all | Telling it apart from a human |
Why the distinction actually matters
This isn't taxonomy for its own sake. Which category a visitor falls into decides three things:
Its value. A training crawler is a cost. A search crawler or a citing agent might be a discovery channel. Block them as one and you may cut off readers to stop extraction.
How you can even see it. A self-identifying crawler shows up by name in your logs. A disguised scraper or a human-looking agent doesn't, so you can only spot it by how it behaves. Identification difficulty rises sharply from left to right across that table.
How you can control it. robots.txt only governs crawlers that choose to obey it; scrapers and agents need detection plus enforcement at the edge. (We cover the limits of robots.txt in why robots.txt won't protect your content.)
In other words, the right response (block, charge, negotiate, or allow) depends entirely on correctly identifying what you're looking at. That's the whole premise of the block, negotiate, or monetize framework, and it's impossible without measurement.
A snapshot of the major AI bots (mid-2026)
The roster changes constantly, which is itself a key point, since any list is out of date the moment a new agent ships. With that caveat, here's roughly how the best-known names sort as of mid-2026:
- Training crawlers: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Meta-ExternalAgent and FacebookBot (Meta), Bytespider (ByteDance), Amazonbot (Amazon).
- Search / retrieval crawlers: OAI-SearchBot (OpenAI), PerplexityBot (Perplexity), Applebot (Apple), and Anthropic's retrieval crawler.
- User-triggered agents: ChatGPT-User (ChatGPT browsing), Perplexity-User, and Google's user-triggered fetchers.
A common trap: control tokens aren't crawlers
Some names you'll see in robots.txt guides aren't crawlers at all. Google-Extended and Applebot-Extended are control tokens, meaning they don't make requests and never appear in your logs as traffic. They only govern whether content that was already fetched may be used for AI training. Disallowing Google-Extended doesn't turn a crawler away; it files a preference about usage. Confusing a control token for a bot is one of the most common mistakes in AI-crawler policy.
How to tell which is actually hitting you
For the honest crawlers, the user-agent in your server logs tells you most of what you need. For the scrapers spoofing browsers and the agents that look like people, the name is useless and behavior is everything, including request patterns, volume, timing, and how the traffic moves through your site. That's a measurement problem, and it lives in your logs, the one place every one of these bots leaves a trace regardless of what it calls itself. Turning those logs into a clear, continuous, classified view of who's crawling, scraping, and agent-fetching your content is exactly what HoneyLog does. (For why your normal analytics can't, see the AI traffic your analytics can't see.)
Frequently asked questions
What's the difference between a crawler and a scraper?
A crawler fetches pages systematically, broadly by following links and usually identifying itself, to index them or build a dataset. A scraper targets specific content to extract, often without announcing itself or while disguised as a browser. The practical difference: crawlers are a naming problem, scrapers are a detection problem.
Is an AI agent a bot?
Yes. It's automated software making requests, but it's triggered by a user in real time and often renders pages like a person, which makes it behave more like human traffic and harder to detect than a traditional crawler.
Is GPTBot a crawler or a scraper?
GPTBot is OpenAI's training crawler. OpenAI also runs OAI-SearchBot (a search/retrieval crawler) and ChatGPT-User (a user-triggered agent), representing three different bots with three different purposes.
Which AI bots send me traffic?
Generally the search and retrieval crawlers, and some agents, specifically the ones that cite and link back to you. Training crawlers extract content and send nothing in return.
How do I tell which type is hitting my site?
For self-identifying crawlers, the user-agent in your server logs is enough. For disguised scrapers and human-looking agents, you have to analyze behavior, not names, which is what server-log measurement is for.
Related reading:
Last updated: June 2026. The bot roster changes constantly; we keep this field guide current.