Block, Negotiate, or Monetize: A Framework for AI Traffic
Short answer: Every AI crawler hitting your content is really one of four decisions: allow it, block it, charge it, or negotiate with it. The right choice turns on two questions: does the crawler send value back to you (citations, referrals), or does it purely extract? And how much does it take? But none of these decisions can be made, let alone defended in a negotiation, without first measuring your AI traffic at the server level. Measurement is step zero. The framework is what you do with what you find.
Most publishers are stuck on a single default (block everything) because it's the only move available without data. This is the framework for everyone else.
Why you need a framework at all
"Block by default" feels decisive, but it isn't a strategy; it's the absence of one. It applies a single rule to crawlers that want completely different things from you, and it makes the most consequential choice (deny everything) before you've seen a single number. That's fine as a safety setting and a bad place to stop. (We dig into why in why blocking by default isn't enough.)
A real strategy treats each crawler as a separate decision, because the crawler harvesting your archive to train a model and the crawler fetching today's article to cite it in an AI answer are not the same problem. One is a cost. The other might be your next distribution channel. A framework is just a disciplined way of telling them apart and responding accordingly.
Step zero: you can't decide what you can't see
Before any of the four moves, you need answers to a short list of questions, per crawler:
- Who is it? Which AI company, and which crawler (a training bot, a search/answer bot, a scraper, an agent)?
- What is it taking? Which sections and articles, and how much of your most valuable content?
- How much, how often? Volume and frequency, and how both are trending.
- Does it send anything back? Citations, referral visits, visibility in AI answers.
Conventional analytics can't answer any of these, because they can't see bots at all (the full explanation is in the AI traffic your analytics can't see). The answers live in your server logs, and surfacing them continuously (at scale, across whatever CDN or security stack you run) is exactly what HoneyLog is built for. Everything below assumes you can see your traffic. If you can't yet, that's the first thing to fix.
The two questions that drive every decision
Once you can see a crawler clearly, two questions decide its fate:
- Does it bring value back? A crawler that cites you and refers readers is a channel. One that extracts and returns nothing is a cost.
- How much does it take, and how valuable is what it takes? Scale and content value determine whether a crawler is worth charging, worth a full negotiation, or simply worth blocking.
Plot any crawler against those two questions and the right move falls out.
The four moves
Allow, the one people forget
If a crawler cites your content and sends readers back, blocking it is self-sabotage. The answer engines behind AI citations are becoming a genuine discovery channel, and for some publishers the visibility is worth more than any access fee. Allowing isn't passive, though: you still measure what you allow, so you can confirm the value is real and notice if a "good" crawler's behavior changes.
Best for: search and answer crawlers that drive citations and referrals.
Requires: ongoing measurement to verify the value is actually flowing back.
Block, deny the pure extractors
Some crawlers take and give nothing, including training bots harvesting content with no referral upside and scrapers ignoring your terms. If there's no value coming back and no realistic prospect of payment, blocking is the right call. It saves resources and protects content you'd rather not feed into a competitor.
Best for: pure-extraction crawlers with no upside and no willingness to pay.
Requires: confidence you're not also blocking a referrer, which means measuring first, not blanket-blocking blind.
Monetize, charge for access
For crawlers that extract but are willing to pay, you can turn routine crawling into a metered revenue stream, for example through a pay-per-crawl arrangement that charges per request. This captures revenue from the mid-tier of AI consumers who won't justify a bespoke deal but will pay to keep crawling.
Best for: willing-to-pay extractors at a scale that doesn't warrant a full negotiation.
Requires: knowing your own numbers, because marketplace mechanics set the price and you want to enter on terms you understand rather than accept whatever clears.
Negotiate, strike a licensing deal
The largest AI consumers, taking your highest-value content at scale, are worth a direct conversation. The publishers winning here aren't settling for a per-crawl rate; they're signing licensing agreements. The catch is leverage: a direct deal is only as strong as your evidence of exactly what's being taken and what it's worth. Independent measurement is that leverage. (More in what your AI traffic data is worth in a negotiation.)
Best for: the heaviest takers of high-value content, or strategic players.
Requires: independent, defensible data on what's being consumed, which serves as your strongest card at the table.
The decision in one view
| If a crawler… | The move is | Because |
|---|---|---|
| Cites you and refers readers back | Allow | The discovery and traffic outweigh any fee |
| Purely extracts, low volume or low-value content | Block | No upside, so you deny it and save the resources |
| Extracts but is willing to pay for access | Monetize (charge) | Turn routine crawling into metered revenue |
| Takes heavily, high-value content, strategic player | Negotiate | A licensing deal captures far more than a per-crawl fee |
It's not one rule, and it's not forever
Two cautions on using the framework.
First, it's per crawler, not per site. The point is that different bots get different treatment. You might allow the answer crawler that cites you, block the scraper that disguises itself, charge a mid-tier training bot, and negotiate with the largest. A single blanket policy throws all of that nuance away.
Second, it's a moving target. New crawlers and agents appear constantly, and a crawler's behavior, along with its value to you, can change. A bot that only extracted last quarter might start citing you this one. The framework isn't a one-time sorting exercise; it's a posture you maintain, which again comes back to continuous measurement rather than a single audit.
There's also no universally "correct" answer. A publisher betting on AI distribution might allow far more than one protecting a premium archive. The framework doesn't prescribe your strategy; it makes sure whatever strategy you choose is a deliberate decision rather than a default you backed into.
Where HoneyLog fits
Every move in this framework depends on the same foundation: seeing your AI traffic clearly, continuously, and independently. HoneyLog reads your server logs and turns them into a real-time view of every AI bot reaching your content (who it is, what it's taking, and whether it sends anything back) regardless of your CDN. That's what makes the difference between guessing and deciding, and it's the evidence base you bring to a negotiation. The framework is the strategy; HoneyLog is what makes it actionable.
Frequently asked questions
What should I do about AI crawlers on my site?
Treat each one as a separate decision: allow the crawlers that cite and refer readers, block the pure extractors with no upside, charge the ones willing to pay, and negotiate with the largest, highest-value takers. Which is which depends on measuring your traffic first.
Should I just block all AI bots?
You can, but blanket-blocking also cuts off the crawlers that send you readers, and it forfeits any chance to charge or negotiate. Decide per crawler, with data, not one rule applied blind.
How do I monetize AI crawler traffic?
Two main routes: charge for access programmatically (for example via pay-per-crawl), or negotiate a direct licensing deal. Charging suits willing-to-pay mid-tier crawlers; negotiation suits the heaviest takers of your most valuable content.
Do I need data to negotiate an AI licensing deal?
Yes. Your leverage is independent, defensible evidence of exactly what's being taken and what it's worth. Without it, you're negotiating blind against a counterparty that isn't.
Does the right strategy change over time?
Constantly. New crawlers appear, and existing ones change behavior, as a bot that only extracted may start citing you. Revisit the decision per crawler rather than setting it once.
Related reading
Last updated: June 2026. The AI traffic market moves fast; we keep this framework current as the options evolve.