Cloudflare Blocks AI Bots by Default, So Why Measure Them?
Short answer: Blocking AI crawlers by default stops some traffic, but it doesn't tell you anything, and a blunt block quietly costs you in three ways. It can't see what slips through disguised as human traffic. It makes a strategic decision (block, negotiate, or monetize) before you have any information. And it cuts off the AI crawlers that send you readers along with the ones that just extract. Blocking is an action. Measurement is what tells you whether it was the right one. You need both, and measuring never means letting anything in.
That last point trips up a lot of people, so let's start there, because it dissolves the whole objection.
"If I block them, what's left to measure?"
The instinct is reasonable. You flipped on block-by-default, the dashboard says crawlers are being stopped, and it feels handled. Why pay to measure something you've already shut out?
Here's the thing most people miss: a blocked request still lands in your server logs. When a crawler hits your origin and gets a 403 or a 402, that attempt is recorded (the bot, the page it wanted, the time, and the frequency). Blocking doesn't make the traffic invisible; it just denies it. So measuring what you block doesn't require letting any of it in. You get a complete picture of everything you're turning away, which is exactly the information you need to know whether you're turning away the right things.
In other words, blocking and measuring aren't alternatives. Measuring is how you find out if your blocking is working, what it's missing, and what it's costing you. Without it, "block by default" isn't a strategy; it's a guess you never check.
Let's look at the three things that guess hides.
Blind spot #1: a block only reports what it caught
Default blocking works by recognizing bots (known user-agents, verified signatures, and published IP ranges). That catches the well-behaved crawlers that announce themselves. It does much less against the ones that don't:
- Crawlers that disguise themselves or ignore the rules entirely, as a persistent share of AI traffic does exactly this.
- New crawlers and agents that appear faster than any blocklist updates, so there's always a lag where the newcomer gets through.
- Browser-based AI agents and scrapers that render pages like a real person and look, to bot detection, like ordinary human traffic.
The uncomfortable part: the tool doing the blocking can only report on what it identified. It cannot report on what it missed, which is the definition of a blind spot. You can't measure the gap in your defenses using the thing that has the gap. Independent, origin-level measurement is what surfaces the traffic that slipped past, because it reads every request that reached your content, not just the ones the gate recognized.
Blind spot #2: you decided before you understood
Block-by-default makes the most consequential choice for you (deny everything) before you've seen a single number. But the right move isn't the same for every crawler or every section of your site. The strategic options are:
- Block the bots that purely extract and bring nothing back.
- Charge the ones willing to pay for access.
- Negotiate a licensing deal with the companies taking the most.
- Allow the ones that actually drive value (more on that next).
You can only sort crawlers into those buckets if you can see who's taking what, how much, and how it's trending. Blocking everything by default skips that step entirely. It's a sensible safety setting, but a safety setting is not a strategy, and treating it as one means leaving the actual decisions unmade.
Blind spot #3: you may be blocking the bots that bring you readers
Not all AI crawlers are the same animal. Cloudflare itself categorizes them by purpose (training, search, and generation) for a reason: they have very different value to you.
A pure training crawler consumes your content and sends nothing back. But a search or user-facing crawler, the kind behind AI answers that cite sources, can drive real referral traffic and visibility when it links to you. Blanket-blocking treats both identically, which means a default block can quietly cut off the AI surfaces that are becoming a genuine discovery channel, right at the moment that channel is growing.
Without measurement, you'll never see this cost, because it shows up as an absence, specifically readers who never arrived. Measuring which crawlers cite and refer versus which only extract is the only way to block the parasites while keeping the partners.
What blocking does and doesn't do
| Blocking by default does | Blocking by default doesn't |
|---|---|
| Stop known crawlers at the edge | Show what slipped through disguised as human traffic |
| Reduce bot load on your servers | Tell you which AI companies took which content |
| Give you a fast, safe baseline | Separate extractive bots from ones that send readers |
| Apply one rule to everything | Give you evidence to charge, negotiate, or monetize |
The left column is genuinely useful. The right column is where the money and the strategy live, and none of it is available without measurement.
The reframe: measurement makes blocking smart
Blocking is defense. On its own, it's defense with the lights off. You're swinging at threats you can't see, with no idea what you hit, what you missed, or what you cost yourself.
Measurement turns the lights on. It tells you which bots to keep blocking, which to charge, which to let in, and which to build a licensing conversation around. And it converts a defensive posture into leverage: the publishers signing direct deals with AI companies aren't the ones who blocked quietly and kept no records; they're the ones who can prove exactly what's being taken and what it's worth. Block first and measure nothing, and you've thrown away both the evidence and the revenue.
So keep blocking by default. It's a fine place to start. Just don't mistake the starting line for the finish.
What to actually do
- Measure first. Put origin-level measurement in place so you can see all AI traffic, including what your blocking is catching and what it's missing.
- Sort your crawlers. Identify which extract, which refer, and which are worth charging.
- Set policy per crawler, not one blanket rule. Block, charge, allow, or negotiate based on what the data shows.
- Keep watching. New agents appear constantly; measurement is your early-warning system for the ones your defaults haven't learned yet.
That's the difference between handling AI traffic and hoping you've handled it.
Frequently asked questions
If I block AI bots, do I still need analytics?
Yes. Blocking acts on traffic; it doesn't tell you what the traffic is, what got through anyway, or what blocking cost you. Measurement answers all three, and it's what lets you charge or negotiate instead of just denying.
Doesn't measuring AI bots mean I have to let them in?
No. Blocked requests still appear in your server logs, so you can measure everything you're blocking without granting any access. Measurement reads the record; it doesn't open the door.
Won't Cloudflare just tell me what it blocked?
It can report what it caught on its own network, in its own dashboard. It can't report what it missed, what bypassed it, or what's reaching origins outside its edge, and it isn't an independent source of truth. (We cover this distinction in HoneyLog vs. Cloudflare.)
Which AI bots should I allow?
It depends on your goals, but the ones that cite and refer readers back to you are often worth keeping, while pure-extraction training crawlers are the usual candidates to block or charge. You can only tell them apart by measuring.
Is blocking by default a bad idea?
Not at all; it's a sensible safety default. It's just incomplete. Pair it with measurement and it becomes a strategy instead of a guess.
Related reading:
Last updated: June 2026. The AI-crawler landscape changes quickly; we revisit these pieces as the tools and the licensing market evolve.