HoneyLog vs. Cloudflare: Block AI Bots, or Understand Them?

Short answer: Cloudflare and HoneyLog solve two different layers of the same problem. Cloudflare is infrastructure. It sits at the edge of your network and blocks, allows, or charges AI crawlers as they arrive. HoneyLog is independent intelligence. It measures every AI bot that reaches your content from any source and turns that into the evidence you need to decide whether to block, negotiate, or monetize. Cloudflare acts on the traffic. HoneyLog tells you what the traffic is worth. Most publishers end up wanting both, and this article explains exactly where the line sits.

If you're a publisher weighing how to handle AI crawlers, that distinction is the whole decision. Let's unpack it.

The real problem: AI traffic is exploding, and most of it is invisible

For two decades the deal was simple: if a page was public, a crawler could read it, and in return search engines sent you readers. The AI boom broke that deal. Models now consume your content to generate answers that never send a visitor back, and the volume has gone vertical. By most industry estimates, AI-related bot activity grew several-fold over the past year, with individual crawlers like GPTBot and Meta's agents posting triple-digit annual growth.

The harder problem isn't the volume. It's the visibility. Your standard analytics stack (Google Analytics, Adobe, even most product analytics) runs on JavaScript that bots don't execute. So the single fastest-growing category of traffic to your content is also the one your dashboards can't see. You can't manage, price, or defend what you can't measure.

There are two very different responses to this, and Cloudflare and HoneyLog represent each.

What Cloudflare does (and does well)

Cloudflare is an infrastructure company that sits in front of a large share of the public web, and over the past two years it has moved aggressively to give site owners control over AI crawlers:

• Block by default. New sites on Cloudflare now block AI crawlers unless the owner grants access, a posture Cloudflare describes as giving publishers a “default of control.”

• Allow, Charge, or Block. For each crawler, you can grant free access, deny it entirely, or charge for it.

• Pay-Per-Crawl. Cloudflare’s marketplace returns an HTTP 402 Payment Required response with a price; participating AI companies signal willingness to pay, and Cloudflare settles the transaction in the middle.

• Managed controls. Features like an auto-updating robots.txt and AI crawl-control rules keep your blocklists current as new bots appear.

This is genuinely powerful, and for a lot of sites it's enough. If your goal is to stop unwanted crawlers quickly and cheaply at the edge, Cloudflare is excellent at it. (See Cloudflare's Pay-Per-Crawl announcement for the mechanics.)

But notice what kind of tool this is. Cloudflare is a gate. Its job is to decide, in the moment, what gets through. That's a different job from understanding what's happening, and three structural limits follow from it.

1. Its view stops at its own network. Cloudflare can only see and act on traffic that routes through Cloudflare. If you run multiple CDNs, sit behind a different provider, or have origins that bypass the edge, that traffic is outside its line of sight.

2. The default action is to decide before you understand. “Block by default” is a sensible safety setting, but it means access is being granted or denied before you’ve seen who’s crawling what, how often, and against which of your most valuable content.

3. One party controls blocking, analytics, and payments. Cloudflare brokers the crawl fees and partners with the AI companies paying them and supplies the numbers. That’s convenient. It is not independent.

None of this is a knock on Cloudflare. It's the nature of edge infrastructure. It's also exactly the gap HoneyLog is built to fill.

What HoneyLog does differently

HoneyLog isn't a gate. It's a measurement layer. It reads your server logs (at the origin, where every request lands regardless of which CDN or security stack it passed through) and turns them into continuous, readable intelligence about AI and bot traffic, even at very high volume.

In practice, that means HoneyLog tells you:

• Who is actually reaching your content, including which AI companies, crawlers, scrapers, and automated agents, identified in real time.

• What they’re taking, such as which sections, articles, and assets are being consumed most heavily, and how that’s trending.

• What it’s worth, providing the evidence base you need to choose your response, and to defend that choice when you’re across the table from an AI company.

Crucially, HoneyLog is independent of your infrastructure. It doesn't care whether you're on Cloudflare, a different CDN, several at once, or none. It measures what hits your origin, period. That independence is the entire point: it gives you a neutral source of truth that isn't owned by the same party selling you the gate.

Infrastructure vs. intelligence: the core difference

Here's the cleanest way to hold the distinction in your head:

Cloudflare answers the question "should this request get through right now?"
HoneyLog answers the question "what is all of this traffic, and what should I do about it?"

The first is an enforcement decision made in milliseconds at the edge. The second is a strategic decision (block, negotiate, or monetize) that requires seeing the full picture over time from a source you trust. You can't make the second decision well by relying solely on the tool that's optimized for the first.

HoneyLog vs. Cloudflare at a glance

"I already use Cloudflare. Isn't that enough?"

Often, no, and the reason is the most valuable scenario for a publisher: negotiation.

The biggest content owners aren't settling for whatever per-crawl rate clears in a marketplace. They're cutting direct licensing deals with AI companies, the way major publishers have done over the past two years. To negotiate one of those, you need independent proof of exactly what's being taken: which models, how much, against which content, and worth how much. Walking into that room with numbers supplied by a platform that also partners with the company on the other side of the table is a weak position. Walking in with your own independent measurement is leverage.

The same logic applies in three other common cases:

• You’re multi-CDN or not fully behind Cloudflare. Edge-level analytics can’t see what doesn’t pass through the edge. Origin-level measurement can.

• You want to verify, not just trust. An independent count lets you sanity-check what any infrastructure provider reports.

• You want to decide before you block. Blanket blocking can quietly cost you because some AI crawlers drive citations and referral value. Measuring first tells you which bots to keep, charge, or shut out, instead of guessing.

So do you need both?

For most serious publishers, yes, and they're complementary, not competing.

Use Cloudflare (or your bot-management layer of choice) as the enforcement arm: the gate that acts on your decisions at the edge. Use HoneyLog as the intelligence arm: the independent measurement that tells you which decisions to make and gives you the evidence to back them. Enforcement without intelligence is guesswork. Intelligence without enforcement is a report nobody acts on. Together, they let you move deliberately from "AI is taking our content" to "here's exactly what's being taken, and here's our terms."

That's the shift from blocking blindly to negotiating from strength, and it's the entire reason HoneyLog exists.

Frequently asked questions

Is HoneyLog a Cloudflare alternative?
Not exactly, because they do different jobs. Cloudflare blocks and brokers crawler access at the edge; HoneyLog independently measures AI traffic at your origin so you can decide what to do. Many publishers run both.

Can't Cloudflare's analytics already show me AI bot traffic?
Cloudflare can show you crawler activity that passes through its network, within its own dashboard. HoneyLog measures all AI traffic reaching your origin regardless of CDN, as an independent source of truth. This matters most when you're verifying numbers or negotiating a licensing deal.

Does HoneyLog block AI crawlers?
HoneyLog's role is measurement and intelligence, not enforcement. It tells you which bots to block, charge, or allow; your CDN or bot-management layer carries out the action.

Do I need HoneyLog if I'm not on Cloudflare?
Yes, as that's a core advantage. Because HoneyLog reads origin server logs, it works the same whether you're on Cloudflare, another provider, several, or none.

Should I just block all AI crawlers and be done with it?
You can, but it's a decision worth making with data. Some AI crawlers create citation and referral value; others purely extract. Measuring first tells you which is which. (We cover this in depth in our guide on whether blocking removes the need for analytics.)

Related reading:

• The AI traffic your analytics can’t see

• Cloudflare blocks AI bots by default, so why measure them?

• Block, negotiate, or monetize: a framework for AI traffic

Last updated: June 2026. The AI-crawler landscape changes quickly; we revisit this comparison as Cloudflare's features and the licensing market evolve.