.png)
Between 19 and 22 October 2026, an AI search crawler identifying itself as ShapBot sent more than 1.2 million requests to Money.it, one of Italy's leading independent financial publishers.
Its primary target was aziende.money.it, a database containing company profiles, financial reports, and business comparisons.
In just three days, the crawler accessed 275,940 distinct company profiles, systematically requesting pages across the database.
The activity coincided with a dramatic increase in server errors, raising concerns about website performance and access to valuable proprietary content.
Using HoneyLog, the Money.it team was able to identify the crawler, analyze its behavior, implement a targeted firewall change, and verify the result almost immediately.
The collected evidence also enabled the team to prepare a formal cease-and-desist notice.
The discovery: a sudden surge in server errors
On the morning of October 22, Money.it's team was performing its routine traffic analysis using HoneyLog.
One metric immediately stood out.
The previous day, aziende.money.it had recorded 9,917 HTTP 500 errors, compared with just 41 the day before.
A closer examination of the server logs revealed that between 15:25 and 15:33 on October 21, the company website had experienced 9,441 server errors in just eight minutes, including 174 errors affecting real visitors.
At the same time, automated traffic had increased sharply.
Between midnight and 8 a.m. on October 22, HoneyLog recorded approximately 639,000 suspicious bot requests, compared with 121,500 during the same period the previous day.
Two crawlers accounted for much of this activity: ShapBot and Baiduspider.
By breaking traffic down by subdomain, user agent, and time interval, HoneyLog made it possible to isolate the activity that would otherwise have been difficult to distinguish from normal website traffic.
Identifying the crawler: 275,940 company profiles accessed
The investigation quickly focused on traffic identifying itself as ShapBot/0.1.0.
ShapBot is an automated web crawler associated with AI-powered search infrastructure. Its purpose is to collect web content for indexing and retrieval through AI search services.
Until October 18, its activity on Money.it had been relatively limited, with only a few thousand requests per day.
Starting October 19, that changed dramatically.
The crawler began sending up to 1,000 requests per minute, predominantly targeting aziende.money.it.
HoneyLog revealed a systematic access pattern:
- 275,940 distinct company profiles were accessed over three days.
- For individual profiles, the crawler typically sent a HEAD request followed by a GET request to retrieve the page.
- Over the preceding 30 days, the logs also recorded requests to 134,375 company comparison pages and 42,240 free company reports.
- The crawler also accessed company-claim pages and attempted to reach login, registration, and checkout pages.
This was not simply a crawler checking a handful of pages for updates. The request pattern was consistent with a systematic effort to retrieve large portions of the website's company database.
And because many of these requests did not correspond to human visits, conventional web analytics alone would not have revealed the full extent of the activity.
When robots.txt wasn't enough
Another important finding emerged from the server logs.
Between October 15 and 22, the crawler requested Money.it's robots.txt file 94 times.
Yet the logs also showed subsequent requests to URL paths that Money.it had explicitly excluded from crawling, including company comparison and free-report sections.
This distinction matters.
The robots.txt file communicates which parts of a website automated crawlers should or should not access. But it is not an access-control system.
Compliant crawlers can follow these instructions voluntarily. Crawlers that do not respect them can still attempt to retrieve the affected pages.
In this case, HoneyLog provided a record of both the robots.txt requests and the subsequent activity on restricted paths.
That evidence became important not only for the technical investigation, but also for documenting the incident.
The solution: a targeted Cloudflare rule
Money.it already maintained a sophisticated set of Cloudflare Web Application Firewall (WAF) rules to protect its websites.
However, the existing configuration had two limitations.
First, its protections against certain datacenter-based crawlers did not cover the public company profiles being accessed.
Second, a rule designed to exempt verified search engine bots from unnecessary challenges was preventing a later rule against Baiduspider from taking effect.
With all available custom rule slots already occupied, the team chose to modify an existing blocking rule rather than create a new one.
The updated rule specifically matched requests identifying themselves as ShapBot or Baiduspider.
The change went live on October 22 at 08:45:59 local time.
The result was immediately visible in HoneyLog.
Before the change, the two crawlers were receiving approximately 1,900 successful responses per minute.
During the minute the change was applied, that number dropped to 23.
From the following minute onward, HoneyLog recorded zero successful requests from those crawlers reaching the monitored origin.
Importantly, the targeted change preserved access for Googlebot, Bingbot, user-initiated AI assistant requests, and ordinary readers.
This illustrates an important principle of bot management: protecting a website does not necessarily require blocking all automated traffic.
The goal is to identify unwanted behavior and apply precise controls without disrupting legitimate crawlers or human visitors.
From traffic analysis to documentary evidence
Stopping unwanted crawler traffic was only part of the response.
HoneyLog also provided the Money.it team with a detailed record of what had happened.
This included:
- The volume and timing of requests.
- The user agents and source IP addresses associated with the activity.
- The number of distinct URLs accessed.
- The request patterns across company profiles and other sections.
- The crawler's requests to robots.txt and subsequent access to excluded paths.
- The measured impact of the firewall change.
This evidence enabled the team to prepare a formal cease-and-desist notice (lettera di diffida) addressing the automated activity.
The distinction is significant.
Rather than relying on general claims about aggressive scraping, the publisher could document the incident using timestamped server records and measurable request volumes.
Such records can help support discussions with crawler operators, investigate compliance with website access policies, and inform potential legal action.
Server logs do not, by themselves, establish a legal violation. But they provide a concrete factual basis for examining what occurred.
What other publishers can learn from this incident
Money.it's experience illustrates several challenges that publishers, marketplaces, directories, and other data-rich websites increasingly face.
1. Bot traffic needs its own analytics
Traditional analytics focus primarily on human activity. Server logs reveal automated requests that may not appear in conventional pageview metrics.
2. Monitoring individual sections matters
The surge was especially visible on aziende.money.it. Looking only at the entire Money.it network could have made the anomaly harder to detect.
3. robots.txt compliance should be verified
Publishing crawling rules is not the same as confirming that crawlers follow them. Comparing robots.txt directives with actual request behavior provides a clearer picture.
4. Firewall changes need measurable validation
A configuration can appear correct without producing the expected result. HoneyLog allowed Money.it to verify the effectiveness of its new rule at minute-level granularity.
5. Request logs are valuable evidence
Knowing which systems requested which pages, how frequently, and under which declared identities can be valuable for technical investigations, business discussions, and formal notices.
Beyond analytics: understanding and protecting digital assets
For publishers, online content is more than a source of traffic. It is an asset that requires investment to produce, maintain, and distribute.
As AI search platforms and automated data-collection systems expand, understanding how machines access that content is becoming an increasingly important operational concern.
The Money.it case demonstrates what server-side log analytics can contribute.
HoneyLog did not simply identify an unusual traffic spike. It provided the information needed to understand the incident, guide a targeted response, confirm its effectiveness, and document the activity.
For publishers dealing with increasing volumes of AI crawlers and scrapers, that visibility is the first step toward making informed decisions about their content.
Discover how HoneyLog helps publishers monitor AI crawlers, analyze bot behavior, and investigate automated access at honeylog.io.