Ammar Sabilarrohman
The digital landscape is undergoing a fundamental structural transformation. One year after Cloudflare initiated a seismic shift in web traffic management by allowing publishers to block AI crawlers by default, the environment has moved from a binary "allow or deny" model to a complex, fragmented ecosystem of declared "good bots," ambiguous "mixed-use" agents, and an unchecked, illicit gray market of data scrapers.
As of this week, the battle for content ownership has entered a new phase. Cloudflare has announced a stringent update to its bot management protocols, effectively setting a "deadline to end the free pass" for crawlers that oscillate between serving traditional search engines and feeding resource-hungry AI training models. This pivot highlights the growing desperation of content creators to reclaim value in an era where their intellectual property is increasingly harvested without compensation or consent.
The Chronology: A Year of Escalation
The current climate is the culmination of twelve months of aggressive adaptation.
- September 2023: Cloudflare flipped its default settings, empowering millions of websites to block AI scrapers with a single click. This was widely hailed as a victory for the publishing industry.
- Early 2024: The rise of "agentic" AI—autonomous systems capable of browsing the web to perform tasks—blurred the lines between search and training. Publishers realized that blocking an AI crawler often meant sacrificing visibility on search engines, creating a "catch-22" for site owners.
- August 2026: Cloudflare announced new rules for "mixed-use" crawlers, slated to go into effect on September 15. The new policy mandates that unless specific agreements are in place, the default settings will permit search functionality while explicitly blocking AI training and agentic scraping for ad-supported pages.
- Present: The industry is now bracing for a confrontation with the "mixed-use giants," most notably Google, whose crawlers have historically been treated as essential infrastructure for search but are now becoming the primary conduits for AI training data.
The "Mixed-Use" Dilemma
At the heart of this conflict lies a technical and philosophical challenge: how to decouple search discovery from AI model training. Currently, the most prominent crawlers in the world perform dual roles. They index content to help users find information (a service publishers value) while simultaneously ingesting that same content to refine Large Language Models (LLMs) (a service publishers increasingly view as an existential threat).
Stephanie Cohen, Chief Strategy Officer at Cloudflare, characterizes this as an "impossible position" for content creators. "We understand how we got here with the mixed-use crawlers, but it doesn’t work for the ecosystem," Cohen stated. "There has to be a better way than forcing people into the decision of all or nothing."
Cloudflare is not claiming the power to unilaterally sever these functions from tech behemoths. Instead, they are positioning themselves as the arbiter of a "flourishing agentic internet." By setting the default to block training on September 15, Cloudflare hopes to force a dialogue between AI companies and publishers. The goal is to ensure that discoverability—the lifeblood of digital media—does not come at the cost of providing free labor for proprietary AI models.
The Gray Market: The Invisible Threat
While the industry focuses on the "polite" bots that respect robots.txt and declare their intent, a massive, unregulated gray market continues to thrive. Frederick Jahn, founder of Centinel Analytica and a veteran of reverse-engineering scraping tools, argues that the current fixation on declared bots ignores the true magnitude of the problem.
"Cloudflare only governs what’s behind Cloudflare," Jahn notes. "Well-funded scrapers can route around it using residential proxies or by purchasing datasets from second-hand brokers."
Jahn estimates that for major news brands, while 20% to 30% of traffic originates from identified crawlers, nearly a quarter of total traffic consists of "stealth" crawlers. These bots mimic human behavior, making them virtually indistinguishable from legitimate readers. This opacity is the foundation of a "scraping-as-a-service" economy. Because this traffic is routed through complex proxy infrastructures, it is nearly impossible for publishers to link a specific scrape to a specific AI company, rendering legal recourse ineffective.
For publishers, the strategy must shift from passive compliance to active defense. Jahn advocates for "high-friction" tactics:
- Client-side challenges: Deploying "are you human?" tests that trigger upon page load.
- Systematic blocking: Implementing rigorous, non-negotiable filters for non-essential crawlers.
- Benchmarking: Holding vendors accountable to 90%+ detection rates for malicious bot activity.
The ultimate objective is to make scraping so expensive and unreliable that intermediaries are forced into a technical arms race, eventually pushing AI companies toward formal, transparent licensing agreements.
Emerging Models: The "Good Bot" Business Case
Amid the hostility, a new model of "good bots" is emerging—one based on transparency and economic reciprocity. A notable example is Ceramic, an AI search startup founded by former Google Engineering VP Anna Patterson.
Ceramic is partnering with infrastructure gatekeepers like Cloudflare to create a premium search API that pays publishers. The unit economics are designed to be radically more efficient than current AI search models. While traditional APIs might charge $5 to $14 per 1,000 queries, Ceramic claims to operate at approximately $0.05 per 1,000.
By crawling a page once and serving high-quality, 100- to 300-word snippets to LLMs, the system avoids the need for constant, server-hammering requests. Crucially, the model utilizes a revenue-share system. If a publisher’s snippet is used to answer an AI query, they receive a payout, regardless of whether the final output explicitly cites them.
This model changes the incentive structure entirely. Instead of "begging for scraps" through occasional headline licensing deals, publishers gain a metered, usage-based income stream. "It’s an experimental time," Patterson says, "and we’ll likely see more pretty interesting experiments in this direction."
Regulatory and Industry Implications
The ripples of these technical changes are being felt in regulatory halls. Chris Dicker, CEO of Candr Media, points to the U.K. Competition and Markets Authority (CMA) as a critical player. Until now, Google has defended its refusal to separate search and AI crawling by claiming it is technically infeasible.
However, if Cloudflare successfully demonstrates that other, smaller AI players can manage split or declared crawling, Google’s defense becomes significantly weaker. A successful implementation of Cloudflare’s new defaults could provide the CMA with the necessary evidence to mandate that search and AI training be treated as distinct services, fundamentally altering the legal landscape for tech monopolies.
Furthermore, the "financialization" of the advertising community is accelerating this trend. As Daniel Knapp, IAB Europe’s chief economist, notes, there is a growing demand for "quick, rapid metrics" that correlate digital activity with bottom-line growth. Publishers are no longer content to allow their content to be "donated" to AI training; they are demanding the same level of accountability and financial transparency that has defined the ad-tech industry for decades.
Conclusion: The Path Forward
The "all-or-nothing" era of web crawling is coming to a close. While the threat of stealth scraping persists, the move toward "good bot" ecosystems and stricter default protections represents a significant rebalancing of power.
For the average publisher, the path forward requires a dual approach: high-friction technical defenses to deter illicit scrapers, and the adoption of transparent, revenue-sharing partnerships to monetize the "agentic" future. The landscape is no longer just about traffic; it is about data sovereignty. As the lines between search, training, and agency continue to blur, the publishers who survive will be those who treat their content not as a public good, but as a high-value asset in an increasingly automated economy.
Lina Irawan
Nana Muazin
