Cloudflare AI Crawler Block Now Default for New Domains
Cloudflare has made a bold move in the battle over AI data scraping by enabling its AI crawler block feature by default. This change comes amid growing concerns from publishers about their content being used by AI models without permission or pay. If you're wondering what the Cloudflare AI crawler block means for your site, here’s what you need to know: new domain owners on Cloudflare will now be asked whether they want to allow AI scrapers access to their websites, with blocking enabled by default. This aligns with increasing calls across the industry for ethical content usage and compensation.
Why Cloudflare Is Blocking AI Crawlers by Default
Cloudflare’s new default setting aims to protect creators from having their content scraped by AI without consent. The company’s updated features include identifying known AI crawlers—regardless of whether they respect robots.txt—and blocking them automatically. This builds on Cloudflare’s earlier efforts to help sites push back against unwanted scraping, such as launching the “AI Labyrinth,” a tool designed to trap and confuse aggressive bots. By shifting the default stance to block AI, Cloudflare is putting more control back into the hands of website owners and publishers.
Introducing 'Pay Per Crawl' for Ethical AI Training
Another significant update is Cloudflare’s Pay Per Crawl program, which allows select publishers to set a price for AI crawlers to access their content. AI companies will need to view the pricing and opt in—or walk away. While this model is currently available only to a limited group of large publishers, Cloudflare has indicated plans to expand it. This system creates a monetization opportunity for publishers while encouraging responsible AI training practices—making the Cloudflare AI crawler block not just a barrier, but a possible business model.
How Publishers and AI Companies Are Responding
Major platforms like The Atlantic, Quora, Fortune, and Stack Overflow are supporting Cloudflare’s new measures, seeing them as a step toward preserving original content in an AI-dominated web. Cloudflare is also working with AI companies to clearly identify their crawlers, allowing site owners to decide which bots to allow based on their intended use—whether for training, inference, or search. As Cloudflare CEO Matthew Prince put it, “We’re restoring power to the content creators.” The cloudflare ai crawler block isn’t just a tech update—it’s part of a larger movement to ensure fair use and protect the future of the open web.
Comment