Cloudflare Gives Website Owners Granular Control Over AI Crawlers: Search, Agents, and Training Now Separately Managed

Cloudflare is fundamentally updating how it handles automated traffic, replacing its simple “Block AI Bots” toggle with more nuanced options that let site owners differentiate between search indexing, AI agents, and model training. The changes, announced on July 1, 2026, aim to restore balance in the relationship between content creators and crawlers in the age of generative AI.

The move builds on Cloudflare’s “Content Independence Day” initiative from a year earlier, which introduced one-click blocking of AI bots and a pay-per-crawl marketplace. Now, the company is moving beyond binary choices to a pragmatic taxonomy that reflects how bots actually behave on websites.

Three Categories for Better Control

Cloudflare Cloudflare’s new system classifies automated traffic into three main AI-related use cases, available to all customers — including those on the free plan:

Search: Crawlers that index content to help answer user queries later, typically driving referral traffic back to the site.
Agent: Real-time, user-directed automation (e.g., chat fetch bots like ChatGPT-User or browser agents powered by models like Gemini or Claude) acting on behalf of a person to complete tasks.
Training: Crawlers that absorb content to train or fine-tune AI models, permanently incorporating data into the system’s capabilities.

Site owners can now manage each category independently — for example, allowing search crawlers while blocking training and agents. This level of control applies network-wide through managed rules and presets.

New Defaults Starting September 15, 2026

Bot Traffic Surpasses Human Traffic for the First Time — Thanks to AI Agents, Says Cloudflare CEO For new domains onboarding to Cloudflare, defaults will shift on September 15, 2026, particularly on pages displaying ads:

Training and Agent bots → Blocked by default.
Search bots → Allowed by default.

The rationale is straightforward: ads signal that a site owner wants human visitors and monetization through attention. Training and agent bots, which often consume content without returning equivalent value, are restricted on those pages, while traditional search indexing — which can drive discoverability and traffic — remains permitted. Existing customers can opt out of the new defaults if they prefer no changes.

The Multi-Purpose Crawler Challenge

A key change addresses “mixed-use” or multi-purpose crawlers. Bots like Googlebot, Applebot, and BingBot often perform both search indexing and training (or other functions) under the same identity. Under the new system, these will be evaluated based on all their behaviors.

If a site blocks Training, a multi-purpose crawler combining search and training may be blocked entirely — even for its search functions. Cloudflare argues this levels the playing field. Big incumbents currently enjoy an unfair advantage: sites risk losing search visibility if they block training data collection from the same bot.

The company strongly encourages bot operators to separate their crawlers by purpose for greater transparency. Smaller players without established relationships may face a practical ultimatum: split your bots or risk losing access to a significant portion of the web (potentially affecting visibility on domains representing substantial traffic share). Larger search engines may negotiate separate agreements.

Additional Tools: BotBase and Enhanced robots.txt

For Enterprise Bot Management customers, Cloudflare is launching BotBase — a searchable database and dashboard view of all known verified bots and agents. It provides detailed classifications, behaviors, and filtering options, with plans to evolve it into a full control center later in 2026.

Cloudflare is also expanding support for Content Signals in robots.txt with a new optional `use` field that specifies acceptable content usage levels:

`immediate` — Interact but store/reuse nothing.
`reference` (default) — Index, excerpt, and link back.
`full` — Summarize and reproduce.

This gives publishers more expressive ways to signal preferences beyond simple allow/deny. Verified bots that violate these signals risk losing their status.

Broader Implications for the Web Ecosystem

Meme of the Day: Cloudflare's Ironic Pivot – From Shielding Sites to Scraping Them Cloudflare’s updates reflect a maturing view of AI traffic. Not all automation is equal: some drives value back to creators through traffic or compensation, while other uses extract without reciprocation. By empowering even free-tier users with granular controls and pushing for bot separation, the company aims to reduce the “Faustian bargain” many small sites face — allow everything or disappear from search.

This could accelerate the trend toward specialized crawlers and direct compensation models (via Cloudflare’s Pay-Per-Crawl or similar mechanisms). It also puts pressure on major players to increase transparency.

As AI becomes embedded in search, agents, and training pipelines, Cloudflare’s message is clear: website owners should have the tools to set their own rules. “Your site, your rules” is more than a slogan — it’s becoming a practical reality across the entire Cloudflare network.

For full details, see the official announcement on the Cloudflare Blog. Site owners are encouraged to review their settings ahead of the September 15 default changes.