Cloudflare is changing the way we get information. It is doing it quietly, one settings migration at a time.
On September 15, Cloudflare began changing the recommended setting for AI training crawlers from Block to “Disallow AI Training” — a distinction that lets major search crawlers such as Applebot, Googlebot and Bingbot continue indexing a site for search while instructing them not to use the content for model training. Site owners who previously ran the blunt “Block AI Bots” or “Managed robots.txt” switches are being migrated automatically over the coming week to three separate controls — Search, Training, and Agent. The migrated default is telling: Search gets Allow, Training gets Disallow, and Agent gets Block on pages that serve ads. Owners who never touched the settings stay on Allow. Nothing is forced. Everything is nudged.
The prior thread
This is not a new direction. It is the next installment of one. On July 1, 2025, Cloudflare’s first Content Independence Day declared “no AI crawl without compensation.” On July 1, 2026, its second announced the three-classification scheme and the September 15 defaults above. I wrote about the same July announcement’s other half here under the title “Cloudflare is Helping to Paywall the Internet”: the Monetization Gateway, the x402 protocol, and the case that the web’s 30-year bargain — content for human attention — is being replaced by content-for-payment when the consumer is a machine.
Why a settings migration is a step toward 402
The crawler-controls change looks like product management. It is the load-bearing step. Separating crawlers into Search, Training, and Agent forces every machine client to carry a declared purpose the network can act on, and Cloudflare enforces the most restrictive applicable policy when one bot serves several purposes — which means mixed-use crawlers like Googlebot get blocked wherever Training is blocked. The pressure this puts on AI companies is to split their bots into separate, verifiable identities, tracked in Cloudflare’s BotBase registry of known bots and agents. A machine that declares what it is today is one protocol away from a machine that declares what it will pay tomorrow. Cloudflare has already begun that hand: its AI Crawl Control docs now carry a beta Pay Per Crawl feature — site owners set a price, select which verified crawlers to charge, and receive payouts. That is the 402 economic handshake arriving over conventional rails while x402 settles in stablecoins. Not the finished paywall. The plumbing a paywall stands on: identity, classification, and edge enforcement. A concrete step toward 402.
What changes for information itself
For anyone whose work depends on pulling information off the Web — research desks included — the practical effect is that the default path now runs through a filter with three lanes. The training commons is being carved out of the search commons by default, not by lawsuit. Agentic traffic, the lane that matters most for what AI will do rather than what it has read, is blocked by default where ads pay the bills. Information will still flow; it will flow in declared categories, at prices, through an edge that knows who is asking.
What to watch
The migration is complete when the old Block AI Bots switch disappears from the dashboard. Watch whether BotBase verification survives scrutiny from the AI companies it sorts, and watch the moment a publisher moves a topic from Disallow to a price — the first price printed is the number everyone else benchmarks to. Blocking is a defensive move. Classification is the precondition for an economic one.