Cloudflare Declares the End of the Crawler Black Box: AI Training Data Acquisition Enters a Mandatory Paid Era
On September 15, 2026, Cloudflare began blocking AI training and Agent crawlers by default on ad-bearing pages for newly onboarded domains while allowing search crawlers through, and introduced tiered content-use permissions. The move shifts crawler economics from pay-per-crawl to pay-per-use and forces AI companies into licensing, payment, or clean-data alternatives.