Cloudflare Rolls Out Permission-Based Web Scraping For AI Bots

Cloudflare has reversed the default policy for AI web crawlers, shifting from open access to a permission-based model. This update grants website owners explicit control over whether their content is scraped by artificial intelligence systems. By requiring AI companies to seek consent before accessing data, the change aims to correct the historical imbalance where creators lost revenue and traffic due to unauthorized data extraction for training models. The policy mandates that AI bots clearly identify themselves and declare their intended use for collected data, such as training or inference. This transparency has been strongly endorsed by major publishers and media groups, who view it as a crucial step toward establishing fair value exchanges. The move supports emerging economic models where creators can negotiate compensation, helping to sustain quality journalism and independent content creation in an era where AI dependency on public data is growing. This shift is vital for open data governance, as it challenges the unchecked "free-for-all" scraping that currently underpins many large language models. By prioritizing creator consent and accountability, Cloudflare’s approach offers a framework for balancing innovation with copyright protection. It highlights the urgent need for standardized protocols that respect intellectual property rights, ensuring that the advancement of AI technology does not come at the expense of the economic viability of the open web and its contributors.

Source: medianama.com
Published on 2025-07-03

Cloudflare Rolls Out Permission-Based Web Scraping For AI Bots

Related news