Google joins OpenAI and Anthropic in promising new scraping controls - Engadget

Major tech companies have agreed to honor a unified robots.txt directive, allowing publishers to easily control whether their content is scraped for generative AI training. This standardizes what was previously a fragmented process, requiring site owners to identify specific crawlers individually by adopting a common signal proposed at the IETF. By introducing a dedicated user agent for AI training, publishers can now block data collection broadly while retaining control over specific sections, such as press kits. Crucially, this mechanism only prevents future scraping; it does not remove data already collected, nor does it impact standard search engine indexing, ensuring a balance between AI development and publisher rights. This development is vital for open data discourse as it establishes a transparent, standardized method for data provenance and consent. It empowers content creators to define their data sharing preferences explicitly, setting a precedent for ethical data usage in AI development. This shift promotes accountability and clarifies the boundaries of public information utilization in the rapidly evolving landscape of artificial intelligence.

Source: engadget.com
Published on 2026-11-01

Google joins OpenAI and Anthropic in promising new scraping controls - Engadget

Related news