Google, Anthropic and others to support new AI crawling opt-out standard - SiliconANGLE

Major technology companies are collaborating to establish a standardized web crawling protocol that empowers website operators to control how their content is used for AI training. By introducing a unified method within the existing Robots Exclusion Protocol, this initiative simplifies the process of blocking AI scrapers. Previously, site owners had to manually add individual entries for every AI provider and specific scraper, a cumbersome task that often led to inconsistent protections. This new standard allows publishers to disallow all generative AI training data collection with a single entry, significantly reducing administrative burdens while enhancing content sovereignty. The proposed framework also distinguishes between training data collection and real-time user interactions. It specifically targets models engaged in generative tasks, excluding those performing exclusive non-generative functions. Additionally, Microsoft plans to introduce "AI Grounding" settings, which give publishers control over whether their websites are accessed to answer immediate user questions versus being used for model improvement. This granularity ensures that consent can be managed according to the specific nature of the AI’s interaction with the website, balancing innovation with operator rights. This development is crucial for open data advocacy as it highlights the tension between the vast data requirements of AI development and the rights of content creators. It demonstrates a move toward more ethical and transparent data practices, ensuring that the open web remains accessible for human use while allowing publishers to opt out of automated scraping for commercial AI training. By formalizing these controls, the industry is establishing precedents for digital consent, reinforcing the importance of respecting source data integrity in the era of generative artificial intelligence.

Source: siliconangle.com
Published on 2026-11-01

Google, Anthropic and others to support new AI crawling opt-out standard - SiliconANGLE

Related news