OpenAI, Google, Microsoft and Anthropic are joining forces to give website owners more control over AI training — here's how to opt out

Major technology firms, including OpenAI, Google, Microsoft, and Anthropic, have committed to respecting a standardized opt-out signal within website robots.txt files. This agreement allows site owners to explicitly prohibit AI crawlers from accessing their content for model training purposes. By establishing a shared protocol, these companies are attempting to balance the insatiable data hunger of artificial intelligence with the rights of creators and website operators to control how their digital assets are harvested. The article highlights that while this new standard is a significant step toward ethical data usage, it is not a complete solution. Current web infrastructure relies on a "defense-in-depth" strategy, combining robots.txt directives with other measures like account requirements, IP limiting, and CAPTCHAs. However, these additional methods often harm user experience by creating friction for legitimate visitors or failing against sophisticated bots that ignore rules or use rotating IP addresses, leaving website owners vulnerable to continued unauthorized scraping. This development is highly relevant to the open data community because it formalizes the tension between open web accessibility and proprietary AI training needs. For researchers and open data advocates, it establishes a precedent for consent-based data extraction, potentially shaping future standards for how public knowledge is collected. It underscores the necessity of transparent, respectful data practices in an ecosystem where the internet serves as the primary archive of human knowledge, urging a shift from unchecked harvesting to negotiated access.

Source: techradar.com
Published on 2026-11-04

OpenAI, Google, Microsoft and Anthropic are joining forces to give website owners more control over AI training — here's how to opt out

Related news