Google's Illyes: Robots.txt Covers Crawling, Not AI Training

Gary Illyes clarifies that robots.txt functions strictly as an access preference protocol, dictating which URLs crawlers may request rather than governing how that data is utilized afterward. This distinction is crucial for website owners seeking to control AI training; while standard directives prevent specific bots from accessing content, they do not inherently stop authorized crawlers from using the retrieved information for model training. To restrict data usage, publishers must currently rely on specific crawler tokens or wait for standardized usage controls to become widely adopted. The article highlights the tension between current technical realities and emerging standards. Although a new IETF draft proposes a "Content-Usage" rule to explicitly define training preferences, this mechanism is not yet a web standard. Currently, controls like Google’s "Google-Extended" token operate based on proprietary documentation rather than universal protocol, meaning their effectiveness is limited to specific ecosystems. This gap leaves website administrators without a universal, standardized method to decouple access permissions from data usage rights across the broader internet. This discussion is highly relevant to open data because it underscores the ambiguity surrounding data provenance and reuse in the age of artificial intelligence. Without standardized usage preferences, it becomes difficult to enforce ethical data handling or ensure that open datasets are used in accordance with original publishing intents. Furthermore, the advice to use percent-encoded paths for non-English URLs emphasizes the need for robust, interoperable technical practices to ensure consistent interpretation across diverse crawling systems.

Source: searchenginejournal.com
Published on 2026-10-23

Google's Illyes: Robots.txt Covers Crawling, Not AI Training

Related news