Wikipedia offers AI developers a training dataset to maybe get scraper bots off its back

Wikipedia faces significant server strain from AI crawlers, prompting the Wikimedia Foundation to launch a structured dataset via Kaggle. This initiative aims to divert bot traffic while providing developers with optimized data for training generative models. The dataset offers freely licensed content, including abstracts and key-value data, formatted specifically for machine learning. Although it excludes references and multimedia, the materials remain available under permissive licenses, ensuring legal usability for AI development. This shift highlights the tension between open access and infrastructure sustainability in the open_data ecosystem. By providing curated data, Wikimedia balances the demand for AI training resources with the need to protect human user experience, setting a precedent for managing digital commons against automated extraction.

Source: engadget.com
Published on 2025-04-18

Wikipedia offers AI developers a training dataset to maybe get scraper bots off its back

Related news