A.I. Companies Are Running Out of Training Data: Study

The rapid expansion of artificial intelligence models is colliding with a critical shortage of high-quality training data, driven by a surge in restrictions from web sources. Many websites are now blocking automated crawlers, significantly reducing the accessible pool of public content for both commercial and academic developers. This "crisis in data consent" suggests that the free flow of information underpinning current AI advancements is becoming increasingly unsustainable. This scarcity is forcing major technology companies to pivot from open collection to costly proprietary agreements and novel sourcing methods. To secure necessary datasets, firms are entering million-dollar partnerships with media publishers, acquiring entire libraries, or exploring synthetic data generation as a future alternative. These market shifts indicate a transition where data access is becoming a controlled commodity rather than an open resource, potentially altering the competitive landscape by favoring well-funded incumbents over smaller developers. For the open data community, this trend highlights a fundamental tension between proprietary data accumulation and the democratization of information. The move toward closed ecosystems and paid data access threatens to undermine the open principles that have historically fueled rapid innovation. It underscores the urgent need to advocate for sustainable, ethical data practices that balance intellectual property rights with the broader societal benefit of accessible, high-quality information for research and development.

Source: observer.com
Published on 2024-07-20

90 per cent of Israelis have had their children vaccinated »

Open Data

Related news