Perplexity accused of breaking a major online AI scraping rule - but it says it has done nothing wrong
Cloudflare discovered that Perplexity AI systematically ignored standard web crawling rules, including robots.txt directives, to scrape data from protected and hidden test sites. The company employed obfuscation techniques, such as rotating IP addresses and impersonating legitimate browsers, to evade detection and bypass security measures like Cloudflare’s blocks. This behavior contrasts sharply with responsible operators like OpenAI, which adhere to transparent identification and respect site restrictions. By disregarding explicit signals, Perplexity undermined the foundational etiquette of web data harvesting, prompting Cloudflare to remove their bots from verified lists and implement stricter detection heuristics. For open data advocates, this incident highlights the critical tension between unrestricted AI training and ethical web access. It underscores the urgent need for transparent, compliant data collection practices to ensure the sustainability of the open web. Ignoring established norms threatens the integrity of public information infrastructure and erodes trust among data providers.
Source: techradar.comPublished on 2025-08-06
Related news
- Perplexity says Cloudflare's accusations of 'stealth' AI scraping are based on embarrassing errors
- Cloudflare Blocks Perplexity AI Bots Amid Scraping Allegations
- Some people are defending Perplexity after Cloudflare ‘named and shamed’ it | TechCrunch
- From Open Data to AI-Ready Data: Building the Foundations for Responsible AI in Development
- OpenAI's first new open-weight LLMs in six years are here