The Open Data Institute reveals that popular large language models frequently provide unreliable information regarding essential public services like health and benefits. By testing over 22,000 user-like prompts, the study highlights that AI systems often deliver confident but incorrect advice, failing to acknowledge uncertainty or burying key facts in lengthy, misleading responses. This poses significant risks for citizens relying on these tools for critical life decisions. This research is highly relevant to the open data community because it underscores the urgent need for transparency, accountability, and open evaluation standards in public sector AI. The findings challenge the assumption that larger, proprietary models are superior, suggesting instead that smaller, cost-effective solutions may be more suitable if rigorously tested. It emphasizes that trustworthiness must be measured from the user’s perspective, requiring independent benchmarks and public testing rather than reliance on vendor claims or technical benchmarks alone. Furthermore, the article warns against the growing crisis of unverified, AI-generated data poisoning future training sets, which threatens the fundamental reliability of language models. As regulatory frameworks lag behind rapid technological adoption, governments are urged to avoid vendor lock-in and prioritize open, incremental development. The core implication is that without robust open data practices and strict verification mechanisms, the integrity of public information services could be severely compromised by automated hallucinations.
Source: computerweekly.comPublished on 2026-02-12