Summing columns in remote Parquet files using DuckDB

This article demonstrates the practical utility of open datasets for analyzing large-scale digital infrastructure. By leveraging the publicly available Midjourney messages dataset on Hugging Face, researchers can perform deep investigative analysis on commercial platforms without needing direct access to proprietary backend systems. The dataset’s open availability allows for transparent scrutiny of how AI-generated content impacts external hosting providers, highlighting the importance of open data in understanding the broader ecological and economic costs of AI proliferation. The core finding reveals that the total size of Midjourney images hosted on Discord’s CDN amounts to approximately 148 terabytes, incurring substantial hosting costs for the platform. This conclusion was reached not by downloading the entire multi-terabyte collection, but by using DuckDB to query remote Parquet files efficiently. This technique extracts only the necessary metadata regarding file sizes, proving that valuable aggregate insights can be derived from open datasets with minimal bandwidth consumption and computational overhead. This approach is highly relevant to the open data community as it showcases advanced, efficient methods for analyzing massive, remote datasets. It illustrates how tools like DuckDB can utilize HTTP range requests to sum large columns without retrieving the full volume of data. This serves as a best practice example for data scientists working with open data, emphasizing that significant analytical value can be extracted through clever querying techniques rather than full-scale downloads, thereby promoting sustainable and scalable open data analysis.

Source: til.simonwillison.net
Published on 2023-11-18

Related news

SEDENA debe informar al INAI »