Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics
This paper reveals that web-PDF corpus statistics suffer from a significant "unit bias" by reporting metrics per document rather than per token, which obscures the fact that a small fraction of large documents contains the majority of the text and that current truncation policies result in the loss of over half of the corpus's total content.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the vast, chaotic library of the internet, a new kind of book has become essential for teaching artificial intelligence how to think. These are not the web pages we browse with a mouse, but digital documents in a format called PDF, the standard for sharing reports, academic papers, and official forms. For years, researchers building these intelligent systems have treated every digital file as a single, equal unit. If a dataset contains a million documents, they assumed it held a million equal portions of information, regardless of whether one file was a single-page flyer and another was a five-hundred-page encyclopedia. This assumption has shaped how we measure the size and quality of the data used to train the world's most advanced computer models. But a closer look reveals that counting files is a poor way to count words, and the difference between the two units changes everything we thought we knew about the data.
A recent study by Luca Foppiano of the Common Crawl Foundation challenges the standard way researchers describe these massive collections of text. The team examined a specific snapshot of the internet from 2021, a dataset containing nearly eight million web PDFs. While the original creators of this data reported its size in terms of the number of documents, Foppiano and his colleagues decided to measure it by the actual amount of text inside those files. They found a startling imbalance: the documents were not interchangeable at all. A tiny fraction of the files, specifically those that were very long, contained the vast majority of the words. In fact, just over three percent of the documents held half of the entire text, while the remaining ninety-seven percent shared the other half. This means that if you look at the collection by file count, it appears to be a library of short notes and forms, but if you look at it by word count, it is a library dominated by massive, lengthy volumes.
This skew in length has profound consequences for how the data is processed and what is lost along the way. The internet archive that provided these files has a rule to save storage space: it cuts off any file larger than a certain size. For a long time, that limit was one megabyte. The study shows that this rule affected nearly a quarter of the documents in the collection, which sounds manageable. However, because the longest documents were the ones being cut, those truncated files contained nearly two-thirds of all the text. When the researchers tried to reconstruct the missing parts of these cut-off files, they discovered that the damage was often permanent. Using two different standard tools to read the broken files, they found that one tool could recover only about eleven percent of the missing words, while the other recovered barely one percent. For the vast majority of these damaged files, the tools could not extract any text at all, leaving the computer models with a significant gap in their knowledge.
The researchers also looked at what happens when the storage limit is raised, a change that occurred in 2025 when the limit was increased to five megabytes. While this adjustment reduced the number of cut-off files, it did not solve the problem of lost text. Even with the higher limit, nearly thirty percent of the total words in the collection would still be cut off. The study demonstrates that simply increasing the size limit is not enough to save the most valuable content, because the largest documents are often so massive that even a five-megabyte slice is too small to capture their beginning. The researchers concluded that the way we talk about these data collections is misleading. Reporting statistics only by the number of documents hides the reality that a few huge files hold almost all the information. To truly understand what these artificial intelligence systems are learning, we must count the words, not just the files, and recognize that the current methods of storing and filtering the internet are discarding the bulk of the text we need.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.