Redistributing Lossless-Compression Test Corpora: An Audit of Nine Corpora and a File-Level Publication Procedure

For decades, scientists and engineers have relied on a small, aging collection of digital files to test how well computers can shrink data without losing a single bit of information. These files, assembled between the late 1980s and the mid-2000s, act as a universal yardstick. Because they are small, stable, and easy to find, researchers can compare their new compression tools against results from thirty years ago. However, a quiet problem has grown around these files. While the data itself is public, the legal permission to copy, share, or modify them is often missing. In the world of digital research, if you cannot prove you have the right to share a file, you cannot build a new, reliable test set on top of it. This uncertainty creates a fragile foundation: a benchmark built on files that might legally belong to someone else cannot be safely distributed, mirrored, or updated.

This uncertainty is not just a legal technicality; it is a barrier to progress. When researchers want to create a modern test set to replace these old files, they face a daunting task. They must check the legal status of every single file, one by one, to ensure it can be shared. There was no established method for doing this, and no one had audited the most famous collections to see what they actually told users about their rights. Without clear answers, the field risks building new science on a foundation that could be pulled out from under it at any moment.

Eugen Spivak, an independent researcher, set out to fix this gap. He first conducted a thorough audit of nine widely used test collections that form the backbone of data compression research. He examined their public pages and download archives to see what information they provided about ownership, origin, and the specific rules for using each file. The results were stark. Out of the nine collections he reviewed, eight provided no license statement for any of their files. In total, 76 classic test files were circulating without any stated permission for redistribution. While a few collections offered vague descriptions of where files came from, none provided the specific legal text needed to prove a file could be shared. Furthermore, none of these collections recorded the exact computer commands used to create or modify the files, and none offered a permanent, unchanging link to the data that would survive as the internet changed.

To solve this, Spivak developed a new, rigorous procedure for publishing data. Instead of treating a whole collection as a single unit with one blanket license, his method treats every single file as its own legal entity. For each file, the researcher must gather a "lineage record." This record includes the direct link to where the file came from, a digital fingerprint of the original source to prove it hasn't been altered, and the exact computer command used to generate the file if it was created by software. Crucially, the researcher must also find and save the specific sentence from the source's license page that explicitly permits redistribution, along with the date it was read. This evidence is saved so that even if the original website changes or disappears years later, the proof of permission remains.

The procedure includes a strict safety mechanism: a file is only allowed into the public record if two independent reviewers agree on its status. The first reviewer compiles a list of candidates, and a second reviewer, who does not see the first list, checks every single file again from scratch. This "blind" second review is designed to catch errors that a simple glance at a license name might miss. When Spivak applied this method to a new set of game-related data—specifically textures, 3D models, and audio files used in video games—the process worked exactly as intended. From an initial pool of 774 candidate files, the second reviewer removed 30 files, a rejection rate of nearly 4 percent. These were not files with the wrong license name; they were files with subtle issues, such as unclear author credits, photos containing identifiable people, or sources where the rights holder could not be confirmed. A single-review process would have published these 30 files, only for them to be withdrawn later, damaging the trust in the entire collection.

The final result was two public records of game assets, totaling over one billion bytes of data. One record contained 744 files that were free for anyone to use without restriction, and another contained 13 files that required simple credit to the original creator. The publication included a manifest, a detailed list that acts as a map for the data, showing exactly how every file was made and what legal permission covered it. To ensure the data could be trusted, the researcher also tested whether the digital archives could be rebuilt exactly as they were. He discovered that an archive's digital fingerprint is incredibly sensitive; even a tiny, invisible difference in how the computer software writes the file—such as a setting that changes based on whether the computer is running Windows or a different system—would change the fingerprint. This meant that for data to be truly reproducible, the publication must list every single setting used to create the archive, not just the files inside it.

After the data was published, a follow-up check found minor errors in the documentation, such as incorrect tool names or missing details in the list of commands. These were corrected by releasing a new version of the record, rather than silently changing the original files. This approach ensures that anyone looking back at the data can see exactly what was available at any point in time. The study concludes that it is possible to publish a test corpus with clear, file-by-file legal evidence, but it requires a significant investment of time and a second, independent review to catch the subtle errors that slip through a single check. The classic collections that have been used for decades remain in a state of legal uncertainty, but this new procedure offers a clear path forward for anyone wishing to build the next generation of reliable, shareable data for scientific research.