Institutional Books - Enriched Text: A customizable multilingual open-source pipeline for denoising, deduplicating, and annotating OCR text at scale
This paper introduces "Enriched Text," a customizable, open-source multilingual pipeline that preserves metadata and enables user-controlled filtering of the 983,004-volume Institutional Books: Harvard Library collection by annotating OCR text with language detection, deduplication clusters, and quality scores rather than applying aggressive global preprocessing.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a library that holds nearly a million books, spanning centuries of human thought, written in hundreds of different languages. For decades, these volumes have been scanned by machines, turning physical pages into digital text. But this digital text is often messy. It contains the noise of the scanning process: page numbers repeated at the bottom of every page, headers that run across the top, and words broken awkwardly at the end of a line. For a computer trying to learn from this text, such noise is a major obstacle. Standard methods for cleaning this data often involve throwing away large chunks of information, such as removing all text that isn't in English or deleting any paragraph that looks similar to another. While this makes the data cleaner, it also strips away the rich, multilingual history and the structural context that makes a book a book.
A team of researchers at Harvard Law School has taken a different approach. Instead of discarding the messy parts, they have built a system that carefully organizes them. They call their result "Enriched Text." Rather than producing a single, simplified stream of words, their pipeline keeps the original text intact but adds a layer of digital annotations. Think of it as adding a set of invisible labels to every paragraph, telling a computer exactly what that paragraph is, what language it is in, and whether it is a duplicate of something else. This allows anyone using the data to decide for themselves what to keep and what to ignore, rather than having those decisions made for them by a rigid filter.
The project began with a massive collection of 983,004 volumes from the Harvard Library, originally digitized through a partnership with Google. This collection contains roughly 242 billion units of text, known as tokens, covering about 250 different languages. The researchers wanted to make this collection usable for modern artificial intelligence without losing the integrity of the source material. They faced a tension: standard computer science practices for preparing data often aggressively filter out non-English text and delete anything that looks like a copy to save space. However, for a historical library, deleting non-English text would mean losing half the collection, and deleting duplicates would mean breaking the continuity of long books.
To solve this, the team created a processing pipeline that acts like a meticulous librarian. First, they cleaned up the text by fixing common errors introduced by the scanning process. They removed the page numbers and headers that appear at the top and bottom of pages, which often get mixed into the middle of sentences. They fixed words that were split across lines with hyphens, rejoining them correctly. They also separated the "end matter"—the front and back sections of a book like the table of contents, index, and bibliography—from the main story or argument in the middle. Crucially, they did not delete these sections. Instead, they labeled them so that a user could choose to read just the main text or study the index separately.
The system then broke the books down into smaller, meaningful pieces. It identified individual sentences and grouped them into paragraphs based on the topic being discussed, rather than just where the page turned. This is important because a single page might contain two different topics, or a single topic might stretch across many pages. By grouping text by its subject matter, the researchers made it easier for computers to understand the flow of ideas. They also identified paragraphs that appeared in multiple books. Instead of deleting these duplicates, they marked them. One version of the paragraph is highlighted as the "representative" copy, while the others are tagged as duplicates. This preserves the history of how ideas were reused and shared across different books and languages.
One of the most sophisticated steps involved measuring the quality of the text. The researchers used a small computer model to read every paragraph and assign it a score based on how predictable and fluent the language was. This score, called "bits-per-byte," helps identify paragraphs that are likely to be high-quality prose versus those that are just garbled noise from a bad scan, such as a table that was read as a single block of text. This allows users to filter out the worst quality text without having to run the expensive calculation themselves.
The final result is a dataset containing 217 billion tokens, organized into over 1.3 billion annotated paragraphs. The researchers found that this approach successfully cleaned the text while preserving its complexity. For example, they discovered that while English text made up the largest portion of the collection, the system correctly identified that many non-English books contained significant amounts of Latin or Greek, often used for quotes or scholarly references. They also found that the text became more predictable to modern computers as time went on, with books from the 19th and 20th centuries sounding more like the language we use today than the books from the 17th century.
The team released the entire pipeline as open-source software, meaning anyone can use it to process their own collections of books. They also provided a small tool to help users parse the data, allowing them to strip away the annotations if they want a simple list of words, or to keep the annotations if they want to study the structure of the books. By choosing to annotate rather than delete, the researchers have created a resource that is both cleaner for machines and richer for human study. They have shown that it is possible to make historical data accessible to modern technology without sacrificing the nuance and diversity that make that history valuable. The work suggests that the future of data processing lies not in throwing away the messy parts, but in learning how to organize them.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.