Institutional Books - Visual Elements: An open-source pipeline for extracting, classifying, deduplicating, and captioning visual elements from digital book collections

Libraries are vast repositories of human history, but for decades, the digital versions of their books have told only half the story. When institutions scan old volumes to make them searchable, they have traditionally focused on the text, using software to read the words and turn them into data. This process, known as optical character recognition, has been a triumph, allowing researchers to search through millions of pages in seconds. However, the images within those books—the illustrations, photographs, engravings, and decorative borders—have largely remained invisible to machines. These visual elements hold a unique layer of context and nuance that text alone cannot capture, yet they have been difficult to study at scale because they were not structured in a way computers could easily understand. As artificial intelligence systems become more capable of "seeing" and interpreting the world, a significant gap has emerged: these systems are trained mostly on modern images from the internet and often struggle to make sense of historical visual culture. To bridge this gap, researchers need a massive, diverse collection of historical images that is organized and ready for machines to learn from.

A team of researchers from Harvard University has addressed this challenge by building a new, open-source system designed to find, sort, and describe visual elements inside nearly one million digitized books. Their work, detailed in a technical report, creates a pipeline that acts like a highly skilled librarian who can scan a book, spot every picture, figure out what it is, and write a short description for it, all without human intervention. The result is a dataset containing over 22 million visual elements extracted from 983,004 volumes. This collection includes everything from scientific diagrams and musical scores to portraits and decorative art, each tagged with information about its type and, in many cases, a computer-generated caption. By making this data available, the team hopes to help artificial intelligence models learn to understand historical materials better, creating a cycle where better tools lead to more data, which in turn leads to even better tools for exploring our shared past.

The process begins with the most basic task: finding the pictures. The researchers developed a system that scans the digital pages of the books to locate any region that contains an image. Because the collection is so vast, containing hundreds of millions of pages, the system had to be incredibly efficient. They trained a specialized computer model to recognize visual elements by showing it thousands of examples of pages with and without pictures. This model learned to ignore the text and focus on the visual parts, drawing a box around every illustration, photo, or diagram it found. In the final run, the system processed the entire collection and identified over 28 million potential images. To ensure high quality, the researchers applied strict filters to remove false alarms, such as smudges or scanning errors, leaving a final set of 22.6 million distinct visual elements.

Once the images were found, the next step was to understand what they were. The system sorted the 22.6 million images into five main categories: general illustrations and photographs, musical notation, decorative elements like bookplates, charts and graphs, and scanning artifacts like stray fingers or paper clips that accidentally appeared in the scans. The researchers trained another model to make these distinctions, teaching it to tell the difference between a musical score and a page of text, or between a decorative border and a scientific diagram. The system proved highly accurate, correctly identifying the vast majority of images. For example, it could distinguish musical notation with nearly perfect confidence, though it found some categories, like decorative borders, slightly more difficult to separate from other types of illustrations. This sorting process is crucial because it allows researchers to ask specific questions, such as "show me all the charts from the 18th century" or "find every instance of a specific bookplate."

After sorting, the team tackled the problem of duplicates. In large digitization projects, the same image often appears many times, either because it was used in multiple editions of a book or because the scanning process accidentally captured the same page twice. To clean this up, the researchers used two different methods to find matching images. The first method compared the pixel patterns of the images to find exact copies, while the second method looked at the deeper meaning of the images to find near-duplicates that might look slightly different due to scanning variations. By combining these approaches, they identified and grouped together millions of repeated images, ensuring that the final dataset represented unique visual content rather than just a list of every single time an image appeared. This step reduced the total number of items by about 20 percent, leaving a cleaner, more valuable collection for study.

The final and most experimental part of the project involved teaching the system to write descriptions for the images. Using a large language model, the researchers asked the computer to generate a short caption for each image, explaining what it showed. To help the computer understand the context, they fed it the text from the surrounding page along with the image itself. The system was instructed to be precise and to avoid guessing, describing only what was clearly visible. While the researchers noted that these captions are experimental and should be used with care, they successfully generated nearly 18 million descriptions. These captions, written in the original language of the book whenever possible, provide a new way to search for visual content, allowing users to find images based on their content rather than just their file names or categories.

The researchers also paid close attention to the practical side of this massive undertaking, ensuring that the process was efficient and could be repeated by other institutions. They found that the biggest bottleneck was not the computer's ability to think, but the time it took to load and prepare the images for processing. By optimizing their workflow, they managed to process the entire collection in a matter of weeks using standard computing resources. The team released the entire pipeline, the models they trained, and the resulting dataset as open-source tools, inviting other libraries and researchers to use and improve upon their work. They also included clear warnings about the limitations of their data, noting that some images contain historical biases or harmful language that reflects the time in which they were created, and that the computer-generated captions are not perfect.

This project represents a significant step forward in making the visual history of human culture accessible to both humans and machines. By turning a massive, unstructured collection of book images into a searchable, classified, and described dataset, the researchers have provided a new foundation for studying the past. The dataset includes over 22 million visual elements, offering a rich resource for training artificial intelligence to understand historical contexts and for enabling new forms of digital humanities research. The work demonstrates that with careful planning and the right tools, the visual treasures hidden inside millions of old books can be brought to light, offering fresh insights into the stories they tell.