JoinFinder: A Multi-Modal Architecture for Historical Manuscript Page Identification
JoinFinder is a multimodal deep learning framework that leverages manuscript identity as a scalable supervised proxy to learn robust embeddings for identifying and retrieving historical manuscript fragments, achieving superior performance in discovering candidate joins across the Cairo Geniza corpus without requiring explicit join annotations.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
History is often written in fragments. For centuries, scholars have pieced together ancient manuscripts by physically matching torn edges or comparing handwriting styles, a laborious process that relies on the eye and intuition of experts. When a single book is scattered across different libraries, or when a page is damaged beyond recognition, the task of finding its missing pieces becomes nearly impossible. This challenge is particularly acute with the Cairo Geniza, a vast collection of Jewish manuscript fragments stored in the genizah (a storage room) of a synagogue in Cairo. Over the last century, these fragments have been dispersed to institutions worldwide, often cataloged separately without any indication that they once belonged to the same book. The sheer volume of material—hundreds of thousands of pages—makes manual reconstruction unfeasible, leaving countless historical connections hidden in plain sight.
To solve this, researchers need a way to teach computers to recognize the unique "fingerprint" of a manuscript. This fingerprint is not just the text written on the page, but the physical characteristics of the document itself: the texture of the paper, the layout of the lines, the specific shape of the letters, and the way the ink sits on the surface. While computers have become excellent at reading text, they have struggled to understand these subtle visual and structural clues, especially when the documents are old, damaged, or written in different languages. The goal is to create a system that can look at a fragment and instantly find other fragments that share its history, even if they are stored in a different country or look slightly different due to age and lighting.
A team of researchers at Tel Aviv University has developed a new approach to this problem called JoinFinder. Instead of trying to teach a computer to find specific "joins" or connections between pages—a task for which there are very few examples to learn from—they taught the system to recognize the identity of entire manuscripts. They used a large collection of known manuscripts as a training ground, showing the computer thousands of pages and asking it to group them by which book they came from. By learning to distinguish one manuscript from another, the system inadvertently learned the subtle visual and structural details that make each manuscript unique. Once trained, the system could be applied to a completely new set of fragments it had never seen before, searching for matches based on the patterns it had learned.
The researchers built a system that looks at a manuscript page in three different ways simultaneously. First, it examines the overall appearance of the page, looking at the layout, the density of the writing, and the texture of the material. Second, it zooms in on individual letters, analyzing the specific shapes and strokes that characterize the handwriting. Third, it reads the text itself, using optical character recognition to understand the words and their arrangement. The system does not treat these three sources of information as equal; instead, it weighs them based on their reliability. If a page is too damaged to read the text clearly, the system relies more heavily on the shape of the letters or the overall look of the page. This flexibility allows it to handle the messy, incomplete reality of historical documents.
The results of this approach were tested on a massive collection of Geniza fragments. When the system was asked to find pages that belonged to the same manuscript, it succeeded in placing the correct match within the top ten results for nearly half of the queries. When the researchers looked for pages that shared the same catalog number or shelf location, the success rate rose to over 60 percent. More importantly, the system found connections that the existing catalogs had missed. It identified pairs of fragments from different libraries that shared the same handwriting and layout, suggesting they were once part of the same book. These discoveries were confirmed by human experts, proving that the computer had found genuine historical links that were previously unknown.
The study also revealed that the system works even when the conditions change drastically. It successfully matched black-and-white images of manuscripts, even though it was trained primarily on color images. It also found connections in Arabic manuscripts, despite being designed with Hebrew letters in mind. This suggests that the system learned the fundamental visual language of manuscripts—the way the ink flows, the rhythm of the lines, and the structure of the page—rather than just memorizing specific letters or colors. The researchers found that the most important clues came from the shapes of the letters and the overall layout of the page, while the text content provided helpful but secondary support.
This work represents a significant step forward in the digital reconstruction of history. By using the identity of a manuscript as a proxy for learning, the researchers created a tool that can scale to millions of pages, a task that would take human scholars centuries to complete. The system does not replace the expert; rather, it acts as a powerful filter, sifting through vast archives to present the most promising candidates for human review. It turns the impossible task of finding a needle in a haystack into a manageable search, allowing scholars to focus on the most likely connections. The success of JoinFinder suggests that with the right approach, the scattered fragments of the past can be brought back together, revealing the full stories they once told.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.