GLaDViT: interpretable gated fusion of frozen vision, text and layout features for document image classification
GLaDViT is a lightweight, interpretable document classification model that achieves high accuracy by fusing frozen CLIP, BERT, and Tesseract-derived layout features through a trainable gated head, enabling exact attribution of decisions to individual modalities while demonstrating that layout features provide significant complementary value despite receiving low gate weights.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the vast digital archives of modern businesses and government agencies, millions of scanned pages sit waiting to be sorted. These are not just images; they are letters, invoices, reports, and forms that need to be routed to the right department, indexed for future search, and processed automatically. For a computer to do this work, it must first recognize what kind of document it is looking at. This task, known as document image classification, has traditionally relied on massive, complex artificial intelligence systems. These systems are often trained on millions of pages to learn the subtle visual and textual patterns that distinguish a memo from a scientific report. While powerful, these large models are expensive to run, difficult to update, and often act as "black boxes," making decisions without explaining why. Researchers are now asking whether a lighter, more transparent approach can achieve similar results without the heavy computational cost or the loss of clarity.
A team led by Murat Simsek at Ostim Technical University has proposed a new method called GLaDViT to answer this question. Instead of training a giant neural network from scratch, their approach uses three distinct "eyes" to look at a document, each frozen in place so they cannot change. The first eye is a vision model that sees the page as a picture, capturing the overall shape and layout. The second is a text model that reads the words on the page, understanding the language. The third is a specialized tool that maps the physical boxes where words sit, creating a geometric map of the document's structure. The innovation lies not in these eyes, but in a small, trainable "gate" that sits between them. This gate decides how much weight to give to the picture, the text, and the layout map for every single document it sees, combining them into a final decision.
The researchers tested this system on a corrected version of a well-known dataset called Tobacco-3482, which contains 2,737 pages of documents ranging from advertisements to resumes. They found that the vision model alone could correctly identify the document type about 88 percent of the time, while the text model reached 84 percent. The layout map, when used by itself, was much weaker, succeeding only about 60 percent of the time. However, when the researchers combined the vision and text models, accuracy jumped to nearly 90 percent. Adding the layout map to this mix provided a significant boost, pushing the system's accuracy to 93.7 percent. This improvement suggests that even though the layout map is not strong enough to work alone, it provides crucial structural clues that the picture and text models miss, such as the specific arrangement of fields on a form or the spacing of a letter.
A critical part of this research was understanding how the system makes its choices, a quality known as interpretability. In many complex AI systems, the internal weights that determine importance are often mistaken for explanations, but they can be misleading. The researchers demonstrated that in their system, the gate's decision on how much to weigh the layout map does not tell the whole story. Even though the gate assigned the layout map a very small role—giving it less than 10 percent of the total attention in most cases—the layout information still made a measurable difference in the final answer. By breaking down the math behind the decision, they showed that a small amount of layout information can be highly influential if it aligns perfectly with the specific features of a document. This allows the system to provide a faithful, exact explanation for every decision it makes, showing exactly how much the picture, the words, and the structure each contributed to the final classification.
The study also explored whether a technique called contrastive alignment, which forces different types of data to agree with each other during training, could improve performance. The researchers found that when the main vision and text models are frozen and cannot learn new patterns, this alignment technique actually hurt performance, dropping accuracy by several points. This suggests that trying to force these fixed models to align their internal views is counterproductive when they cannot adapt. The final result is a system that is both highly accurate and transparent. It achieves a success rate of 93.7 percent on the test set, with a statistical confidence interval between 90.2 and 96.0 percent, proving that a lightweight, gated fusion of frozen features can be a powerful and auditable alternative to massive, opaque models. The work confirms that while layout analysis is not the strongest signal on its own, it is an essential piece of the puzzle that, when combined correctly with vision and text, leads to a clearer and more reliable understanding of document types.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.