LongDocBench: Benchmarking TOC Hierarchy and Contextual Relationship Recovery in Long Documents
This paper introduces LongDocBench, a new benchmark comprising 85 real-world long documents with human-verified annotations for Table-of-Contents hierarchy and contextual relationship recovery, to address the limitations of existing benchmarks in evaluating document-level structure and to demonstrate that recovering these structures significantly improves downstream long-document question-answering performance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine trying to understand a massive library by looking at only one bookshelf at a time. You could read every word on that single shelf, identify the titles, and even understand the layout of the books right in front of you. But if you need to find a specific argument that spans three different sections, or trace how a chart on page fifty relates to a footnote on page two hundred, looking at isolated shelves leaves you lost. This is the current challenge in the field of document intelligence, where computers are taught to read and understand visual documents like reports, textbooks, and academic papers. For years, researchers have focused on teaching machines to recognize text, formulas, and tables on individual pages with high accuracy. They have become very good at this local work. However, real-world documents are not just collections of isolated pages; they are complex structures where information is woven together across hundreds of pages through deep hierarchies of headings and connections between figures and their explanations.
A new study introduces a way to test whether computers can truly understand these long, connected documents. The researchers built a benchmark called LongDocBench, a collection of eighty-five real-world documents including financial reports, textbooks, and academic papers. These documents span over two and a half thousand pages in total, with some individual reports stretching to over one hundred pages. The team did not just gather these files; they painstakingly created a "gold standard" map for each one. Human experts manually traced the entire structure, identifying every heading and how it fits into a parent-child hierarchy, much like a table of contents that spans the whole book. They also mapped out thousands of specific relationships, noting when a chart or image was linked to a caption, a note, or a source that might be located far away on a different page. This careful human work created a reference point to see if current computer systems could do the same job.
When the researchers tested the best available document parsers against this human-made map, the results revealed a significant gap. The computers performed exceptionally well at the page level, correctly identifying text and local layouts with high accuracy. Yet, when asked to reconstruct the full document structure, they struggled. The systems failed to correctly organize the deep hierarchies of headings, often collapsing the complex tree of chapters and sub-chapters into a flat list. Similarly, they had trouble connecting figures to their distant notes or sources, missing the typed links that give context to the data. Even the most advanced models managed to recover only about half of the correct structural relationships, a stark contrast to their near-perfect performance on single-page tasks. This suggests that being able to read a page does not automatically mean a machine can understand the document as a whole.
The study also explored why this structural understanding matters. The researchers used the human-verified maps to answer questions about the documents, comparing the results to a system that had no structural map at all. When the system used the correct, human-verified hierarchy and relationships, its ability to answer questions improved dramatically. It was not just a small improvement; the accuracy jumped significantly, showing that knowing how the document is organized helps the computer reason through complex queries. However, when the system relied on the structures it had recovered on its own, the improvement was much smaller. The computer's own attempts to build the map were not accurate enough to unlock the full potential of the information.
Ultimately, this work highlights that the next frontier in document intelligence is not just reading words, but understanding the architecture of information. The researchers have released their benchmark and the human-verified maps to the public, providing a clear target for future development. They have shown that while machines are excellent at seeing the trees, they still need to learn how to see the forest. Until computers can reliably reconstruct the deep hierarchies and cross-page connections that humans take for granted, their ability to truly understand long, complex documents will remain limited. The path forward requires moving beyond simple page recognition to mastering the intricate, multi-page relationships that define how we organize and retrieve knowledge in the real world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.