Technical Summary: Locating Failure in Multi-Page Visually Rich Document Understanding

Problem Statement

Multi-page Visually Rich Document Understanding (MP-VRDU) addresses the challenge of answering questions over long documents where evidence is sparse, distributed across multiple pages, and often exceeds a model's context window. Unlike single-page settings, MP-VRDU systems must operate under a fixed compute budget, requiring them to decide what evidence to acquire, encode, select, and reason over. The field currently lacks a unified understanding of system failures, with competing claims regarding the necessity of vision versus text extraction, the impact of irrelevant context (distractors), and the causes of performance drops on long inputs (context limits vs. integration failures). These claims are difficult to compare because they are typically reported across disparate pipelines and datasets.

Methodology: An Attribution Framework

The paper proposes a decomposition of MP-VRDU system failures into three hierarchical loci: Representation, Selection, and Reasoning. This framework allows for controlled empirical attribution by intervening on one locus while holding the others fixed.

1. Representation (Encoding): Focuses on how faithfully a document is encoded into tokens.

   • Mechanisms: Modality Ceiling (the inherent limit of a modality to represent specific signals, e.g., text cannot capture graphical details) and Conversion Fidelity (accuracy of the encoding process, e.g., OCR errors or parser corruption).
   • Interventions: The authors vary encoding forms (Text-only, Layout-aware Text, Text+Vision, Vision-only) and parser quality/resolution while keeping the reasoner and selected pages constant.

2. Selection (Retrieval): Focuses on what evidence is delivered to the reasoner.

   • Mechanisms: Evidence Coverage (recall of required evidence units) and Distractor Exposure (presence of irrelevant pages).
   • Interventions: The authors construct page sets by withholding gold pages (to test coverage) or adding non-evidence pages (to test distractor tolerance) while fixing the encoding and reasoner.

3. Reasoning (Generation): Focuses on how the model processes the provided evidence.

   • Mechanisms: Evidence Integration (combining information across pages) and Response Calibration (aligning responses with evidence support, distinguishing between hallucination and false abstention).
   • Interventions: The authors supply complete gold evidence sets in the richest encoding and vary the reasoning strategy (e.g., Chain-of-Thought prompting, abstention instructions) and model scale.

The study utilizes the MMLongBench-Doc dataset (1,091 questions across 135 documents) and a single-pass retriever-generator pipeline using Qwen3-VL models (ranging from 2B to 32B parameters) and various parsers (PaddleOCR-VL, MinerU, Unlimited-OCR).

Key Findings

1. Representation: Vision is Necessary but Insufficient

• Vision does not replace text extraction: While vision is necessary for graphical and spatial signals (charts, figures) that text cannot express, vision-only pipelines underperform compared to combined text-vision encodings.
• Complementary nature: Combined encodings (Text + Layout + Vision) achieve the highest accuracy (52.5%), significantly outperforming vision-only (45.6%) and text-only variants. Vision recovers information lost to poor text extraction (e.g., layout-dependent structures or OCR errors), reducing the accuracy spread between weak and strong parsers from 15.6 points to 4.2 points.
• Modality Ceiling: Text channels preserve signals (like reading order and table structure) that visual channels alone lose, even at high resolutions.

2. Selection: Asymmetry of Failure

• Missing evidence bounds accuracy: Omitting a single required gold page causes a sharp, systematic drop in accuracy (e.g., from 38.6% to 18.5% in multi-page scenarios). Retrieval rank is a weak proxy for evidential necessity; lower-ranked pages can be critical.
• Distractors are largely tolerated: Adding irrelevant pages (distractors) while retaining required evidence has a minimal negative impact on accuracy (e.g., a drop of only ~1.5% with three distractors). This contradicts the common assumption that irrelevant context severely dilutes attention within realistic retrieval depths.

3. Reasoning: Integration and Calibration Limits

• Cross-page integration is a primary bottleneck: Even when all required evidence is supplied in a rich encoding, accuracy drops significantly as evidence spans multiple pages (e.g., from 64.6% on single-page to 38.6% on two-page questions). This deficit persists even with larger models (32B) and is not solely a function of context length.
• Prompting helps but is limited: Chain-of-Thought (CoT) prompting narrows the single-to-multi-page gap (improving multi-page accuracy from 38.6% to 44.9%), but does not eliminate it.
• Calibration trade-offs: Abstention instructions reduce hallucinations on unanswerable questions but introduce "false abstentions" on answerable ones. The model often refuses to answer even when it possesses sufficient evidence, particularly when the encoding lacks the necessary modality (e.g., text-only for visual questions).

Contributions

1. Formal Attribution Framework: The paper introduces a structured framework defining three failure loci (Representation, Selection, Reasoning) and their specific mechanisms, providing a common formalism for analyzing MP-VRDU systems independent of specific model architectures.
2. Controlled Empirical Study: By isolating failure modes within a unified pipeline, the study resolves several competing claims in the field, demonstrating that vision is complementary to text, that missing evidence is far more costly than irrelevant evidence, and that integration failures are a distinct reasoning limitation.
3. Practical Guidance: The findings offer actionable design principles for building MP-VRDU systems under fixed compute budgets, such as prioritizing recall over precision in retrieval, matching encoding strategies to document types, and utilizing prompting or scaling to address integration deficits.

Significance and Claims

The paper claims that MP-VRDU bottlenecks are more structured than current debates imply. It argues that the field has gathered design intuitions without a corresponding account of which ones hold; this work provides that account through controlled attribution.

The authors modestly state that their findings establish a diagnostic basis for more targeted investigation rather than a final solution. They note that while prompting and scaling partially address reasoning failures, they do not fully close the gap, suggesting that future work should focus on training strategies specifically targeting cross-page integration. The study is limited to the MMLongBench-Doc dataset and the Qwen3 family of models, and the authors acknowledge that extending these findings to other datasets and model families is a necessary next step.