Technical Summary: Doc-CoB

Problem Statement

Document understanding involves performing question answering (QA) and information extraction (IE) over document images, which are characterized by high information density and complex layouts. Existing Multimodal Large Language Models (MLLMs) typically employ a "one-pass" strategy, processing the entire image uniformly. This approach implicitly assumes all regions are equally important, often leading models to be misled by redundant visual information and infer answers from incorrect regions. Conversely, existing coarse-to-fine strategies in general visual QA often resort to direct cropping or zooming into specific regions. While this reduces visual redundancy, it discards critical layout information unique to documents, such as spatial relationships, relative positioning, and cross-region dependencies, causing failures in layout-sensitive queries.

The core challenge addressed is how to follow a coarse-to-fine reasoning paradigm for document understanding while preserving the global layout context necessary for accurate inference.

Methodology: Doc-CoB

The authors propose Doc-CoB (Chain-of-Boxes), a framework that enables MLLMs to perform document understanding through coarse-to-fine, layout-aware visual reasoning without modifying the underlying model architecture. The framework operates in two distinct stages, utilizing visual prompting to guide attention:

1. Stage 1: Key Box Selection

   • A layout analyzer ($\Lambda$) extracts a set of candidate layout boxes ($B$) from the document image ($x_I$).
   • Each box is assigned a unique ID and overlaid on the image to create a visually prompted image ($X^S_1$).
   • The MLLM ($f$) is prompted to select the subset of box IDs ($I_{key}$) most relevant to the query ($Q$). This is formulated as a multiple-choice prediction over candidate boxes.

2. Stage 2: Focused Answering

   • The selected key boxes ($B_{key}$) are highlighted with red borders on the original image to create a second visually prompted image ($X^S_2$).
   • Crucially, the entire document image is retained rather than cropped. This preserves the global context and spatial relationships while emphasizing the relevant regions.
   • The MLLM generates the final answer ($A$) conditioned on this highlighted image and the query.

Reasoning Tasks and Data Construction

To support this paradigm, the authors introduce two specific reasoning tasks to enhance the model's layout and semantic understanding:

• Box Recognition: The model learns to map a specific visual box to its corresponding ID.
• Box Reasoning: The model learns to describe the semantic role of a specific box ID in answering a given query.

To train these capabilities, an automatic data generation pipeline was constructed using GPT-4o as a linguistic annotator and MinerU as a visual layout analyzer. This pipeline processed nine existing document datasets (including DocVQA, DeepForm, SROIE, FUNSD, etc.) to generate:

• 249,601 training samples for the Doc-CoB framework.
• 126,970 samples for Box Recognition.
• 269,108 samples for Box Reasoning.
  These samples include intermediate visual supervision, explicitly labeling "helpful" boxes (containing evidence) and "confusing" boxes (resembling the query but incorrect).

Key Contributions

1. Doc-CoB Framework: A novel two-stage paradigm that integrates coarse-to-fine, layout-aware visual reasoning into MLLMs via visual prompting, balancing key region focus with global context preservation.
2. Reasoning Tasks & Dataset: The proposal of Box Recognition and Box Reasoning tasks, supported by a large-scale dataset of 249k+ samples with intermediate visual supervision tailored for document layout reasoning.
3. Performance and Generalization: Extensive experiments demonstrating that Doc-CoB significantly improves performance across seven benchmarks and four different MLLM architectures (ranging from 2B to 40B parameters), including models not specifically pre-trained for documents.

Experimental Results

The paper evaluates Doc-CoB on seven benchmarks (DocVQA, DUDE, DeepForm, FUNSD, SROIE, VRDU-Ad, VRDU-RF) using four MLLMs: InternVL2 (2B/8B), Qwen2.5-VL, and DocOwl 1.5.

• Performance Gains: Doc-CoB consistently improves performance across all models and datasets. Notably, the InternVL2-8B-CoB model achieves state-of-the-art results on six datasets and surpasses the much larger InternVL2-40B and the closed-source GPT-4o on all seven benchmarks. For instance, on DeepForm, InternVL2-8B-CoB achieves 80.06%, a significant improvement over the 36.33% SFT baseline.
• Ablation Studies:
  • Removing the two-stage paradigm (using only the CoB dataset for one-pass inference) yields improvements but remains inferior to the full two-stage approach, confirming the value of the reasoning paradigm itself.
  • Removing the specific reasoning tasks reduces performance, indicating that current MLLMs lack the intrinsic capability to fully exploit the framework without targeted supervision.
  • Replacing the whole-image input in Stage 2 with cropped images leads to significant performance drops (5–15 points), validating the necessity of preserving global layout context.
• Efficiency: Despite processing two images, Doc-CoB allows for lower effective resolution settings, resulting in comparable or lower total token usage than vanilla inference while achieving higher accuracy.
• Robustness: The method remains effective when using different layout analyzers (e.g., OCR+K-means, Marker) and demonstrates strong zero-shot generalization to unseen domains like infographics and multi-page documents.

Significance and Limitations

The paper claims that Doc-CoB demonstrates that explicit, layout-aware visual reasoning can significantly enhance document understanding capabilities in MLLMs without architectural changes. It highlights that the combination of a coarse-to-fine reasoning paradigm and targeted intermediate supervision is critical for handling the unique challenges of document layouts.

Limitations acknowledged by the authors:

• Noise Accumulation: While data generation strategies suppress noise from layout analysis, inference-time noise cannot be fully eliminated and may accumulate across stages.
• Single-Round Selection: The current pipeline performs only one round of box selection. The authors suggest that a self-verification mechanism for iterative refinement could further improve robustness.
• Granularity: The framework addresses layout-level reasoning but does not resolve intra-box ambiguities (e.g., OCR errors or ambiguous text within a selected box), which remain a source of failure.