Technical Summary: Structured Layout Priors for Robust Out-of-Distribution Visual Document Understanding

Problem Statement

Vision-Language Models (VLMs) have simplified document understanding by enabling end-to-end conversion of page images to structured markup. However, these models frequently fail on layouts that differ from their training distribution (Out-of-Distribution or OOD). The authors attribute this failure to a "two-hop bottleneck":

1. Hop 1 (Entity Resolution): The decoder must first classify and localize the enclosing layout entity (e.g., identifying a table's boundaries).
2. Hop 2 (Content Extraction): The decoder then extracts the content within those boundaries.

When Hop 1 fails, Hop 2 collapses, resulting in omissions, malformed structures, or autoregressive repetition (infinite loops). This issue is particularly acute in compact models where decoder capacity is constrained, as the model must simultaneously discover spatial structure and generate text from raw pixels.

Methodology

The paper proposes a two-stage pipeline that pre-resolves Hop 1 outside the autoregressive loop, effectively decoupling layout localization from content extraction.

1. The Two-Stage Pipeline

• Stage 1: Layout Precomputation (Hop 1 Resolution):
  • A lightweight, RT-DETR-based detector (docling-layout-heron) processes the input image to predict document regions and their coordinates.
  • These detections are serialized into the parser's native DocTags vocabulary (e.g., <text><loc_100>...</text>), rather than plain text.
  • This serialized layout prior is injected directly into the prompt alongside the full page image.
• Stage 2: VLM Parsing (Hop 2 Execution):
  • The base VLM (granite-docling-258M) receives both the full page image and the structural blueprint.
  • Unlike "analyze-then-parse" approaches that crop the image based on detector boxes, this method keeps the full page in view. This allows the decoder to use the original image as a global fallback if the detector is imperfect.
  • The decoder's capacity is thus freed to focus on content extraction (Hop 2) rather than rediscovering layout structure.

2. Training Strategy

The fine-tuning process involves three phases:

• Phase 1 (Pretraining): Joint optimization of modality alignment and document understanding on a hybrid corpus (synthetic and human-labeled), focusing on dense text and simple layouts initially.
• Phase 2 (Layout Injection): Transitioning to a guided architecture where the model is trained on the serialized layout priors. The model learns to condition generation on the provided "blueprint."
• Phase 3 (Guided Fine-tuning): A specific loss masking strategy is employed. Since location tokens are copied verbatim from the prompt, their prediction is masked out of the loss function. The loss is applied only to label and content tokens, ensuring the model focuses on structural context and content generation rather than learning to copy coordinates.

Key Contributions

1. Guided-Decoding Framework: A pipeline that injects structured RT-DETR layout priors directly into the VLM prompt using the parser's native DocTags vocabulary. This targets the two-hop bottleneck without requiring architectural changes to the base VLM.
2. Mechanistic Evidence: Attention analysis reveals a bimodal phase shift. The decoder attends primarily to injected layout tokens when emitting structural syntax (Hop 1) and shifts attention to image patches when emitting textual content (Hop 2). This confirms the pipeline successfully offloads spatial discovery.
3. Robust OOD Generalization: The method demonstrates significant improvements in structural generalization across severe shifts in layout, language, and complexity, while maintaining in-distribution performance.

Experimental Results

The method was evaluated on a 10k-page structural OOD benchmark, a Chinese subset of OmniDocBench, and the 26k-page ViDoRe V3 benchmark.

• Structural OOD (NoveltySet): Markdown F1 score improved from 0.37 to 0.92.
• Linguistic OOD (OmniDocBench Chinese): Table extraction (TEDS) improved from 0.01 to 0.36.
• Stability (ViDoRe V3): Infinite-loop decoding failures dropped significantly across all seven industrial domains tested (mean drop from 3.9% to 2.4%).
• In-Distribution Performance: The method matched or exceeded baselines on DocLayNet, confirming that robustness gains do not come at the cost of in-distribution accuracy.
• Efficiency: The approach adds only 15% wall-clock latency and a median of 74 prompt tokens, with no architectural changes to the base model.

Significance and Claims

The paper claims that by explicitly separating entity resolution from content extraction, compact VLMs can achieve robustness comparable to larger models without architectural overhauls. The authors emphasize that:

• Global Context Preservation: Keeping the full page image in view allows the model to recover from detector errors, a key advantage over crop-based pipelines.
• Native Vocabulary Alignment: Using DocTags (the parser's own vocabulary) for priors, rather than plain text, aligns the prior with the decoder's generation space, facilitating better integration.
• Practicality: The solution offers a practical path to reliable document parsing with compact models, adding minimal computational overhead while significantly reducing catastrophic failures like infinite loops.

The authors note limitations, specifically that the pipeline relies on the upstream detector (though it shows tolerance to noise) and that the mechanism is supported by empirical evidence and attention analysis rather than complete causal explanations.