Internalized Reasoning for Long-Context Visual Document Understanding

1. Problem Statement

Visual Long-Context (LC) Document Understanding is critical for enterprise, legal, and scientific applications but remains a significant challenge. While Large Language Models (LLMs) have shown leaps in performance via reasoning (e.g., Chain-of-Thought) in math and coding, this capability has not been effectively transferred to long-document Visual Language Models (VLMs).

• The Gap: Existing open-weight VLMs (even large ones like Qwen3 VL 235B) struggle with documents spanning hundreds of pages. The "Thinking" variants of these models often fail to improve document understanding scores compared to their "Instruct" counterparts.
• The Limitation: Current approaches lack high-signal rewards (like verifiable math answers) to train reasoning, leading to a reliance on non-reasoning fine-tuning (SFT) which often results in "context rot" or the "lost-in-the-middle" effect.
• The Goal: To develop a method that enables VLMs to perform structured, multi-hop reasoning over long documents and, crucially, to internalize this reasoning so it occurs implicitly without the token overhead of explicit "thinking" traces during inference.

2. Methodology

The authors propose a synthetic data pipeline designed to generate structured reasoning traces, which are then used to train models via Supervised Fine-Tuning (SFT) and Model Merging.

A. Synthetic Reasoning Pipeline

The pipeline generates training examples in two stages:

1. Evidence Extraction & Relevance Scoring:
   • An extractor VLM (Qwen3 VL 32B) processes each page of a document independently against a question.
   • It extracts natural language evidence snippets and assigns a relevance score (0–10).
   • Bounded Top-K Selection: Instead of processing the whole document sequentially, the pipeline selects only the top-$K$ (default 24) most relevant pages/snippets, sorted by relevance. This avoids the "looping" failure mode seen in naive sequential scans.
2. Answer Generation (Dual Branch):
   • Visual Branch: A strong teacher VLM (Qwen3 VL 235B) answers using only the top-$K$ page images.
   • Text Branch: A strong teacher LLM (Qwen3 235B) answers using only the extracted text evidence.
   • Training data is drawn equally from both branches to ensure the model learns to ground answers in both visual and textual evidence.

B. Trace Design (v1 vs. v2)

• v1 (Naive): Included all pages in order, marking irrelevant ones. This caused models to learn pathological sequential scanning and loop indefinitely during inference.
• v2 (Bounded): Only includes the top-$K$ relevant pages, sorted by relevance. This teaches a finite, retrieval-based algorithm: "Rank, Attend, Stop."

C. Internalization via Model Merging

To make reasoning "internalized" (implicit) rather than explicit:

• Control Token: Training examples include a <cot> token. In 95% of cases, the model sees the reasoning trace inside <thinking> tags; in 5%, it sees only the answer. This allows the model to toggle reasoning modes at inference.
• Task Arithmetic (Model Merging): The final model is created by merging the SFT-trained weights with the base instruction-tuned weights:
  $$\theta_{merged} = \theta_{base} + \alpha \cdot (\theta_{SFT} - \theta_{base})$$
• Low-Strength Merging ($\alpha = 0.25$): The authors found that a low merge strength allows the model to internalize the reasoning capability. The model achieves high performance without generating explicit <thinking> tokens, keeping output lengths comparable to non-reasoning baselines.
• High-Strength Merging ($\alpha = 0.5$): Results in explicit reasoning with massive token overhead (12.4x more tokens).

3. Key Contributions

1. Synthetic Reasoning Pipeline: A novel method for generating structured, relevance-sorted reasoning traces for long-document VQA.
2. Internalized Reasoning: Demonstrated that reasoning can be compressed into the model weights via low-strength merging, allowing the model to "think" implicitly without the latency or token cost of explicit CoT.
3. Inference-Time Control: Showed that the <cot> control token allows users to switch between "reasoning mode" and "direct answer mode" at inference, proving the capability is active and controllable.
4. Superiority over Distillation: Proved that their structured synthetic traces outperform distillation from existing "Thinking" models (e.g., Qwen3 VL 235B Thinking), highlighting that trace design matters more than just the presence of reasoning.

4. Experimental Results

The authors evaluated on Qwen3 VL 32B and Mistral Small 3.1 24B across benchmarks like MMLongBenchDoc (MMLBD) and MMLBD-C.

• State-of-the-Art Performance:
  • Qwen3 VL 32B achieved 58.3 on MMLongBenchDoc, surpassing the 7x larger Qwen3 VL 235B A22B (57.0) and the Qwen3 VL 235B Thinking variant (56.2).
  • Mistral 24B showed a 19% relative improvement over its base model on MMLBD-C.
• Efficiency (Internalization):
  • The internalized model ($\alpha=0.25$) achieved performance comparable to the explicit reasoning model ($\alpha=0.5$) but with 12.4x fewer output tokens (e.g., 250 tokens vs. 1637 tokens for Qwen).
  • Removing the <cot> token at inference degraded performance by 3.8 points (Qwen) and 2.1 points (Mistral), confirming the reasoning capability is active even when not explicitly output.
• Trace Design Impact:
  • The v2 trace design (bounded, relevance-ordered) significantly outperformed v1 (sequential scan), eliminating infinite loops and improving accuracy by ~4 points on Mistral.

5. Significance

• Bridging the Reasoning Gap: This work demonstrates that structured reasoning is essential for long-document understanding, even in domains without verifiable rewards like math.
• Efficiency vs. Performance: It solves the trade-off between reasoning quality and inference cost. By internalizing reasoning, models can perform complex multi-hop tasks without the massive latency and cost associated with generating long "thinking" traces.
• Scalability: The method allows smaller models (32B) to outperform much larger models (235B), suggesting that data quality and reasoning structure are more critical than sheer parameter count for this specific domain.
• Reproducibility: The authors release their synthetic data pipeline, enabling further research into structured reasoning for VLMs.

In conclusion, the paper establishes that structured synthetic reasoning, when internalized via model merging, creates a highly efficient and powerful capability for long-context visual document understanding, outperforming both larger models and existing "thinking" paradigms.