Lang2Act: Fine-Grained Visual Reasoning through Self-Emergent Linguistic Toolchains

1. Problem Statement

Visual Retrieval-Augmented Generation (VRAG) aims to enhance Vision-Language Models (VLMs) by incorporating external visual documents to answer queries. While existing VRAG frameworks improve upon text-based RAG by preserving global visual context, they face significant limitations when handling fine-grained visual reasoning:

• Rigid Tool Dependency: Current state-of-the-art methods (e.g., VRAG-RL, Pixel-Reasoner) rely on pre-defined, external image tools (such as cropping, zooming, or selecting regions via APIs).
• Information Loss: These explicit image operations often decouple visual perception from logical reasoning. Mechanical cropping can lead to irreversible loss of critical visual information (e.g., cropping out a number while trying to read it) and disrupts the global layout context.
• Inflexibility: Fixed tool interfaces constrain models to mechanical operations, preventing them from adapting their perception strategy dynamically based on the specific reasoning needs of a query.

2. Methodology: Lang2Act

The authors propose Lang2Act, a framework that replaces rigid external image tools with self-emergent linguistic toolchains. Instead of physically manipulating the image pixels, the model learns to "simulate" visual actions through autoregressive text generation, effectively internalizing visual perception within the reasoning process.

The framework operates via a two-stage Reinforcement Learning (RL) training pipeline:

Stage 1: Linguistic Action Emergence & Tool Curation

• Goal: Enable the VLM to self-explore high-quality visual reasoning trajectories and distill them into a reusable "linguistic toolbox."
• Process:
  1. The VLM is trained using the Decoupled Clip and Dynamic sampling Policy Optimization (DAPO) algorithm.
  2. The model generates reasoning trajectories ($\tau$) that include implicit visual actions (e.g., "locate the title," "read the value").
  3. Successful trajectories (those leading to correct answers) are analyzed. The model abstracts these actions into standardized linguistic tools (triplets of: tool name, description, and parameters).
  4. A global Tool Pool is constructed, and the top-$K$ most frequent and effective tools are selected to form the final Linguistic Toolbox ($T_{box}$).
  • Example Tools: read_text_element, read_numeric_value, compare_values, locate_visual_element.

Stage 2: Tool-Enhanced Visual Reasoning Optimization

• Goal: Optimize the VLM to effectively utilize the curated linguistic toolbox for downstream reasoning.
• Process:
  1. The VLM is fine-tuned using RL with the fixed $T_{box}$.
  2. The model generates a structured output containing:
     • Visual Routing ($z_{route}$): Selects the relevant document pages from the retrieved set.
     • Tool Execution ($z_{exec}$): Generates a sequence of linguistic tool invocations (e.g., <tool name="read_numeric_value" args="...">) followed by the model's internal observation of the result.
  3. Reward Function: The optimization maximizes a weighted reward combining:
     • Answer Correctness ($r_{ans}$): Whether the final answer matches the ground truth.
     • Tool Validity ($r_{tool}$): A regularization term ensuring the model uses tools strictly within the defined structure and the curated toolbox.

3. Key Contributions

1. Self-Emergent Linguistic Toolchains: The paper introduces a paradigm shift from external image manipulation to internalized linguistic perception. By treating visual actions as text generation, Lang2Act avoids the information loss associated with image cropping while maintaining fine-grained attention.
2. Two-Stage RL Framework: A novel training mechanism that first discovers effective visual grounding patterns through self-exploration and then consolidates them into a robust, reusable toolbox for reasoning.
3. Unified Perception and Reasoning: Unlike decoupled approaches, Lang2Act integrates visual grounding directly into the autoregressive generation process, allowing the model to maintain global context while focusing on specific details.
4. Comprehensive Evaluation: Extensive experiments across multiple benchmarks (SlideVQA, ViDoSeek, MMLongBench-Doc) and model scales (3B, 7B, 32B) demonstrate the method's robustness and generalizability.

4. Experimental Results

• Performance Gains: Lang2Act outperforms all baselines (including Prompting methods, VLRMs, MRAGs, and Tool-Enhanced VLMs like VRAG-RL) by over 4% in average accuracy. On SlideVQA, it achieves 83.62% accuracy compared to the second-best (VRAG-RL) at 81.74%.
• Fine-Grained Perception: Analysis shows a strong positive correlation between the model's ability to attend to "golden regions" (ground truth areas) and answer accuracy. Lang2Act achieves the highest V-Precision and V-Recall, indicating superior ability to locate and extract relevant visual evidence without losing context.
• Hallucination Reduction: By relying on linguistic tools rather than risky image cropping, Lang2Act significantly reduces hallucinations. Case studies show that while VRAG-RL often fails due to incorrect cropping (losing the answer "43%"), Lang2Act successfully identifies the correct region and answers accurately.
• Generalization: The curated toolbox transfers effectively to out-of-domain datasets (FetaTab, PaperTab) without re-training, proving the tools are generalizable primitives rather than dataset-specific artifacts.
• Efficiency: Lang2Act achieves lower inference latency (50s) compared to iterative image-tool methods like VRAG-RL (123s), as it avoids the overhead of external API calls and image processing.

5. Significance

Lang2Act addresses a critical bottleneck in Visual RAG: the trade-off between global context preservation and fine-grained detail extraction.

• Theoretical Impact: It demonstrates that complex visual reasoning can be achieved through linguistic abstraction rather than physical image manipulation, suggesting that VLMs can "think" about visual details as effectively as they "see" them.
• Practical Impact: The framework offers a more robust, efficient, and scalable solution for document understanding, reducing the risk of information loss inherent in cropping-based systems. It provides a blueprint for building self-improving agents that can curate their own reasoning tools, moving beyond static, pre-defined toolkits.

In conclusion, Lang2Act represents a significant step forward in making VLMs more capable of handling complex, document-based queries by unifying perception and reasoning through a self-emergent, linguistic approach.