Technical Summary: ROVER

Problem Statement

Multimodal Large Language Models (MLLMs) have advanced significantly in vision-language reasoning, particularly through Chain-of-Thought (CoT) paradigms. However, existing approaches to grounded multi-image reasoning face structural limitations:

1. Region-Centric Limitations: Current "visual CoT" methods often rely on injecting cropped Regions of Interest (RoIs) or RoI-specific features into the reasoning stream. This design tends to isolate regions, weakening the model's holistic scene understanding and its ability to model inter-object relations. Furthermore, the decoding cost scales with the spatial size and number of RoIs, making high-resolution and multi-object processing expensive.
2. Adaptive Selection Challenges: Alternative approaches involving adaptive visual feature selection or pruning often require fine-grained supervision, complex heuristics, or are confined to single-image scenarios, hindering scalability for multi-image reasoning.
3. Tool-Augmentation Complexity: Methods using external tools or executable programs to iteratively acquire evidence often incur high latency, complexity, and execution brittleness.

There is a need for a decoding-efficient mechanism that transcends isolated regions to facilitate holistic scene understanding and the modeling of inter-object relations without incurring variable-length decoding overheads.

Methodology: ROVER

The authors propose ROVER (Routing Object-Centric Visual Evidence for grounded multi-image Reasoning), a lightweight, learnable plugin designed to route global visual evidence during autoregressive generation. ROVER operates by appending a constant-length token triplet to the reasoning stream upon each object grounding prediction.

Core Components

1. Trigger-and-Insert Interface:

   • ROVER initiates routing upon detecting a valid grounding event (e.g., <obj>...</obj><box>...</box>).
   • Instead of injecting variable-length image patches, it appends a fixed Link/Sift/Weave (LSW) token triplet ($T_k \in \mathbb{R}^{3 \times d}$) to the input embedding sequence. This ensures a constant decoding overhead (3 tokens) per grounded object, independent of the object's spatial size.

2. Routing Primitives (The Token Triplet):

   • Link Token ($t_{Link}$): Absorbs the evolving decoding context, compactly encapsulating the current reasoning state.
   • Sift Token ($t_{Sift}$): Distills intra-image cues. It utilizes an Object-Centric Differential Attention mechanism to aggregate complementary context from non-RoI regions while suppressing irrelevant distractors.
   • Weave Token ($t_{Weave}$): Integrates history-aware evidence. It routes and combines relevant information from previously grounded objects across different images.

3. Routing Modules:

   • Sift (via DiffAttn): Given an object bounding box, the module computes a compact RoI query via average pooling. It then employs Differential Attention to query the complementary context (patches outside the box) while suppressing distractors. The attention operator is defined as:
     $$\text{DiffAttn}(Q, K, V) = \left(\text{softmax}\left(\frac{Q_1 K_1^\top}{\sqrt{d_h}}\right) - \lambda \cdot \text{softmax}\left(\frac{Q_2 K_2^\top}{\sqrt{d_h}}\right)\right)V$$
     This allows the model to extract scene context (e.g., traffic lights near a braking car) without re-encoding the entire image.
   • Weave (via VWS): Maintains a Visual Working Space (VWS), a unified, history-ordered memory ($W_k$) shared across all routing events. The Weave token attends to this VWS using autoregressive masking to integrate historical evidence across objects and images, mimicking "return fixations" in human visual cognition.

4. Training Pipeline:

   • The authors integrate ROVER into Qwen2.5-VL-7B and employ a unified Interleaved SFT-to-GRPO pipeline.
   • Interleaved SFT: Trains the model to emit accurate grounding patterns and task-specific responses, inserting the LSW triplet automatically upon grounding completion.
   • Interleaved GRPO: Uses Group Relative Policy Optimization to optimize trajectory rewards (answer accuracy) while excluding the inserted routing tokens from the loss calculation to ensure stable policy updates.

Key Contributions

1. ROVER Plugin: A lightweight, learnable mechanism that implements global object-centric visual evidence routing by appending a constant-length token triplet after each grounding prediction, avoiding the overhead of variable-length RoI injection.
2. Novel Architecture: The introduction of an Object-Centric Differential Attention mechanism to distill complementary cues while suppressing distractors, coupled with a Visual Working Space (VWS) for history-aware routing across objects and images.
3. Training Framework: A unified interleaved SFT-to-GRPO training pipeline that yields consistent gains on both single-image and multi-image grounded reasoning tasks under controlled settings.

Experimental Results

The method was evaluated on VideoEspresso (multi-image reasoning) and MM-GCoT (single-image grounded reasoning), adhering strictly to original datasets and evaluation protocols.

• VideoEspresso: ROVER achieved the best performance in a controlled setting, outperforming the previous best baseline by +8.6% in answer accuracy. It also demonstrated strong transferability, improving the base model by +4.7% on average across diverse benchmarks (e.g., Mantis, V-Star, TreeBench) despite being fine-tuned exclusively on VideoEspresso.
• MM-GCoT: The method achieved state-of-the-art results, improving answer accuracy by +4.8% and grounding accuracy by +14.6% over the baseline.
• Efficiency: Compared to RoI-resampling methods, ROVER reduced total output tokens by 57.9% while improving reasoning quality, demonstrating superior decoding efficiency.
• Ablation Studies: Experiments confirmed that Differential Attention (Sift) significantly improves grounding accuracy by suppressing distractors, and the VWS (Weave) is crucial for integrating historical evidence across multiple images.

Significance and Claims

The paper claims that ROVER addresses the trade-off between holistic scene understanding and decoding efficiency in grounded reasoning. By conceptually aligning with the dual-process cognitive framework (rapid frontend aggregation followed by deliberate backend reasoning), ROVER enables MLLMs to:

• Revisit and reorganize visual evidence dynamically without the cost of re-encoding large image patches.
• Model inter-object relations and maintain a global context across multiple images via the Visual Working Space.
• Achieve robust generalization to diverse multimodal domains (including zero-shot transfer to benchmarks like TreeBench and HallBench) without requiring task-specific fine-tuning or complex external tools.

The authors position ROVER not as a replacement for visual perception but as a structured routing mechanism that bridges local details and global reasoning, offering a scalable solution for complex, multi-hop visual reasoning tasks.