Technical Summary: Look on Demand (CSMR)

1. Problem Statement

Current multimodal reasoning approaches generally follow one of two paradigms, both of which exhibit significant structural limitations regarding the integration of visual evidence:

1. Pre-reasoning Visual-to-Text Conversion: These methods convert images into textual descriptions before reasoning begins. While they leverage the strong reasoning capabilities of Large Language Models (LLMs), this static conversion inevitably compresses fine-grained visual details. Crucially, because the reasoning trajectory has not yet unfolded, the initial textualization cannot capture evidence that may become decisive at later stages.
2. Unified Vision-Language Representation: These methods perform end-to-end reasoning within a unified embedding space, avoiding explicit conversion. However, the authors argue this paradigm suffers from linguistic dominance. Due to the self-attention mechanisms in LLMs and standard training objectives, visual representations are biased toward linguistic priors. This results in weakened faithfulness to the original image evidence, leading to hallucinations where the model relies on text rather than visual grounding.

The central challenge identified is determining when and how visual evidence should be introduced into the reasoning process to avoid both information loss and representational bias.

2. Methodology: CSMR Framework

The authors propose CSMR (Cognitive Scheduling for Multimodal Reasoning), a framework that decouples perception from reasoning and treats visual evidence acquisition as a dynamic, state-driven process. The framework consists of two distinct modules:

A. Cognitive Reasoning Core (CRC)

• Role: An LLM that serves as the cognitive controller. It maintains an explicit global reasoning state ($h_t$) containing the reasoning trace and accumulated visual evidence.
• Function: The CRC iteratively decides whether the current evidence is sufficient to answer the question or if additional visual information is required.
• Decision Mechanism: At each step, the CRC generates an output that is parsed into one of two intents:
  1. Query ($q^v_t$): A targeted visual query requesting specific evidence.
  2. Answer ($a_{final}$): The final conclusion.
• Dynamic Control: Unlike static approaches, the CRC dynamically schedules queries based on the evolving reasoning state, allowing for iterative validation and early termination.

B. Primary Visual Perception Module (PVP)

• Role: An independent Vision-Language Model (VLM) that acts as a specialized perception tool.
• Function: Upon receiving a visual query ($q^v_t$) from the CRC, the PVP analyzes the original image ($I$) and returns textualized visual evidence ($a^v_t$).
• Design Choice: The PVP is kept independent of the reasoning core to prevent the visual representations from being influenced by the linguistic priors of the reasoning process.

Reasoning Workflow

1. The CRC initializes with an empty state.
2. It generates an intermediate output based on the current state.
3. If a query is issued, the PVP analyzes the image and returns textual evidence.
4. The CRC updates its state with this new evidence and repeats the process.
5. The loop terminates when the CRC decides to output a final answer or when a maximum token budget is reached.

3. Key Contributions

1. Structural Analysis: The paper identifies and empirically demonstrates a structural limitation in unified multimodal reasoning: visual representations are systematically biased by linguistic priors due to attention mechanisms and training objectives, leading to weakened visual grounding.
2. CSMR Framework: The proposal of a cognitive scheduling framework where an LLM maintains an explicit reasoning state to dynamically govern the acquisition of task-relevant visual evidence from an independent perception module.
3. Empirical Validation: Extensive evaluations across multiple benchmarks (M3CoT, ScienceQA, LLaVA-W) demonstrating that CSMR consistently outperforms representative baselines in zero-shot settings without additional training.

4. Experimental Results

The authors evaluated CSMR against baselines including No-CoT, Caption-based reasoning, DDCoT, CCoT, SCAFFOLD, and ICoT.

• Accuracy: CSMR achieved state-of-the-art results across all three benchmarks. For instance, on ScienceQA, CSMR achieved 78.2% accuracy, significantly outperforming the unified paradigm baseline ICoT (56.8%) and the text-centric DDCoT (71.9%). On M3CoT, it reached 45.7% compared to ICoT's 44.1%.
• Ablation Studies:
  • Dynamic Querying: Restricting the system to a single visual query dropped accuracy from 45.7% to 40.0%, proving the necessity of iterative, feedback-driven acquisition.
  • Pre-planning: Forcing all queries to be generated at the first step (without intermediate reasoning) reduced accuracy to 40.1%, highlighting the need for dynamic scheduling.
  • Termination: Fixing the number of interaction steps to a high number (7) reduced accuracy to 42.7%, indicating that flexible, early termination is crucial to avoid redundant information and semantic drift.
• Hallucination Reduction: CSMR showed a 9-percentage-point increase in samples without hallucinations compared to DDCoT, attributed to the evidence-conditioned reasoning trajectory.
• Efficiency: While CSMR involves multiple model invocations, it was more efficient (fewer seconds per sample) than DDCoT because it avoids generating unnecessary reasoning steps through early termination.

5. Significance and Claims

The authors claim that the primary significance of this work lies in shifting the focus from tighter multimodal fusion to principled evidence scheduling.

• Decoupling Benefits: By decoupling perception and reasoning, CSMR prevents the "linguistic dominance" that plagues unified models, allowing the reasoning core to remain faithful to visual evidence.
• Scalability: The framework offers a path for capability scaling where the reasoning module (LLM) can be upgraded independently of the perception module (VLM). This is presented as more training-efficient than retraining unified models.
• Training-Free: The current implementation achieves these gains without fine-tuning, relying on the inherent capabilities of existing LLMs and VLMs.
• Limitations and Future Work: The authors acknowledge that the multi-model collaboration introduces inference overhead compared to single-pass unified models. They suggest future work could explore quantization to improve efficiency. They also note that while the current version is training-free, the framework is structurally compatible with future parameter optimization (e.g., training the CRC for better information-seeking behaviors).

The paper concludes that effective multimodal reasoning may depend less on the depth of cross-modal fusion and more on the cognitive ability to schedule when and how visual evidence is acquired and integrated.