Imagination Helps Visual Reasoning, But Not Yet in Latent Space

1. Problem Statement

The paper addresses the growing paradigm of Latent Visual Reasoning (LVR) in Multimodal Large Language Models (MLLMs). LVR aims to mimic human imagination by reasoning through the model's hidden states (latent tokens) rather than generating explicit text or using external tools. While LVR methods (e.g., Monet, Mirage, LVR) have shown promising empirical results, their internal mechanisms remain a "black box."

The core problem investigated is: Do latent tokens actually perform the visual reasoning they are designed for? The authors hypothesize that despite high performance, these latent representations may not effectively encode visual information or causally influence the final answer, suggesting a disconnect between the "imagination" process and the actual reasoning outcome.

2. Methodology

The authors employ a two-pronged approach: Causal Mediation Analysis to deconstruct existing LVR methods, and the proposal of a new Text-Space Imagination method called CapImagine.

A. Causal Mediation Analysis of Latent Tokens

The authors model the reasoning process as a causal chain: Input ($X$) $\rightarrow$ Latent Tokens ($Z$) $\rightarrow$ Output ($Y$). They investigate the validity of this chain through systematic perturbations:

1. Input-Latent Disconnect ($X \rightarrow Z$):

   • Method: They perform instance-level perturbations on the input (changing images/questions) and measure the cosine similarity of the resulting latent tokens.
   • Finding: Latent tokens exhibit high homogeneity. Even with diverse inputs, the latent tokens remain nearly identical across instances and tasks. As reasoning progresses, tokens collapse into uniform states, indicating they fail to attend to specific input details.

2. Latent-Answer Disconnect ($Z \rightarrow Y$):

   • Method: They perform intervention analysis (replacing latent tokens with Gaussian noise, zero vectors, or identical tensors) and probing analysis (using only latent tokens to answer new questions).
   • Finding:
     • Intervention: Drastic changes to latent tokens result in negligible changes to the final answer (performance drops are minimal, e.g., <1%).
     • Probing: Latent tokens encode minimal visual semantics. When used as the sole input for answering questions, performance falls below random guessing, proving they cannot support reasoning independently.

Conclusion of Analysis: Latent tokens act more like "soft prompts" or placeholders than active carriers of visual reasoning. The model likely relies on implicit shortcuts rather than the latent space for deliberation.

B. Proposed Method: CapImagine

Motivated by the failure of latent space, the authors propose CapImagine, a method that forces the model to imagine visually through explicit text.

• Core Concept: Instead of compressing visual thoughts into hidden states, the model generates textual descriptions of visual manipulations (e.g., "zooming in," "highlighting regions," "drawing lines").
• Data Construction Pipeline:
  1. Source: Based on the Monet-SFT-125K dataset.
  2. Rewriting: Intermediate images in the reasoning chain are converted into textual captions describing the visual change (e.g., "A red rectangle highlights the region...").
  3. Refinement: An MLLM refines the reasoning chain to ensure logical coherence and fluency.
  4. Filtering: A rigorous quality control step filters out ambiguous questions or conflicting reasoning traces, retaining only high-quality instances (17k instances).
• Training: The model is fine-tuned on this text-reformulated data using a standard CoT-SFT approach, learning to "imagine" via text tokens.

3. Key Contributions

1. Systematic Causal Dissection: The first rigorous study using Causal Mediation Analysis to reveal that latent tokens in current LVR methods are causally ineffective. They are insensitive to input changes and do not drive the final output.
2. CapImagine Framework: A novel, simple, and effective method that replaces latent reasoning with explicit text-space imagination.
3. Empirical Superiority: Demonstrating that explicit text-based imagination outperforms complex latent-space baselines across multiple benchmarks.
4. Insight into Mechanism: Providing evidence that "imagination" in MLLMs is more effective when it is explicit and interpretable (text) rather than implicit and opaque (latent).

4. Experimental Results

The authors evaluated CapImagine against state-of-the-art baselines (Monet, LVR, DeepEyes, PixelReasoner) on several vision-centric benchmarks:

• High-Resolution Perception:
  • HR-Bench-8K: CapImagine outperformed Monet by 4.0%.
  • MME-RealWorld-Lite: CapImagine outperformed Monet by 4.9%.
  • V Benchmark:* Showed consistent improvements (approx. 2.6% average gain).
• Abstract Reasoning:
  • BLINK (Jigsaw/Multi-view): Surpassed LVR and Monet by over 10 points, demonstrating strong generalization to compositional tasks.
  • TableVQA: Achieved a 6.1% improvement over Monet.
• Ablation Studies:
  • Removing the text rewriting (using only a placeholder token) caused performance to drop, confirming the necessity of explicit text descriptions.
  • Removing data filtering led to performance degradation, highlighting the importance of data quality.
• Efficiency:
  • CapImagine achieved inference speeds comparable to the latent-space method Monet and was nearly twice as fast as the tool-based method DeepEyes, offering a better trade-off between performance and cost.

5. Significance and Conclusion

This paper fundamentally challenges the assumption that "latent space reasoning" is a superior or necessary paradigm for visual imagination in MLLMs.

• Theoretical Impact: It reveals that current latent reasoning methods may be suffering from causal disconnection, where the model ignores the latent tokens it generates. This suggests that the "imagination" is happening elsewhere (likely in the text generation stream or via implicit shortcuts) rather than in the designated latent tokens.
• Practical Impact: The authors propose that explicit text-based imagination is a more faithful, interpretable, and effective alternative. By verbalizing visual transformations, models can achieve higher accuracy without the complexity of designing latent supervision or dealing with mode collapse.
• Future Direction: The work advocates for moving away from opaque latent reasoning toward transparent, text-driven reasoning chains that can be audited and verified, paving the way for more robust and trustworthy multimodal AI.

In summary, the paper concludes: "Imagination helps visual reasoning, but not yet in latent space." The path forward lies in making the imagination process explicit through text.