L2V-CoT: Cross-Modal Transfer of Chain-of-Thought Reasoning via Latent Intervention

1. Problem Statement

While Large Language Models (LLMs) have demonstrated remarkable reasoning capabilities through Chain-of-Thought (CoT) prompting, Vision-Language Models (VLMs) still struggle with complex multi-step reasoning tasks (e.g., chart analysis, geometric problems). This gap is primarily attributed to the scarcity of high-quality multimodal reasoning data, which is expensive and time-consuming to generate.

Existing solutions to transfer reasoning from LLMs to VLMs face significant limitations:

• Training-based methods: Require fine-tuning VLMs on massive amounts of textual CoT data, leading to high computational costs and limited generalization.
• Model merging: Requires the VLM and the source LLM to share the exact same architecture (backbone), which is rarely the case in real-world scenarios.
• Prompting: Few-shot CoT prompting often fails because VLMs lack the internal reasoning patterns to effectively utilize the examples.

The core challenge is: How can we transfer general CoT reasoning capabilities from an LLM to a VLM with a different architecture without retraining the model?

2. Methodology: L2V-CoT

The authors propose L2V-CoT, a training-free, architecture-agnostic latent intervention method. The approach is grounded in the hypothesis that LLMs and VLMs share similar low-frequency latent representations of reasoning, despite architectural differences.

The methodology consists of three main stages:

A. Empirical Analysis via Linear Artificial Tomography (LAT)

Before proposing the solution, the authors used LAT to analyze the internal representations of both LLMs and VLMs:

1. Contrastive Sampling: They generated pairs of inputs: CoT prompts (eliciting step-by-step reasoning) and Non-CoT prompts (direct answers).
2. Direction Extraction: They computed the difference in hidden states between CoT and Non-CoT responses to form "CoT direction representations."
3. Frequency Domain Analysis:
   • They found that VLM CoT representations are highly dispersed (noisy) compared to LLMs due to multimodal training drift.
   • Applying Low-Pass Filtering (LPF) to VLM representations significantly reduced this dispersion, concentrating the data and revealing that low-frequency components contain the essential CoT reasoning features, while high-frequency components are noise.
   • Crucially, the low-frequency CoT representations of VLMs and LLMs exhibited substantial distributional overlap, suggesting a shared latent structure that enables cross-modal transfer.

B. Representation Extraction and Alignment

To transfer reasoning from an LLM (e.g., DeepSeek-R1) to a VLM (e.g., Qwen2-VL):

1. Extraction: The LLM processes CoT and Non-CoT prompts to extract the mean CoT direction representation ($v$).
2. Low-Pass Filtering: The LLM representation is filtered in the frequency domain to retain only the essential low-frequency reasoning patterns.
3. Dimension Matching (Resampling): Since LLMs and VLMs often have different hidden state dimensions, the authors apply the LMN (Linear Modulation Network) method in the frequency domain to resample the filtered LLM representation to match the VLM's dimension. This preserves the semantic direction while adapting the size.

C. Latent Intervention (Injection)

During the inference phase of the VLM:

1. The resampled, low-frequency CoT representation from the LLM is injected into the VLM's hidden states at a specific layer ($l_V$).
2. The injection is controlled by a coefficient $\alpha$ to modulate strength.
3. Normalization: To prevent the injection from distorting the VLM's original semantic space, the updated hidden states are normalized.
   $$\hat{h}_V = \text{Normalize}(h_V + \alpha \cdot \hat{v}_{LPF})$$
   This effectively "steers" the VLM into a "slow-thinking" mode, forcing it to decompose problems into intermediate steps.

3. Key Contributions

1. First LAT Analysis for Cross-Modal Reasoning: The paper is the first to use Linear Artificial Tomography to empirically demonstrate that LLMs and VLMs share similar low-frequency latent structures for CoT reasoning, despite architectural and modality differences.
2. L2V-CoT Framework: A novel, training-free method that transfers reasoning capabilities via latent intervention. It overcomes the architecture-alignment constraints of model merging and the data scarcity issues of fine-tuning.
3. Frequency-Domain Resampling: The introduction of frequency-domain resampling (using LPF and LMN) to solve the dimension mismatch problem while preserving critical reasoning information.
4. Comprehensive Validation: Extensive experiments showing that L2V-CoT outperforms both training-free baselines (like Few-shot CoT, RoT) and supervised fine-tuning methods.

4. Experimental Results

The authors evaluated L2V-CoT on multiple benchmarks (MathVista, MathVerse, MMStar, DynaMath, MathVision) across diverse VLM architectures (LLaVA, InternVL, QwenVL, Idefics).

• Performance Gains: L2V-CoT consistently outperformed all baselines.
  • It achieved an average improvement of 3.7% across benchmarks, with peaks of up to 8.6%.
  • It surpassed Supervised Fine-Tuning (Finetuned CoT) in several cases, proving that latent intervention is more efficient than retraining.
  • It significantly outperformed Model Merging, which is restricted to aligned architectures.
• Ablation Studies:
  • Low-Pass Filtering: Removing the LPF step (using interpolation instead) caused performance degradation, confirming that low-frequency components are critical.
  • Source Model: Using representations from a stronger LLM (32B) yielded better transfer results than weaker ones (7B), showing scalability.
  • Injection Layer: Injecting into middle layers yielded the best balance between perception and reasoning, though reasoning tasks benefited from earlier layers.
• Qualitative Analysis: Case studies showed that L2V-CoT enables VLMs to generate longer, more structured reasoning chains (similar to the source LLM) rather than jumping to incorrect conclusions.

5. Significance

This paper addresses a critical bottleneck in the development of multimodal AI: the lack of reasoning data for VLMs. By demonstrating that reasoning capabilities can be "steered" via latent space intervention without retraining, L2V-CoT offers a cost-effective, plug-and-play solution for enhancing VLMs.

The findings suggest that reasoning is a modality-agnostic, low-frequency latent feature that can be extracted from powerful text-only models and injected into vision models. This opens new avenues for improving VLMs in complex domains (science, math, logic) without the prohibitive costs of generating massive multimodal reasoning datasets or training new models from scratch.