💬 NLP

Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfer

This paper reveals that large language models (LLMs) and vision-language models (LVLMs) share a significant modality-invariant subspace of neurons responsible for multi-step inference, and leverages this discovery to propose Shared Neuron Low-Rank Fusion (SNRF), a parameter-efficient framework that transfers mature inference capabilities from LLMs to LVLMs without requiring large-scale multimodal fine-tuning.

Original authors: Chenhang Cui, An Zhang, Yuxin Chen, Gelei Deng, Jingnan Zheng, Zhenkai Liang, Xiang Wang, Tat-Seng Chua

Published 2026-02-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chenhang Cui, An Zhang, Yuxin Chen, Gelei Deng, Jingnan Zheng, Zhenkai Liang, Xiang Wang, Tat-Seng Chua

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Do AI Brains Share a "Reasoning Room"?

Imagine you have two very different robots:

The Text Robot (LLM): It's a genius at reading books, solving math problems, and writing stories, but it's blind.
The Vision Robot (LVLM): It can see pictures, describe what's in them, and answer questions about images, but it sometimes struggles with complex logic or multi-step math problems.

For a long time, researchers thought the Vision Robot needed to be "retrained" from scratch with millions of image-text pairs to get smarter at reasoning. But this paper asks a fascinating question: Do these two robots actually share the same internal "thinking muscles" for solving problems?

The answer is a resounding yes.

The Discovery: Finding the Shared "Thinking Neurons"

Think of a giant AI model like a massive city with billions of tiny workers (neurons) inside. When the AI solves a math problem, only a specific team of workers gets excited and starts working.

The researchers looked inside the brains of the Text Robot and the Vision Robot and found something surprising: More than half of the workers that get excited during complex reasoning are the exact same people in both robots.

The Analogy: Imagine the Text Robot is a master chef, and the Vision Robot is a master painter. You'd think they use completely different tools. But when they both have to "bake a cake" (solve a logic problem), they both reach for the exact same set of measuring cups and mixing bowls hidden in their kitchens.
The Finding: These shared "measuring cups" (neurons) are mostly located in the part of the brain that handles Key Information (like the "Key" in a Key-Value pair). They are the specific parts of the brain that help the AI connect dots, do math, and follow a chain of thought.

The Proof: Turning the Volume Up and Down

To prove these shared neurons are actually doing the heavy lifting, the researchers did two experiments:

The "Mute" Test: They turned off the shared neurons in the Vision Robot.
- Result: The robot's ability to solve math problems instantly dropped to zero. It became like a baby who forgot how to count.
- Contrast: When they turned off random, non-shared neurons, the robot only got slightly slower, not useless. This proved the shared neurons are the "engine" of reasoning.
The "Loud" Test: They turned up the volume (amplified the signal) on these shared neurons.
- Result: The robot suddenly got much better at math and logic. It was like giving the robot a caffeine shot specifically for its reasoning skills.

The Solution: SNRF (The "Smart Upgrade" Kit)

Since the Vision Robot already has these "reasoning muscles" but just isn't using them effectively, the authors built a tool called SNRF (Shared Neuron Low-Rank Fusion).

How it works (The Analogy):
Imagine the Vision Robot is a car that has a great engine but a weak transmission. The Text Robot is a race car with a perfect transmission.

Old Way: You would try to rebuild the Vision Robot's engine from scratch (expensive, takes forever, requires massive data).
The SNRF Way: You take the perfect transmission from the race car, but you only attach the specific gears that the Vision Robot is already missing. You don't touch the wheels, the paint, or the seats (the visual perception parts).

The Magic:

It identifies the shared "reasoning gears."
It calculates the tiny difference between the Text Robot's gears and the Vision Robot's gears.
It injects just enough "magic dust" (low-rank updates) to align them.
Result: The Vision Robot suddenly becomes a math genius without needing to be retrained on millions of new images. It keeps its ability to see and recognize objects perfectly, but now it can also think like a text expert.

Why This Matters

It's Cheap: You don't need massive supercomputers or millions of dollars to make these models smarter. You just need to "tune" the existing brain.
It's Fast: The upgrade happens in minutes, not months.
It's Safe: Because they only touch the "reasoning" parts and leave the "seeing" parts alone, the robot doesn't start hallucinating (making things up) or forgetting how to recognize a cat.

The Bottom Line

This paper reveals that Large Language Models (text) and Large Vision-Language Models (images) are more similar than we thought. They share a common "language of logic" inside their brains. By finding and upgrading these shared parts, we can instantly give visual AI models the reasoning superpowers of text models, making them smarter, faster, and more reliable.

1. Problem Statement

Large Vision-Language Models (LVLMs) have made rapid progress in perception tasks but consistently lag behind text-only Large Language Models (LLMs) in multi-step inference and compositional decision-making (e.g., complex math reasoning, scientific problem solving).

Current Limitations: The prevailing approach to bridge this gap is scaling LVLMs with more multimodal data and larger model sizes. However, high-quality paired image-rationale data is scarce compared to text corpora, and it remains unclear how to effectively transfer the mature reasoning capabilities of LLMs to multimodal settings without expensive retraining.
Core Question: Can the relatively mature inference capabilities of LLMs be transferred to LVLMs at a low cost by leveraging shared internal computational mechanisms?

2. Methodology

The authors propose a three-stage approach: Discovery, Validation, and Transfer.

A. Discovery: Identifying Shared Inference Neurons

The authors hypothesize that LLMs and LVLMs, despite different modalities and training corpora, utilize overlapping internal pathways for reasoning.

Neuron Profiling: They define a "neuron detection corpus" consisting of inference prompts (e.g., math problems with ground-truth rationales). They profile activation patterns in both text-only LLMs (e.g., Qwen2.5-Math) and LVLMs (e.g., Qwen2.5-VL).
Activation Definition: A neuron is considered "activated" if zeroing its parameters causes a significant change in the model's output embedding (measured by $L_2$ norm).
Intersection: They compute the intersection of top-activated neurons across model pairs. They analyze specific neuron types:
- MLP/FFN: fwd.up and fwd.down (feed-forward layers).
- Attention: attn.q, attn.k, and attn.v (query, key, value).

B. Validation: Causal Probing via Amplification

To verify that these shared neurons are functionally responsible for inference (rather than just coincidental overlap):

Causal Ablation: They zero out the parameters of the shared neurons ( $N_{shared}$ ). This causes LVLM performance on inference tasks to collapse to near zero, whereas random ablation of the same number of neurons causes only a moderate drop.
Amplification Analysis: They amplify the activations of shared neurons ( $N \leftarrow \lambda \cdot N$ ). This systematically strengthens inference behaviors.
Concept Extraction: By analyzing the tokens generated during amplification, they found that shared neurons encode specific, interpretable concepts, particularly mathematical operations (e.g., algebra, geometry, arithmetic) and logical structures.

C. Transfer: Shared Neuron Low-Rank Fusion (SNRF)

Based on the discovery of a shared "inference subspace," they propose SNRF, a parameter-efficient framework to transfer reasoning capabilities from a source LLM to a target LVLM without large-scale fine-tuning.

Mechanism:
1. Identify Shared Indices: Determine the set of shared neurons $S_{\ell,p}$ for each layer $\ell$ and projection type $p$ .
2. Compute Weight Difference: Calculate the difference matrix $\Delta = W_{src} - W_{tgt}$ between the source LLM and target LVLM.
3. Low-Rank Approximation: Perform Singular Value Decomposition (SVD) on $\Delta$ to get a low-rank approximation $\Delta^{(r)}$ .
4. Selective Injection: Update the target weights only within the shared subspace:
  $\tilde{W}_{\ell,p} = W_{tgt} + \beta \cdot M_{S_{\ell,p}}(\Delta^{(r)}_{\ell,p})$
  Where $M$ is a masking operator that keeps only the shared neurons, and $\beta$ controls the update strength.
Theoretical Guarantee: The authors prove that for small $\beta$ , SNRF incurs lower loss than linear merging (averaging weights) because it avoids updating non-shared parameters where the loss landscape curvature is high, thus preserving the LVLM's perceptual capabilities.

3. Key Contributions

Empirical Discovery of Shared Subspace: The paper provides the first evidence that >50% of the top-activated neurons during multi-step inference are shared between representative LLMs and LVLMs, even across different model families (e.g., Qwen vs. Idefics). These neurons are concentrated primarily in the Key (attn.k) and Value (attn.v) attention matrices.
Causal Interpretability: Through amplification and ablation, the authors demonstrate that these shared neurons are not generic but encode causal, concept-level semantics (specifically math and logic). They are necessary for inference; removing them destroys reasoning ability.
SNRF Framework: A novel, parameter-efficient method to transfer reasoning circuitry. It achieves state-of-the-art performance improvements on inference benchmarks while maintaining the LVLM's original visual perception and hallucination robustness, requiring no new multimodal training data.

4. Experimental Results

The method was evaluated on diverse benchmarks including MathVista (math reasoning), ScienceQA, MMMU/MMMU-Pro (general multimodal reasoning), MME (perception), and POPE (hallucination).

Inference Performance:
- MathVista: SNRF improved Solution scores by +4.7 (3B) and +4.6 (7B) points over the base models.
- ScienceQA: Significant gains, e.g., +22.6 points for Qwen2.5-VL-3B.
- MMMU-Pro: Improved verification-style reasoning scores (e.g., 0.116 $\to$ 0.179 for Qwen2.5-VL-7B).
Preservation of Capabilities:
- Perception (MME): Scores increased or remained stable (e.g., +32 points for Qwen2.5-VL-7B), indicating no degradation in visual understanding.
- Hallucination (POPE): Scores remained effectively unchanged, proving the method does not introduce new hallucinations.
Comparison to Baselines: SNRF outperformed other merging strategies (Linear, DARE, FRANK) and training-based baselines (SFT, RL) on inference-heavy metrics while being significantly more parameter-efficient.

5. Significance

Theoretical Insight: The work challenges the notion that modality-specific training is strictly required for reasoning. It suggests that inference circuitry is modality-invariant and resides in a shared neuron subspace, likely formed by the shared language backbone and the nature of logical reasoning itself.
Practical Impact: SNRF offers a low-cost, training-free pathway to upgrade LVLMs. Instead of collecting massive image-rationale datasets and fine-tuning, practitioners can simply "inject" reasoning capabilities from a mature text LLM into an LVLM by updating a small subset of parameters.
Interpretability: The study bridges the gap between black-box model performance and mechanistic interpretability, showing that specific neurons can be isolated, understood, and manipulated to control model behavior (specifically reasoning vs. perception).

In summary, the paper demonstrates that LLMs and LVLMs share a "reasoning brain" within their neural architecture. By identifying and selectively updating these shared neurons, one can efficiently transfer high-level reasoning skills to multimodal models without compromising their visual capabilities.