💬 NLP

A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models

This paper introduces a novel framework using partial information decomposition to quantitatively analyze the internal decision-making processes of 26 large vision-language models, revealing distinct task regimes, contrasting family-level strategies, and the critical role of visual instruction tuning in achieving true multimodal fusion.

Original authors: Lixin Xiu, Xufang Luo, Hideki Nakayama

Published 2026-04-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lixin Xiu, Xufang Luo, Hideki Nakayama

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of two experts trying to solve a mystery: The Visual Detective (who sees the picture) and The Word Wizard (who knows the language). Together, they form a Large Vision-Language Model (LVLM), a super-smart AI that can look at an image and answer questions about it.

For a long time, we only knew if these AIs were "good" or "bad" based on their final score (accuracy). But we didn't know how they solved the problems. Did they truly combine the picture and the words to create a new understanding? Or did they just ignore the picture and guess based on what they already knew from reading books?

This paper introduces a new "X-Ray machine" called PID (Partial Information Decomposition) to look inside the AI's brain and see exactly how it uses information.

Here is the breakdown of their findings using simple analogies:

1. The Four Types of "Brain Power"

The researchers broke down the AI's decision-making into four ingredients:

Redundancy (The Echo): Both the picture and the text say the same thing. (e.g., The picture shows a cat, and the text says "a cat." They are just repeating each other.)
Visual Uniqueness (The Eye): Only the picture gives the answer. (e.g., "What color is the shirt?" The text doesn't say, so the AI must look at the image.)
Language Uniqueness (The Memory): Only the text (or the AI's training) gives the answer. (e.g., "Who is the president?" The picture is irrelevant; the AI just pulls this from its memory.)
Synergy (The Magic Spark): This is the most important part. It's when the picture and the text combine to create an answer that neither could give alone. It's like mixing flour and water to make dough; you can't get dough from just flour or just water.

2. The Two Types of Tasks (The "Menu")

The researchers found that different questions force the AI to use different strategies:

The "Magic Spark" Tasks (Synergy-Driven): These are puzzles where you need both the image and the text to solve it.
- Example: "Is the dog sitting on the red chair?"
- Result: The best AI models here are the ones that create a strong "Synergy" spark. They truly fuse the two inputs.
The "Memory Book" Tasks (Knowledge-Driven): These are questions where the answer is mostly in the AI's head, and the picture is just decoration.
- Example: "What is the medical term for a broken bone?" (Even if the picture shows a broken bone, the AI needs its internal medical dictionary to answer).
- Result: Here, the "Language Uniqueness" (memory) is the star. The picture adds very little value.

3. The Two Types of AI Families (The "Personalities")

Just like people, different AI families have different personalities:

The "Fusion" Family (e.g., Qwen, InternVL): These models are like true detectives. They love to combine clues. Even when the answer is obvious from the text, they still try to look at the picture to double-check. They rely heavily on that "Magic Spark" (Synergy).
The "Language-Centric" Family (e.g., Gemma, Cambrian): These models are like bookworms. They trust their internal knowledge base more than the picture. If they know the answer from their training, they might ignore the image entirely. They rely on "Language Uniqueness."

4. How the AI Learns (The "School Years")

The researchers watched an AI learn from scratch, like watching a student grow up:

Stage 1 (Alignment): The AI learns to match pictures to words, but it's not really "thinking" yet. It's just memorizing that a picture of a dog often goes with the word "dog."
Stage 2 (Instruction Tuning): This is the "Aha!" moment. When the AI starts getting instructions (like "Answer this question based on the image"), the Synergy (the Magic Spark) suddenly explodes. This is the specific stage where the AI learns to actually fuse the two inputs together.

5. The Layer-by-Layer Journey (The "Assembly Line")

Inside the AI, there are many layers of processing (like stations on a factory line). The researchers found a consistent pattern:

Early Layers: The AI gathers raw data.
Middle Layers: The AI starts building its answer using its language knowledge (reading the question).
Final Layer: This is the climax! In the very last step, the AI suddenly combines the language with the image to make the final decision. It's like a chef tasting the soup and adding the final pinch of salt right before serving.

Why Does This Matter?

Before this paper, we only knew if an AI got the answer right. Now, we know how it got it.

If an AI gets a "Magic Spark" task right by ignoring the picture and guessing from memory, it's actually "cheating" (it's not truly multimodal).
If an AI gets a "Memory" task right by looking at the picture, it might be confused.

This new "X-Ray" helps scientists build better AI in the future. They can now tune the AI to be a better "Fusion" detective or a better "Memory" scholar, depending on what job they need it to do. It moves us from just asking "Is it smart?" to asking "Is it smart in the right way?"

1. Problem Statement

Large Vision-Language Models (LVLMs) have achieved state-of-the-art performance across multimodal tasks. However, their internal decision-making processes remain opaque. Current evaluation relies heavily on aggregate accuracy metrics, which fail to distinguish whether a model's success stems from:

True Multimodal Fusion: The synergistic integration of visual and linguistic information.
Unimodal Priors: Reliance on language-only knowledge (hallucinating visual details based on text priors) or visual-only cues.

Existing interpretability methods often focus on "micro-scope" analyses (e.g., attention maps for a single modality) or lack a rigorous theoretical framework to quantitatively decompose how information flows and combines within the model. There is a need for a principled, quantitative tool to dissect the internal strategies LVLMs use to process multimodal information.

2. Methodology

The authors introduce a novel framework based on Partial Information Decomposition (PID) to analyze LVLMs.

Core Concept: Partial Information Decomposition (PID)

PID decomposes the mutual information between inputs ( $X_1$ : Vision, $X_2$ : Language) and an output ( $Y$ : Prediction) into four non-negative components:

Redundancy ( $R$ ): Information shared by both vision and language.
Vision Uniqueness ( $U_1$ ): Information provided exclusively by the visual modality.
Language Uniqueness ( $U_2$ ): Information provided exclusively by the language modality.
Synergy ( $S$ ): New information emerging only from the combination of vision and language (the core of true fusion).

Technical Framework & Adaptations

To apply PID to modern, high-dimensional LVLMs, the authors adapted the BATCH estimator (Liang et al., 2023a), originally designed for continuous modalities. Key adaptations include:

Input Definition: Vision ( $X_1$ ) and Language ( $X_2$ ) are defined as mean-pooled token embeddings from the LVLM's internal layers.
Unimodal Conditioning via Masking: To estimate unimodal predictions ( $P(Y|X_1)$ and $P(Y|X_2)$ ), the authors mask one modality by replacing its embedding sequence with calibrated Gaussian noise (matching the mean and variance of the original embeddings). This preserves the embedding scale while removing semantic content.
Confidence Thresholding: To prevent overconfidence artifacts when restricting predictions to a finite candidate set (e.g., multiple-choice options), a confidence threshold is applied. If the model's total score for the candidate set is low, the distribution is renormalized to uniform.
Soft Aggregation: Instead of using argmax (which creates artificial peaks for uncertain predictions), the marginal output distribution $P(Y)$ is estimated by averaging regularized predictive distributions across samples.

Experimental Design

The study analyzes 26 LVLMs from 11 families (ranging from 0.5B to 90B parameters) across 4 datasets (MMBench, POPE, Reefknot, PMC-VQA). The analysis is conducted along three dimensions:

Breadth: Cross-model and cross-task comparison.
Depth: Layer-wise information dynamics using the "logit lens" technique to inspect hidden states.
Time: Learning dynamics by tracking PID components across training checkpoints (specifically during the two-stage training of LLaVA-1.5).

3. Key Contributions

First Comprehensive PID Analysis of LVLMs: This is the first study to systematically apply PID at scale to dissect the information spectrum of modern LVLMs, moving beyond accuracy to analyze how decisions are made.
Identification of Two Task Regimes: The authors demonstrate that benchmarks fall into two distinct regimes:
- Synergy-Driven: Tasks where performance relies heavily on cross-modal fusion (e.g., MMBench, POPE).
- Knowledge-Driven: Tasks where performance relies on language-side priors and domain knowledge (e.g., Reefknot, PMC-VQA).
Discovery of Stable Family Strategies: Model families exhibit consistent, contrasting strategies:
- Fusion-Centric: Families (e.g., InternVL, Qwen2.5-VL) that rely heavily on Synergy ( $S$ ).
- Language-Centric: Families (e.g., Gemma3, Cambrian) that rely heavily on Language Uniqueness ( $U_2$ ).
Layer-Wise and Temporal Dynamics:
- Three-Phase Pattern: Information flow follows a consistent pattern: emergence in middle layers, language representation building in later layers, and a decisive synergistic fusion spike in the final layer.
- Visual Instruction Tuning: The study identifies Visual Instruction Tuning (Stage 2 of training) as the critical phase where Synergy ( $S$ ) is unlocked, whereas alignment pretraining (Stage 1) yields negligible fusion.

4. Key Results & Findings

Finding 1: Task Regimes

Synergy-Driven Tasks: On MMBench and POPE, accuracy is strongly correlated with Synergy ( $S$ ) ( $\rho \approx 0.75$ ). Removing the image causes a significant accuracy drop for high- $S$ models.
Knowledge-Driven Tasks: On Reefknot and PMC-VQA, accuracy correlates more with Language Uniqueness ( $U_2$ ). Synergy is present but bounded; models cannot fully compensate for missing domain knowledge via fusion alone.

Finding 2: Model Family Strategies

Fusion-Centric Families: Models like InternVL and Qwen2.5-VL consistently show higher $S$ and lower $U_2$ . Scaling these models (increasing parameter count) tends to increase $S$ and accuracy, rather than just amplifying language priors.
Language-Centric Families: Models like Gemma3 and Cambrian rely more on $U_2$ . Even when they answer correctly, they often do so by overriding incorrect language priors with visual cues (high $U_1$ ) rather than generating new synergistic insights.

Finding 3: Layer-Wise Dynamics

A consistent three-phase pattern is observed across models:
1. Emergence: Information appears in middle-to-late layers.
2. Representation Building: Language Uniqueness ( $U_2$ ) rises in later layers.
3. Decisive Fusion: In the final layer, $U_2$ drops sharply while Synergy ( $S$ ) spikes, indicating the final integration of modalities before output.

Finding 4: Learning Dynamics

During Vision-Language Alignment Pretraining (Stage 1), both $S$ and $U_2$ remain low and stable.
During Visual Instruction Tuning (Stage 2), both components increase significantly.
Scale Effect: In the 7B model, the increase is dominated by Synergy ( $S$ ). In the 13B model, the increase is dominated by Language Uniqueness ( $U_2$ ), suggesting larger models prioritize strengthening language priors during fine-tuning.

5. Significance

This work provides a quantitative lens that complements traditional accuracy-based evaluation. By decomposing the "information spectrum," the authors offer actionable insights for:

Diagnosis: Identifying whether a model fails due to poor fusion or lack of knowledge.
Design: Guiding the development of next-generation LVLMs by balancing fusion-centric and language-centric strategies.
Training: Optimizing training stages (e.g., emphasizing visual instruction tuning to unlock synergy).
Benchmarking: Creating new benchmarks that explicitly test for synergy ( $S$ ) or isolate language priors ( $U_2$ ).

The paper concludes that understanding the mechanism of multimodal processing is as critical as the final performance metric, offering a path toward more transparent and robust AI systems. Code and data are publicly available.