💻 computer science

Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow

This paper introduces a steering-vector-based causal attribution framework to uncover and enhance the "Adapt-Aggregate-Execute" emotional circuits in Large Vision-Language Models, revealing a functional decoupling between visual cue aggregation and narrative generation that enables targeted inference-time interventions to significantly improve emotional reasoning and reduce hallucinations.

Original authors: Chengsheng Zhang, Chenghao Sun, Zhining Xie, Xinmei Tian

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chengsheng Zhang, Chenghao Sun, Zhining Xie, Xinmei Tian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Vision-Language Model (LVLM) as a highly skilled, but sometimes confused, art critic. You show it a picture of a crying person and ask, "What's happening here?" Ideally, it should say, "This person is sad." But sometimes, these models get it wrong, confidently describing a sad face as "joyful" or "excited." This is called an "emotional hallucination."

This paper acts like a mechanic opening the hood of that art critic's brain to see exactly how it processes feelings, and then fixing the wiring so it gets it right.

Here is the breakdown of their discovery and solution, using simple analogies:

1. The Problem: The "Black Box" of Feelings

Previously, we knew these models could do emotion tasks, but we didn't know how. It was like knowing a car drives but not knowing which part of the engine makes it go. The researchers wanted to find the specific "emotional circuits" inside the AI's brain.

2. The Discovery: The "Adapt-Aggregate-Execute" Factory

The researchers found that the AI doesn't just "feel" an image all at once. Instead, it processes emotions through a three-stage assembly line, much like a factory making a custom cake:

Stage 1: Adaptation (The Raw Ingredients)
- What happens: The AI looks at the raw pixels of the image (the flour, eggs, and sugar).
- The Analogy: This is like the delivery truck bringing ingredients to the kitchen. The AI is just getting the visual data ready to be understood, but it hasn't decided what kind of cake it's making yet.
Stage 2: Aggregation (The Mixing Bowl)
- What happens: This is the most critical part. The AI takes those visual clues and starts mixing them into a specific "flavor."
- The Discovery: The researchers found that in the middle layers of the AI, there are special "mixing bowls" (called attention heads) that are specific to certain emotions. One bowl is for "sadness," another for "anger." They take the visual cues and say, "Okay, this is definitely a sad story."
- The Analogy: This is where the baker tastes the batter and decides, "This needs more chocolate." The AI is crystallizing the abstract feeling into a concrete concept.
Stage 3: Execution (The Baking & Decorating)
- What happens: Now that the AI knows the "flavor" (sadness), it starts writing the story.
- The Discovery: In the deep layers (the end of the process), the AI switches gears. It stops using the specific "sadness" bowl and starts using a universal oven that works for any emotion. It takes the "sadness" concept and bakes it into sentences like "I felt a heavy heart."
- The Analogy: The baker is now putting the sad cake into the oven and writing the label on the box. The specific "sadness" decision is made; now it's just about executing the final product.

3. The Breakthrough: "Functional Decoupling"

The most important thing they found is that the AI separates deciding what the emotion is from saying the emotion.

Middle Layers: Specialized chefs who only know how to make "sad" or "angry" dishes.
Deep Layers: A general manager who takes whatever dish is ready and serves it to the customer.

4. The Solution: VEENA (The "Surgical Fix")

Instead of retraining the whole AI (which is expensive and slow), the researchers created a tool called VEENA. Think of VEENA as a "remote control" for the AI's brain that you use while it's thinking.

Visual Emotion Enhancement (VEE): This part turns up the volume on the signal coming from the image. It's like shining a spotlight on the "sadness" ingredients in the mixing bowl so the AI doesn't miss them.
Emotional Neuron Augmentation (ENA): This part turns up the volume on the neurons that actually write the words. It's like giving the baker a little extra energy to make sure the "sadness" label is written clearly and boldly.

5. The Result

When they tested this "remote control" on a massive benchmark (MER-UniBench), the AI became much better at describing emotions correctly.

Fewer Hallucinations: It stopped calling sad faces "happy."
No Extra Cost: Because they just tweaked the existing wiring instead of rebuilding the engine, the AI didn't get slower.
Proof of Concept: The fact that turning up these specific "dials" fixed the problem proved that they had correctly identified the emotional circuits.

In short: The paper found that AI processes emotions in three steps (See -> Decide -> Speak). They discovered that the "Decide" step happens in the middle of the brain using special parts, and the "Speak" step happens at the end using general parts. They built a tool to boost the signal in these specific parts, making the AI much more empathetic and accurate without needing to retrain it from scratch.

Technical Summary: Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow

1. Problem Statement

Large Vision-Language Models (LVLMs) are evolving into empathetic agents but remain prone to emotional hallucinations. Unlike object hallucinations, which result in factual errors, emotional hallucinations involve generating linguistically coherent narratives that contradict the affective content of visual stimuli (e.g., describing a weeping face as expressing joy). This misalignment risks violating social norms and ethical boundaries.

Current solutions rely on black-box, data-driven optimization techniques (e.g., instruction tuning, reinforcement learning) that fit behaviorist fine-tuning without guaranteeing internal alignment. Furthermore, existing mechanistic interpretability research in LVLMs predominantly targets object perception, leaving the internal emotional circuitry unexplored. Bridging this gap is hindered by two primary challenges:

Absence of Counterfactual Visual Benchmarks: Unlike text, isolating emotional attributes in images without altering narrative content or visual semantics is difficult.
Ineffectiveness of Discrete Metrics: Emotional expression is diffusive and holistic, manifesting through long-form generation tone rather than single tokens, rendering standard Next-Token-Prediction metrics insufficient.

2. Methodology

The authors propose a Steering-Vector-based Causal Attribution Framework tailored for descriptive emotional reasoning, followed by a training-free intervention strategy.

A. Emotional Direction Extraction

To overcome the lack of visual counterfactuals, the authors construct a specialized dataset using strictly paired emotional and neutral contrasts.

Process: For an emotional image $I_{emo}$ , they use generative AI (Google Nano Banana) to create a neutral counterpart $I_{neu}$ that preserves identity, composition, and style but removes emotional indicators (e.g., neutralizing facial expressions, removing tears).
Steering Vectors: They compute the difference in hidden state activations between the emotional and neutral inputs to extract global steering vectors ( $S_l$ ) for specific emotions (e.g., sadness, anger).

B. Emotional Circuit Discovery

Using a hierarchical discovery strategy, the authors trace the cross-modal flow of emotion:

Layer Identification: They inject extracted emotional directions into layers during inference to identify critical layers where behavioral impact (hit rate change) is maximized.
Head Localization: They employ Latent Restoration Metrics based on activation patching to locate specific attention heads that align with the emotional direction.
Neuron Tracing: They recursively trace information flow from critical heads back to upstream MLP neurons using Attribution Patching to identify the granular origin of emotional features.

C. The "Adapt-Aggregate-Execute" Mechanism

Through this analysis, the paper uncovers a distinct three-stage mechanism:

Adaptation (Shallow Layers): Visual features undergo modality alignment to bridge the semantic gap between pixels and text.
Aggregation (Middle Layers): The model activates Contextual Trigger Neurons and aggregates signals into the Query token via sentiment-specific attention heads. These heads are selectively active for distinct emotion categories (e.g., specific heads for negative sentiments).
Execution (Deep Layers): The Query token activates Explicit State Neurons and steers narrative generation via emotion-general pathways that function universally across different emotions.

D. VEENA Intervention

Based on these insights, the authors introduce VEENA (Visual Emotion Enhancement and Emotional Neuron Augmentation), a surgical, training-free, inference-time intervention:

Visual Emotion Enhancement (VEE): Regulates information routing by scaling attention weights. It strengthens $V \to Q$ (Visual to Query) attention in upstream layers to facilitate aggregation and $V \to L$ (Visual to Last Token) attention in downstream layers to ensure execution remains grounded in visual details.
Emotional Neuron Augmentation (ENA): Amplifies the activation levels of identified emotion-salient MLP neurons to consolidate semantic expression.

3. Key Results

Extensive experiments were conducted on MER-UniBench, a comprehensive benchmark covering Basic Emotion Recognition, Sentiment Analysis, and Fine-grained Emotion Recognition.

Performance Improvement: VEENA significantly improves performance across all 9 datasets in MER-UniBench, raising the average hit rate from 58.1% to 64.8% (+6.7%). On the challenging open-vocabulary task (OV-MERD+), it achieves a +9.2% gain.
Synergy: The combination of VEE and ENA outperforms using either component alone, confirming the synergy of regulating both information routing and semantic activation.
Causal Fidelity: Ablation studies show that random interventions fail to reproduce these gains, while targeting the specific identified circuits (Top-10 heads and Top-30 neurons) yields state-of-the-art results with minimal latency.
Generalization: The method was validated on different model architectures (Qwen3-VL-4B/8B, LLaVA-OneVision-1.5-4B), demonstrating that the "Adapt-Aggregate-Execute" mechanism is intrinsic to LVLMs.
Safety: Evaluations on general benchmarks (POPE, CHAIR, MME) indicate that VEENA does not induce catastrophic forgetting or universal hallucinations, preserving general vision-language capabilities while specifically targeting emotional sub-networks.

4. Significance and Claims

The paper claims to be the first to:

Localize Emotional Circuits: Provide a precise, vector-based attribution framework that transcends discrete metric limitations to localize emotional circuits in LVLMs.
Demystify Mechanisms: Reveal the functional decoupling between sentiment-specific aggregation (middle layers) and universal execution (deep layers), challenging the notion that emotional processing is a monolithic block.
Enable Surgical Intervention: Demonstrate that training-free, inference-time interventions based on mechanistic understanding can effectively mitigate emotional hallucinations without retraining, offering a more interpretable and reliable alternative to black-box fine-tuning.

The authors emphasize that their work focuses strictly on elucidating internal pathways to enhance reliability and safety, ensuring model responses accurately reflect visual reality rather than generating deceptive emotional content.