💻 computer science

One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination

This paper proposes a unified, training-free framework that mitigates MLLM hallucinations by synergistically leveraging vision tokens through Synergistic Visual Calibration to strengthen visual semantics and Causal Representation Calibration to correct language biases, achieving significant accuracy improvements with minimal inference overhead.

Original authors: Zhan Fa, Yue Duan, Jian Zhang, Lei Qi, Yinghuan Shi

Published 2026-03-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhan Fa, Yue Duan, Jian Zhang, Lei Qi, Yinghuan Shi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly arrogant, robot assistant named MLLM. This robot is great at talking and knows a lot about the world, but it has a bad habit: when you show it a picture, it often ignores what's actually in the photo and just makes things up based on what it thinks should be there. This is called hallucination.

For example, if you show it a picture of a cat, it might confidently say, "I see a dog playing fetch," because in its training data, "dogs" and "playing" often go together, even though the picture clearly shows a cat.

The Old Way: Two Separate Fixes That Didn't Mix

Before this new paper, scientists tried to fix this robot in two separate ways, like trying to fix a car by only tightening the bolts on the left side or only painting the right side:

The "Shout Louder" Approach: They tried to make the robot pay more attention to the picture. But the robot was so used to its own internal chatter (its "text inertia") that even shouting the picture louder didn't stop it from ignoring the visual clues.
The "Scare the Robot" Approach: They tried to trick the robot by showing it a distorted, blurry version of the picture to teach it what not to say. But this was like showing a robot a picture of a cat that looks like a potato. The robot got confused, learned the wrong lessons, and started making more mistakes because the "negative example" was too noisy and weird.

The researchers found that just trying both of these at the same time didn't work well either. It was like trying to steer a car by turning the wheel left with one hand and right with the other at the same time. The signals fought each other.

The New Idea: One Token, Two Jobs

The authors of this paper realized the problem wasn't the methods, but that they were treating the robot's "eyes" (the Vision Tokens) as two separate things. They proposed a unified framework where the robot's "eyes" do two jobs at once to fix the balance between seeing and talking.

Think of the Vision Token as a single, super-versatile tool in the robot's toolbox. The paper gives this tool two distinct "fates" or roles:

Role 1: The "Super-Helper" (Synergistic Visual Calibration)

The Problem: As the robot talks longer, it starts to forget the picture. The visual signal fades away, like a radio station losing signal the further you drive.
The Fix: The robot takes the original picture and creates a slightly different, "augmented" version (like flipping it or adding a little static). It then combines the "memory" of both pictures.
The Analogy: Imagine you are trying to remember a friend's face. If you only look at one photo, you might forget details. But if you look at two photos of the same friend taken from slightly different angles, your memory becomes super sharp. The robot does this by merging two "memories" of the image to keep its visual focus strong, preventing it from drifting back to its own imagination.

Role 2: The "Truth Detector" (Causal Representation Calibration)

The Problem: The robot has a strong bias to say certain things (like "a dog") even when it shouldn't.
The Fix: Instead of showing the robot a weird, distorted picture (which confuses it), the researchers simply remove most of the picture's data, leaving only a tiny, bare-bones skeleton of the image.
The Analogy: Imagine you are trying to figure out if a student is cheating on a test.
- Old Way: You give them a test with the answers scribbled in crayon. They get confused and fail.
- New Way: You take away 90% of the test questions, leaving just a few. If the student still answers those few questions with the "cheating" answers, you know for sure they are relying on their own bad habits, not the test.
- By removing most of the image data (creating an "information gap"), the robot is forced to rely on its internal biases. The researchers then measure exactly what the robot gets wrong in this state and subtract that "wrongness" from its normal thinking. It's like a noise-canceling headphone that identifies the specific hum of the robot's bias and cancels it out.

The Result: A Balanced Robot

By using this single framework, the robot learns to:

Listen harder to the picture (using the Super-Helper).
Ignore its own bad habits (using the Truth Detector).

The result is a robot that is much less likely to lie about what it sees. In tests, this method improved the robot's accuracy by about 2% (which is huge in AI terms) and did it very quickly, without needing to retrain the robot from scratch.

In short: The paper fixed the robot's hallucinations by teaching its "eyes" to work smarter, not harder, using one tool to both sharpen its vision and silence its internal chatter.

1. Problem Statement

Multimodal Large Language Models (MLLMs) suffer from hallucinations, where the model generates fluent text that contradicts visual evidence. The root cause is identified as a fundamental imbalance between visual and language signals:

Visual Fading: As text generation proceeds, the model's attention to visual tokens decays sharply.
Text Inertia: The model's strong internal language priors (linguistic biases) dominate the generation process, overriding visual input.

Limitations of Existing Solutions:
Current training-free methods address this using disjoint strategies that fail when combined naively:

Visual Enhancement: Boosting visual attention often fails against strong language priors.
Text Suppression: Creating negative samples by distorting images (pixel-level masking) introduces "modality-gap" noise, leading to unstable and unreliable corrections.
Naive Combination: Simply stacking an enhancement method and a suppression method degrades performance because they operate at different levels (attention vs. logits) and lack a unified mechanism.

2. Methodology

The authors propose a Unified Latent Calibration Framework that operates entirely on intermediate representations (latent space) using the vision token as the single source for both enhancement and suppression. The framework consists of two core modules:

A. Synergistic Visual Calibration (SVC)

Goal: Counteract visual fading by strengthening visual representations.
Mechanism: Based on the insight of Semantic Complementarity, the module creates an augmented image ( $I_{aug}$ ) via random flipping, Gaussian blur, and noise.
Process:
1. Vision tokens from the original image ( $V$ ) and augmented image ( $V_{aug}$ ) are concatenated to form a synergistic memory bank ( $V_{syn}$ ).
2. At a specific middle layer ( $L_c$ ), the module injects this enriched context into the hidden states using scaled dot-product attention.
3. The original hidden state is interpolated with the new visual context to re-engage the model with richer visual details.

B. Causal Representation Calibration (CRC)

Goal: Suppress text inertia and internal biases by purifying hidden states.
Mechanism: Based on the Information-Gap Principle, the module avoids pixel-level distortion. Instead, it creates "latent-space negative samples" by pruning vision tokens (removing most tokens, retaining only $N_h=5$ ).
Process:
1. Probe Generation: The model runs parallel forward passes with the original tokens and the pruned tokens.
2. Bias Isolation: The difference between the original hidden state ( $H_{org}$ ) and the pruned hidden state ( $H_{neg}$ ) is calculated. This difference vector ( $\Delta H$ ) isolates the pure visual signal lost due to the information gap, effectively capturing the "hallucination direction" caused by language bias.
3. Calibration: This average difference vector ( $v_{crc}$ ) is subtracted from the current hidden states in shallow layers ( $1 \dots L_c$ ) to push the representation away from the hallucination direction and toward visual truth.

3. Key Contributions

Reframing the Problem: The paper reframes hallucination mitigation as a vision-language balance problem, demonstrating that disjoint approaches are insufficient and that naive combinations fail.
Unified Framework: It proposes the first training-free framework that harmonizes enhancement and suppression by leveraging the dual potential of vision tokens, operating entirely on intermediate representations.
Novel Modules:
- SVC: Uses augmented tokens for synergistic visual grounding.
- CRC: Uses pruned tokens (information-gap) to create stable, in-distribution negative samples for precise bias correction, superior to noisy pixel-level masking.
Efficiency: The framework achieves state-of-the-art performance with minimal computational overhead.

4. Experimental Results

The framework was evaluated on four MLLM architectures (LLaVA-1.5, Shikra, MiniGPT-4, InstructBLIP) across multiple benchmarks:

Object Hallucination (POPE):
- Achieved an average 2% absolute improvement in accuracy on the challenging GQA split for LLaVA-1.5 compared to the vanilla model.
- Consistently outperformed strong baselines (VCD, PAI, VISTA, ONLY) across Random, Popular, and Adversarial splits.
Open-Ended Generation (CHAIR):
- Achieved the best CHAIRI (instance-level) and CHAIRS (sentence-level) scores, significantly reducing ungrounded object generation in captions.
General Capabilities (MME & MMHal-Bench):
- Improved both Perception and Cognition scores on the MME benchmark, proving that hallucination reduction does not come at the cost of general reasoning.
- Outperformed baselines across all 8 categories of the MMHal-Bench.
Efficiency:
- Latency: Only 1.06x overhead compared to greedy decoding (significantly faster than VCD at 2.4x and VISTA at 1.1x).
- Memory: Lower peak GPU memory usage than competing methods.

5. Significance

This work is significant because it moves beyond the "patchwork" approach of combining separate fixes. By identifying the vision token as the central pivot, it creates a principled, unified system that simultaneously strengthens visual grounding and purifies language bias. The use of latent-space token pruning for negative sampling is a key theoretical and practical advancement, providing a cleaner signal for bias correction than traditional image distortion methods. The result is a highly efficient, training-free solution that sets a new state-of-the-art for reliable MLLM deployment.