💻 computer science

Dual-Pathway Circuits of Object Hallucination in Vision-Language Models

This paper introduces Dual-Pathway Circuit Analysis to identify and characterize distinct visual grounding and hallucination pathways in vision-language models, demonstrating that targeted suppression of the hallucination pathway can reduce object hallucinations by up to 76% while maintaining accuracy.

Original authors: Jiaxin Liu, Ding Zhong, Yue Wang, Zhidong Yang, Zhaolu Kang, Guangyuan Dong, Qishi Zhan, Pengcheng Fang, Aofan Liu

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiaxin Liu, Ding Zhong, Yue Wang, Zhidong Yang, Zhaolu Kang, Guangyuan Dong, Qishi Zhan, Pengcheng Fang, Aofan Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot assistant that looks at pictures and describes what it sees. Sometimes, this robot is brilliant. Other times, it gets a little too confident and starts describing things that aren't actually there—like saying, "I see a golden eagle flying over the house," when the picture is just a quiet suburban street. This is called hallucination.

For a long time, scientists treated these robots like "black boxes." They could see the robot make a mistake, but they didn't know why or how the mistake happened inside the robot's brain. They tried to fix it by changing the robot's training or how it speaks, but these fixes were often hit-or-miss and didn't work well on different types of robots.

This paper, "Dual-Pathway Circuits of Object Hallucination," is like taking the robot apart to look at the actual wiring. The researchers wanted to find the specific electrical circuits inside the robot that cause it to lie about what it sees.

Here is the story of what they found, explained simply:

1. The Two-Track System (The Dual-Pathway)

The researchers discovered that inside these vision-language models, there aren't just random wires. There are two distinct "highways" or pathways that the information travels on:

The Truth Highway (Visual Grounding Pathway): This is the team of neurons that looks at the picture and says, "Hey, there is no eagle here. It's just a house." This pathway helps the robot tell the truth.
The Lie Highway (Hallucination Pathway): This is a different team of neurons that gets excited and says, "Eagle! I see an eagle!" even when there isn't one. This pathway drives the errors.

The cool part? They found this same two-track system in five completely different types of robot brains (different architectures). It's like finding that every car, whether it's a Ford or a Ferrari, has an engine and a transmission, even if the parts look slightly different. This suggests that the way these robots learn to hallucinate is a fundamental part of their training, not just a quirk of one specific design.

2. The "Polarity Flip" (The Switch That Gets Stuck)

The researchers did something clever. They used a technique called Activation Patching. Imagine you have a video of the robot working correctly. Then, you take a video of it making a mistake. You then swap specific wires from the "correct" video into the "mistake" video to see if it fixes the error.

They found a strange behavior in the Truth Highway:

When the robot is correct, the Truth Highway wires are active and saying, "No eagle!" (Positive signal).
When the robot hallucinates, those same wires flip their sign! They start saying, "Yes, eagle!" (Negative signal).

It's like a traffic light that usually says "Stop" (Red) to keep you safe. But when the robot hallucinates, that same light flips to "Go" (Green) and tells the robot to drive into a wall. The researchers realized that the robot isn't just "missing" the visual evidence; it's actually misusing the very circuits meant to find the truth to support its lie.

3. The "Redundant Backup" Problem

They also found that the Truth Highway is redundant. This means there are many backup wires doing the same job. If you cut one wire, the others pick up the slack. This is good for stability, but it makes it hard to fix the robot by just tweaking one part.

However, the Lie Highway is different. It's more like a specific, concentrated group of wires that, when they fire, push the robot toward the wrong answer.

4. The Fix: Muting the "Lie Team"

To prove they understood the problem, the researchers tried a "surgical" fix. Instead of retraining the whole robot, they simply turned down the volume (scaled down the output) of the specific wires in the Lie Highway.

The Result: They reduced the robot's hallucinations by up to 76% (almost eliminating the lies) while barely hurting its ability to tell the truth.
The Catch: They tried to fix it by pushing the robot in a single "direction" (like a generic "stop lying" command), and it didn't work as well. This proved that the "Lie Highway" isn't a single simple switch; it's a complex group of wires working together in different ways. You have to mute the specific group, not just push a generic button.

5. Does the Fix Work Everywhere?

The researchers tested if this "Lie Highway" was the same for all types of lies.

Object Existence: "Is there an eagle?" (Yes/No). The fix worked great.
Relationships: "Is the eagle on top of the house?" The fix worked well here too.
Attributes: "Is the eagle golden?" The fix did NOT work.

This tells us that while the robot has a specific "hallucination circuit" for saying things exist that don't, describing details (like colors) might be a different kind of problem entirely.

Summary

In short, this paper is like a mechanic discovering that a car's engine has a specific "stutter" circuit that causes it to misfire. They found that:

All these smart robot brains have the same "Truth" and "Lie" circuits.
The "Truth" circuit sometimes gets hijacked to support the "Lie."
By simply turning down the volume on the "Lie" circuit, they can stop the robot from making up things it doesn't see, without needing to rebuild the whole engine.

This gives us a clear map of how these models fail, which is the first step toward building robots that are truly reliable.

Technical Summary: Dual-Pathway Circuits of Object Hallucination in Vision-Language Models

Problem Statement

Vision-Language Models (VLMs) have achieved remarkable performance in bridging visual perception and natural language understanding. However, they frequently suffer from object hallucinations, where models describe entities, attributes, or relations absent from the input image. This behavior reflects an over-reliance on text-centric priors rather than visual evidence, limiting model reliability and interpretability.

Existing mitigation strategies (e.g., training-time alignment, decoding constraints, post-hoc corrections) largely treat VLMs as black boxes, modifying inputs or decoding procedures without investigating the internal computations driving hallucinations. Consequently, these methods are often model-specific and fail to generalize across architectures. Furthermore, while mechanistic interpretability has been applied to language models to identify circuits for behaviors like factual recall, its application to VLMs is challenging due to the integration of visual representations and language components, and the architectural diversity of modern VLMs.

Methodology

The authors propose Dual-Pathway Circuit Analysis, a framework designed to identify and characterize hallucination-related circuits across architecturally diverse VLMs. The study employs a three-stage analysis strategy on five VLMs (Qwen3-VL, LLaVA, Llama-3.2-Vision, InternVL3-8B, and InternVL3-14B) using the POPE-adversarial benchmark.

1. Activation Patching for Circuit Discovery

The authors extend activation patching to VLMs by corrupting visual token embeddings with Gaussian noise (3σ) at the input layer, driving accuracy to chance levels. They then restore specific components (attention sublayers or MLP sublayers) from a "clean" run into the "corrupt" run.

Indirect Effect (IE): Calculated as the difference in logit differences between the patched run and the corrupt run.
Circuit Extraction: Components are classified into two pathways based on the sign of Cohen's d (effect size) comparing correct vs. hallucinating samples:
- Visual Grounding Pathway ( $d > 0$ ): Components with larger indirect effects on correct samples.
- Hallucination Pathway ( $d < 0$ ): Components with larger indirect effects during hallucination.

2. Conditional Pathway Analysis (CPA)

To move beyond single-component analysis, the authors introduce CPA to characterize pathway-level interactions.

Redundancy vs. Synergy: By comparing the joint effect of restoring all pathway components simultaneously against the sum of individual effects, they determine if components carry overlapping information (redundancy) or complementary fragments (synergy).
Polarity Flip Detection: CPA specifically investigates whether the causal contribution of components changes sign between correct and hallucinating samples, a signature invisible to standard component-level analysis.

3. Causal Validation and Transfer

Targeted Suppression: The authors perform causal interventions by scaling the outputs of hallucination-pathway components ( $s \in \{0.0, 0.25, 0.5, 0.75\}$ ) during inference without retraining.
Baselines: They compare per-component scaling against static-direction interventions (Mean-Difference Projection and Inference-Time Intervention [ITI]) and random component controls.
Generalization: The identified circuits are tested on different POPE splits and the AMBER benchmark (covering existence, attribute, and relational hallucinations) to assess transferability.

Key Results

1. Discovery of a Shared Dual-Pathway Organization

Across all five diverse VLMs, the analysis reveals a consistent dual-pathway structure:

Visual Grounding Pathway: Concentrated in mid-to-late layers.
Hallucination Pathway: Concentrated in early layers and at network boundaries.
This organization persists across different backbone families, vision integration strategies (e.g., cross-attention vs. projector), and model scales, suggesting the structure arises during training rather than being inherited from specific architectures.

2. Polarity Flip and Redundancy

CPA reveals two critical findings:

Grounding Pathway Polarity Flip: On correct samples, grounding components support the ground truth ( $d > 0$ ). On hallucinating samples, these same components flip polarity ( $d < 0$ ), actively supporting the hallucinated answer. This indicates that the "grounding" representations are not missing but have been entrained to the model's non-visual output priors.
Consistent Redundancy: Both pathways exhibit strong within-subset redundancy (joint effect magnitude is significantly smaller than the sum of individual effects) in both correct and hallucinating samples. The redundancy does not dissolve during hallucination; rather, the redundant copies are redirected.

3. Causal Intervention Efficacy

Suppression Success: Targeted suppression of hallucination-pathway components reduces object hallucination by 40% to 76% (relative reduction) with minimal accuracy cost ( $\le 2$ percentage points).
Superiority over Static Directions: Per-component scaling outperforms static-direction baselines (ITI and Mean-Difference Projection). The authors attribute this to the hallucination signal being multi-directional; the leading singular direction captures only 45–69% of the variance, bounding the effectiveness of single-direction interventions.
Specificity: Suppressing the grounding pathway collapses accuracy to chance, confirming a double dissociation: hallucination-pathway suppression reduces errors without destroying task performance, while grounding-pathway ablation destroys the task.

4. Selective Transfer

The identified circuits transfer selectively across hallucination types:

Relational Hallucination: The intervention transfers well to relational hallucinations (36–47% reduction), suggesting shared mechanisms with object-existence hallucinations.
Attribute Hallucination: The intervention largely fails to reduce attribute hallucinations, implying that attribute errors involve distinct circuits (e.g., imprecise visual feature extraction) not captured by the object-existence pathway.

Significance and Claims

The paper claims to provide the first mechanistic characterization of object hallucination across diverse VLM architectures. Its primary contributions are:

Mechanistic Insight: It identifies that object hallucination is driven by a shared dual-pathway structure where grounding components undergo a polarity flip, redirecting visual evidence toward language priors during errors.
Methodological Advance: It demonstrates that pathway-level analysis (CPA) reveals critical signatures (polarity flips) that single-component patching misses.
Causal Validation: It establishes that targeted suppression of specific components can causally reduce hallucinations, validating the circuit hypothesis.
Limitations and Scope: The authors modestly note that while their circuit extraction and suppression validate the mechanism, the proposed scaling method is a causal probe rather than a deployable mitigation strategy. They emphasize that the selective transfer to relational but not attribute hallucinations indicates that different hallucination types may require distinct circuit extractions.

The work concludes that object hallucination in VLMs is not an architectural artifact but a learned mechanistic structure, and that effective intervention requires addressing the multi-directional nature of the hallucination signal rather than relying on static linear directions.