💬 NLP

Do We Really Need External Tools to Mitigate Hallucinations? SIRA: Shared-Prefix Internal Reconstruction of Attribution

SIRA is a training-free, internal contrastive decoding framework that mitigates hallucinations in large vision-language models by generating a language-prior-dominated counterfactual reference within the same transformer through masked attention in later layers, thereby reducing reliance on external tools and extra forward passes while preserving descriptive coverage.

Original authors: Tian Qin, Junzhe Chen, Yuqing Shi, Tianshu Zhang, Qiang Ju, Lijie Wen

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tian Qin, Junzhe Chen, Yuqing Shi, Tianshu Zhang, Qiang Ju, Lijie Wen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read assistant (a Large Vision-Language Model) who is trying to describe a picture you just showed them. Sometimes, this assistant gets a little too confident in what they think they see, rather than what is actually there. They might say, "I see a red bicycle," when the picture is actually just a blurry shape that could be a bicycle, or they might invent details because they've read so many stories about bicycles that their brain fills in the gaps. This is called hallucination.

The paper introduces a clever new trick called SIRA to stop this from happening, without needing to retrain the assistant or hire a second person to double-check the work.

The Problem with Old Tricks

Previously, to stop the assistant from lying, researchers tried a method like this:

Show the assistant the original picture.
Then, show them a blurred or messed-up version of the same picture.
Compare the two answers. If the assistant changes their story when the picture is blurry, maybe they were guessing.

The flaw: This is like asking someone to describe a painting, then asking them to describe the same painting while wearing foggy glasses. The "foggy glasses" version isn't a fair comparison because the glasses themselves distort the image. It's messy, takes twice as long (you have to look at the picture two separate times), and the distortion might create new errors.

The SIRA Solution: The "Shared Prefix"

SIRA takes a different approach. Instead of messing with the picture, it messes with the assistant's internal thought process while they are still thinking.

Think of the assistant's brain as a factory assembly line with many stations (layers) processing the image and the question.

The Shared Start: The assistant starts by looking at the picture and the question together. They build a solid understanding of the scene in the early stations of the factory. This is the "Shared Prefix." SIRA makes sure both versions of the assistant agree on this initial understanding.
The Fork in the Road: After the initial understanding is formed, SIRA splits the process into two parallel tracks running inside the same brain:
- Track A (The Full View): The assistant continues to look at the picture details as they finish their sentence.
- Track B (The "Blind" View): The assistant continues from the exact same starting point, but in the later stations, they are blindfolded to the specific picture details. They can only rely on their general knowledge and the sentence they've already started writing.

How It Fixes the Lie

Now, SIRA compares the two tracks at every step of writing the sentence:

If the assistant says, "I see a dog," and Track A (seeing the dog) and Track B (blindfolded) both say "dog" with high confidence, SIRA knows this is likely just a guess based on common language (maybe dogs are common in stories). It lowers the score for "dog" because the blindfolded version didn't need the picture to say it.
If the assistant says, "I see a purple elephant," and Track A says "purple elephant" (because they see it) but Track B says "no way, that's not in the story" (because they can't see it), SIRA knows the picture is the only reason for this claim. It boosts the score for "purple elephant."

Why It's Better

No Extra Steps: It doesn't need to look at the picture twice. It does everything in one go, just by splitting the internal thought process.
No Messy Distortions: It doesn't blur the picture. It just temporarily "turns off" the visual input for one version of the thought process, keeping the rest of the context perfectly aligned.
No Training Needed: It works on existing models immediately. You don't have to teach the model anything new.

The Result

In tests, SIRA successfully stopped the assistants from making up objects that weren't there (like a "banana" that isn't in the photo) while still letting them describe the real things accurately. It's like having a built-in "reality check" that happens instantly inside the model's own brain, ensuring that what they say is actually supported by what they see, not just what they expect to see.

Technical Summary: SIRA – Shared-Prefix Internal Reconstruction of Attribution

Problem Statement

Large Vision-Language Models (LVLMs) frequently suffer from hallucinations, where the model generates descriptions of objects, attributes, or relations that are linguistically plausible but unsupported by the visual input. This issue is particularly acute when visual evidence is weak or ambiguous, allowing high-probability language priors to dominate token prediction.

Existing mitigation strategies, specifically contrastive decoding, attempt to address this by comparing the model's prediction on the original image against a reference prediction derived from a degraded or perturbed input (e.g., blurred or masked images). However, the authors identify two critical limitations in these external approaches:

Computational Overhead: They typically require a second, full forward pass through the model.
Off-Manifold Artifacts: Perturbing the input image can introduce artifacts that are not representative of the true visual distribution. Consequently, the resulting logit discrepancies may reflect mismatches caused by the perturbation itself rather than a clean attribution of visual evidence versus language priors.

The paper posits that an effective counterfactual reference does not need to be constructed externally in the input space. Instead, it argues that the key requirement is an aligned reference that reveals how the same multimodal state would evolve if access to fine-grained visual evidence were restricted in later processing stages.

Methodology: SIRA

The authors propose SIRA (Shared-Prefix Internal Reconstruction of Attribution), a training-free, internal contrastive decoding framework. SIRA constructs a counterfactual reference inside the same LVLM inference trajectory by manipulating the information flow within the transformer layers, rather than altering the input image.

Core Mechanism

SIRA operates by maintaining two synchronized decoding branches that share a common "prefix" and diverge only after a specific split boundary ( $b$ ):

Shared Prefix ($0$ to $b-1$ ): Both the Full Branch (standard inference) and the Counterfactual Branch share the same prompt, model parameters, decoding history, positional structure, and early-layer computations. This ensures that both branches inherit an identical, stable multimodal interpretation and early visual grounding before the split.
Late Branching ( $b$ to $L-1$ ): After the boundary layer $b$ $b$ , the branches diverge:
- The Full Branch continues standard multimodal inference, allowing attention to image tokens.
- The Counterfactual Branch applies a specific attention mask ( $M_{cf}$ ) that blocks access to image-token positions. In this branch, image tokens cannot be read by queries, nor can they produce queries that feed the text stream.
Internal Contrast: At each decoding step, SIRA computes the logits for both branches ( $z_{full}$ $z_{f u l l}$ and $z_{cf}$ $z_{c f}$ ). The final decoding logit ( $z_{cd}$ $z_{c d}$ ) is calculated as:
$z_{cd}(v) = (1 + \alpha)z_{full}(v) - \alpha z_{cf}(v)$
Where $\alpha$ $α$ controls the contrast strength.
- Tokens that remain strong in the Counterfactual Branch (lacking visual access) are penalized, as they are likely driven by language priors.
- Tokens that gain an advantage specifically in the Full Branch are favored, as their probability depends on continued visual evidence.

Efficiency

Unlike methods requiring two full forward passes (e.g., VCD), SIRA shares the computation for the first $b$ layers. It only incurs the cost of the final $K$ layers for the counterfactual branch. This results in an idealized per-token cost ratio of $1 + K/L$ (approximately $1.5\times$ for a mid-layer split), significantly lower than the $2\times$ cost of external contrastive methods.

Key Contributions

Information-Flow Perspective: The paper re-evaluates hallucination mitigation by arguing that effective references must preserve early multimodal grounding while restricting later image-token access, rather than relying on external input perturbations.
SIRA Framework: The proposal of a training-free, internal contrastive decoding method that constructs a counterfactual reference via shared-prefix late branching and post-boundary image-token masking.
Empirical Validation: Comprehensive experiments demonstrating that SIRA reduces hallucinations while preserving descriptive coverage and incurring lower overhead than two-pass methods.

Experimental Results

The authors evaluated SIRA on Qwen2.5-VL and LLaVA-v1.5 across three benchmarks: POPE (object-level hallucination), CHAIR (captioning hallucination), and AMBER (multidimensional captioning).

POPE: SIRA achieved the best results across all 36 metrics (combinations of backbone, dataset, and negative-sampling strategy). It improved accuracy by an average of +5.4 points on Qwen2.5-VL and +4.9 points on LLaVA-v1.5 over base decoders, outperforming strong baselines like DoLa, OPERA, and VCD. Notably, improvements were consistent even in "Adversarial" splits where language priors are strong.
CHAIR: SIRA reduced both sentence-level (CHAIRs) and instance-level (CHAIRi) hallucination rates. For example, on LLaVA-v1.5, CHAIRs dropped from 31.5 to 23.1 at length 64, indicating a reduction in spurious object mentions rather than simple omission.
AMBER: SIRA reduced hallucination metrics (Hal, Cog) while preserving or increasing descriptive coverage (Cover). This contrasts with baselines like OPERA and VCD, which often reduced hallucinations by shortening or hedging captions (over-hedging). SIRA specifically reduced "Cognitive" hallucinations (Cog), which are tightly coupled to language priors.
Efficiency: SIRA incurred a latency overhead of approximately 1.4–1.5 $\times$ compared to standard decoding, significantly lower than VCD ( $\sim2.1\times$ ) and OPERA ( $\sim5.9\times$ ).

Significance and Limitations

Significance:
The paper claims that SIRA offers a more direct and cleaner token-level attribution proxy than externally constructed references. By keeping the counterfactual branch on the same model manifold and sharing the early semantic state, SIRA isolates the contribution of visual evidence without introducing perturbation artifacts. It achieves state-of-the-art hallucination reduction without requiring model retraining, external verifiers, or perturbed inputs, making it applicable to open-weight LVLMs with white-box access.

Limitations:
The authors acknowledge two primary limitations:

Overhead: While lower than two-pass methods, SIRA still carries a non-zero per-step decoding cost due to the re-evaluation of the counterfactual branch.
Access Requirements: Constructing the internal contrast requires white-box access to hidden states, attention masks, and cache states, restricting its application to open-weight or controllable LVLMs rather than black-box APIs.