Hallucination-aware intermediate representation edit in large vision-language models

1. Problem Statement

Large Vision-Language Models (LVLMs) have achieved remarkable success in multimodal reasoning and scene understanding but suffer significantly from hallucinations, where generated outputs contradict the actual visual input (e.g., describing objects that are not present).

Existing mitigation strategies face critical limitations:

• Retraining Methods: Require constructing specialized datasets and fine-tuning the entire model, leading to high computational costs and data collection burdens.
• Contrastive Decoding (CD) Methods: Mitigate hallucinations during inference by contrasting original outputs with perturbed variants. However, they require dual inference passes (doubling latency) and apply uniform adjustments to all tokens, regardless of whether a token is actually prone to hallucination, which harms efficiency and output coherence.
• Lack of Controllability: Most methods cannot flexibly control the degree of hallucination, which is undesirable for scenarios like creative writing where some hallucination enhances expressiveness.

2. Methodology: HIRE Framework

The authors propose HIRE (Hallucination-aware Intermediate Representation Edit), a framework that dynamically detects and edits intermediate representations within the LVLM without modifying the model's weights or requiring retraining. The framework operates in a single inference pass and consists of three core components:

A. The Editor (Feature Editing)

The Editor learns to isolate and suppress hallucination-related features while preserving semantic integrity.

• Architecture: It utilizes a dual-encoder autoencoder ($G_\phi$) comprising a Semantic Encoder ($E_{sem}$) and a Hallucinatory Encoder ($E_{hal}$), a multi-head attention fusion module, and a decoder.
• Mechanism:
  1. Decomposition: Input representations from attention layers are split into semantic ($h_{sem}$) and hallucinatory ($h_{hal}$) components.
  2. Direction Learning: The model calculates a "hallucination-reduction direction" ($\delta$) by analyzing the divergence between authentic and hallucinated representations in the hallucinatory subspace.
  3. Editing: The optimized editing direction ($\Delta$) is computed by fusing semantic and hallucinatory features. The final edited representation is obtained by projecting the original features along this direction: $h_{aug} = h + \alpha \cdot \Delta$.

B. The Router (Selective Editing)

To avoid unnecessary computational overhead, a lightweight Router ($R_\theta$) determines which tokens require editing.

• Strategy: Instead of editing every token, the Router analyzes the first-layer transformer representation to make a binary decision ($c=1$ or $0$).
• Logic: If $c=1$, the Editor is activated for all subsequent layers; otherwise, the token passes through unchanged. This leverages the observation that shallow layers often contain the most informative signals for hallucination detection.

C. Hallucination Regulator

A hyperparameter $\alpha$ controls the intensity and direction of the edit:

• $\alpha > 0$: Suppresses hallucinations (steering features toward low-hallucination directions).
• $\alpha < 0$: Amplifies hallucinations (useful for creative tasks).
• $\alpha = 0$: No editing.

D. Training Strategy

Since labeled hallucination data is scarce, the authors employ two specialized training techniques:

1. Contrastive Learning (for the Editor): Uses pairs of "authentic" (original image) and "hallucinated" (visually degraded/noised image) inputs. The semantic encoder maximizes similarity between matching tokens, while the hallucinatory encoder maximizes separation between authentic and hallucinated groups.
2. Direct Preference Optimization (DPO) (for the Router): Trains the Router to prefer editing decisions that lead to lower hallucination scores (measured by CHAIR) without requiring a reference model.

3. Key Contributions

1. New Paradigm: Introduces a feature-editing approach that mitigates hallucinations by modifying intermediate representations, eliminating the need for model retraining or dual-inference overhead.
2. Dynamic Control: Proposes a framework capable of controllable hallucination generation, allowing users to adjust the degree of hallucination via a simple hyperparameter ($\alpha$).
3. Efficiency: Achieves state-of-the-art performance with minimal computational cost by using a lightweight Router to selectively edit only high-risk tokens.
4. Comprehensive Validation: Demonstrates effectiveness across generative (captioning) and discriminative (VQA) tasks on multiple benchmarks.

4. Experimental Results

The method was evaluated on LLaVA-1.5 and InstructBLIP across three major benchmarks: CHAIR, POPE, and AMBER.

• Hallucination Reduction:
  • On the CHAIR benchmark (long descriptions, 512 tokens), HIRE reduced sentence-level hallucinations (CHAIRS) by ~40% and object-level hallucinations (CHAIRI) by ~50% compared to baselines.
  • It outperformed strong baselines like VCD, M3ID, and Octopus.
• Discriminative Tasks:
  • On POPE, HIRE achieved the highest accuracy and F1 scores, surpassing the previous state-of-the-art (Octopus) by significant margins (e.g., +1.48% accuracy on LLaVA-1.5).
• AMBER Benchmark:
  • Achieved the highest overall AMBER scores, indicating superior performance in both generative and discriminative hallucination identification.
• Efficiency:
  • The method introduces minimal inference overhead (e.g., ~3.45 TFLOPs for InstructBLIP vs. 5.61 for VCD), significantly lower than CD-based methods which require dual passes.
• Generalization & Stability:
  • The method generalizes well across different model scales (TinyLLaVA, Qwen-VL) and datasets (Visual Genome).
  • Ablation studies confirmed that both the dual-encoder Editor and the Router are essential for performance and efficiency.

5. Significance

HIRE represents a significant shift in hallucination mitigation strategies. By moving away from heavy retraining and computationally expensive dual-inference decoding, it offers a practical, plug-and-play solution for LVLMs. Its ability to dynamically control the level of hallucination makes it uniquely versatile, suitable not only for strict factual tasks but also for creative applications where "controlled hallucination" is desired. The approach demonstrates that hallucinations are separable in the latent space and can be effectively neutralized through targeted feature editing.