Technical Summary: Wiener Representation Filtering for VLM Hallucination Suppression

Problem Statement

Vision-Language Models (VLMs) excel at tasks like image captioning and visual question answering but frequently suffer from object hallucination (OH). This phenomenon involves the generation of plausible but factually incorrect descriptions, such as inventing objects, attributes, or spatial relations absent from the input image. Existing mitigation strategies generally fall into two categories:

1. Decoding-time methods: Techniques like constrained beam search or logit debiasing that modulate generation but often incur significant inference overhead (5–10x slowdowns).
2. Post-hoc refinements: Methods like Woodpecker or HALC that require task-specific adaptations or external resources.
3. Training-based methods: Approaches requiring retraining or fine-tuning with hallucination-aware losses, which are computationally expensive.

There remains a gap for a training-free, inference-efficient correction that operates without modifying the model's architecture or increasing runtime costs.

Methodology

The authors propose a Wiener Representation Filtering framework, a post-hoc editing technique that operates in the representation space of the language backbone. The method treats hallucination as a structured spectral distortion and corrects it via a linear minimum mean-square error (MMSE) estimation problem.

1. Theoretical Formulation

The core hypothesis is that hidden states ($h$) in a VLM can be modeled as the superposition of a truthful component ($s$) and a hallucination-associated distortion ($n$):
$$h = s + n$$
Assuming $s$ and $n$ are zero-mean and mutually uncorrelated, the covariance of the observed representation decomposes into the sum of the truthful covariance ($\Sigma_T$) and the hallucination covariance ($\Sigma_H$).

The optimal linear estimator to recover $s$ from $h$ is the Wiener operator:
$$A^* = \Sigma_T (\Sigma_T + \Sigma_H)^{-1}$$

2. Calibration and Estimation

The method requires a one-time, offline calibration on a modest paired dataset (e.g., LURE) containing:

• $c^-$: Human-verified truthful captions.
• $c^+$: Hallucinated captions generated by the model for the same image.

For selected deeper layers, the authors extract sequence-averaged representations ($x^-$ and $x^+$) to estimate:

• $\Sigma_T$: The covariance of truthful representations.
• $\Sigma_H$: The covariance of the distortion ($d = x^+ - x^-$).

Empirical analysis reveals that hallucination manifests as an anisotropic distortion concentrated in a low-rank subspace (a few dominant eigenmodes), rather than isotropic noise.

3. Spectral Filtering

To ensure stability and interpretability, the Wiener operator is approximated in the eigenbasis of $\Sigma_H$. Let $\{q_j\}$ be the eigenvectors of $\Sigma_H$ with eigenvalues $\lambda_j$. The signal variance in mode $j$ is $\tau_j^2 = q_j^\top \Sigma_T q_j$.
The distortion-to-signal ratio is defined as $\nu_j = \lambda_j / \tau_j^2$.
The mode-wise Wiener gain is:
$$\gamma_j = \frac{1}{1 + \nu_j}$$
To allow for tunable sharpness and compensate for estimation errors, a sharpness exponent $\alpha$ is introduced:
$$\tilde{\gamma}_j = (1 + \nu_j)^{-\alpha} = \left( \frac{\tau_j^2}{\tau_j^2 + \lambda_j} \right)^\alpha$$
The final filter $F_\alpha$ is constructed as $Q \text{diag}(\tilde{\gamma}_1, \dots, \tilde{\gamma}_d) Q^\top$.

4. Implementation

The filter is applied by modifying the feed-forward network (FFN) output projections ($W_{out}$) of selected deeper layers:
$$\tilde{W}_{out} = F_\alpha W_{out}$$
Since $F_\alpha$ is precomputed and absorbed directly into the existing weights, the architecture, parameter count, and inference speed remain unchanged. No gradient updates or fine-tuning are performed during deployment.

Key Contributions

• Training-Free Correction: A post-hoc method that requires no retraining, fine-tuning, or gradient updates.
• Inference Efficiency: The correction is absorbed into model weights, incurring zero additional latency or memory overhead at inference time.
• Theoretical Grounding: Formulates hallucination mitigation as a linear MMSE problem, deriving a closed-form Wiener estimator that respects stability criteria (operator-Lipschitz continuity).
• Spectral Analysis: Demonstrates that hallucinations occupy a low-dimensional, anisotropic subspace in representation space, allowing for targeted attenuation without destroying semantic information.

Experimental Results

The method was evaluated on LLaVA-1.5, MiniGPT-4, mPLUG-Owl2, and Gemma3 across multiple benchmarks:

• CHAIR (Caption Hallucination): The method consistently achieved the lowest hallucination rates (CHAIRS and CHAIRI) across all models, outperforming decoding-based baselines (OPERA, VCD, HALC) and representation-editing baselines (Nullu). For example, on LLaVA-1.5, CHAIRS was reduced from 18.87 (Greedy) to 14.93.
• POPE (Object Presence): Improved factual grounding (Accuracy, Precision, F1) across Random, Popular, and Adversarial settings, often achieving the highest scores among compared methods.
• MME (Multimodal Evaluation): Demonstrated superior performance on 10 visual grounding subsets (e.g., Count, Artwork, Celebrity) compared to competing strategies.
• Generalization:
  • Video Understanding: Successfully reduced hallucinations in the TempCompass benchmark (temporal reasoning) despite being calibrated on static images.
  • Discrete Diffusion Models: Applied to Dream-v0-Instruct-7B for grounded dialogue (FaithDial), significantly reducing hallucination rates (e.g., from 92% to 74% via FaithCritic) while improving grounding metrics.
  • Cross-Domain Transfer: Showed that filters learned on one dataset (e.g., LURE) could effectively reduce hallucinations on unseen datasets (e.g., TruthfulQA, CHAIR) without retuning.

Significance and Claims

The paper claims that by modeling hallucination as a structured spectral distortion rather than random noise, it is possible to derive a principled, stable correction that suppresses hallucination-dominated modes while preserving semantically grounded directions.

Key claims regarding the method's impact include:

1. Stability: The continuous formulation ensures that small errors in covariance estimation induce only proportionally bounded changes in the operator, avoiding the instability of hard rank truncation.
2. Preservation of Fluency: Unlike methods that aggressively prune representations, the Wiener filter attenuates modes based on their distortion-to-signal ratio, maintaining caption fluency and overall response quality (BLEU scores remain competitive).
3. Generality: The approach is not limited to specific VLM architectures or fusion strategies (early, bridging, mid-fusion) and extends to temporal video reasoning and diffusion-based language models.
4. Efficiency: It offers a "plug-and-play" solution that eliminates the inference slowdowns associated with decoding-time interventions and the computational cost of training-based methods.

The authors conclude that this work provides a principled mechanism for hallucination suppression that aligns generation with visual evidence without compromising the model's internal representation integrity or inference efficiency.