Technical Summary: VADER – Adaptive Debiasing for Hallucination Mitigation in Video Large Language Models

1. Problem Statement

Video Large Language Models (VideoLLMs) have achieved significant success in open-ended video understanding tasks, such as event summarization and temporal reasoning. However, these models remain prone to hallucinations, generating linguistically fluent responses that lack support from actual video evidence. These errors are particularly prevalent in action recognition, event ordering, causal relationships, and temporal consistency.

Existing hallucination mitigation strategies face two primary limitations when applied to video:

1. Training-based methods require costly retraining, additional supervision, or reward modeling.
2. Training-free methods often rely on globally fixed visual interventions or input perturbations (e.g., noise, frame dropping). These approaches fail to account for video-specific challenges:
   • Static Intervention: A fixed layer range and strength cannot accommodate the video-dependent depths at which visual evidence competes with system prompts and historical text.
   • Inadequate Contrastive Evidence: Coarse input perturbations often leave evidence recoverable through redundant neighboring frames, or they disrupt the temporal dependencies required for consistent reasoning.

The core challenge is to improve video grounding during decoding without retraining the model, while addressing the unique redundancy and temporal dependencies of video data.

2. Methodology

The authors propose VADER (Video-Adaptive Debiasing via Evidence Reweighting), a training-free framework that operates via a dual-branch design to calibrate evidence grounding and suppress prior-biased predictions. The framework consists of two complementary modules and a fusion mechanism:

2.1 Visual Focus Reallocation (VFR)

VFR treats visual enhancement as an automated, video-conditioned policy rather than a fixed configuration. It dynamically determines where (which layers) and how strongly to intervene for each specific video-question input.

• Diagnosis: Before generation, VFR performs a diagnostic forward pass using the video-question input to compute a diagnostic loss ($L_{diag}$) based on the model's initial prediction. It calculates attention saliency by combining attention mass with its first-order sensitivity to this loss.
• Policy Derivation: It measures the cross-modal saliency flow (visual-to-text flow) across decoder layers. The top-$K$ layers with the highest normalized flow are selected as intervention points ($L_{sel}$).
• Adaptive Reweighting: The strength of the intervention ($a$) is derived from the aggregated saliency flow of the selected layers using a logarithmic mapping. VFR then applies multiplicative reweighting to the pre-softmax attention scores:
  • System tokens are suppressed by a factor of $\gamma_{sup} = 1 - a$.
  • Video tokens are enhanced by a factor of $\gamma_{enh} = 1 + a$.
    This reallocates attention toward video evidence specifically where the model relies on it for the current input.

2.2 Selective Evidence Erasure (SEE)

To construct a robust "prior-biased" branch that cannot be easily compensated by temporal redundancy, SEE operates inside the decoder on a per-frame basis.

• Mechanism: Instead of dropping frames or masking input tokens (which can be recovered from neighbors), SEE identifies high-importance visual tokens within each frame based on their attention scores relative to the current decoding position.
• Erasure: It retains only the lowest-scoring tokens (background/weak context) and masks the high-scoring (discriminative) tokens by setting their attention scores to $-\infty$ at a fixed insertion layer ($l^*$).
• Result: This creates a negative branch ($z_{prior}$) that reflects language-prior-driven predictions and background context, effectively removing the specific visual evidence supporting the answer while preserving temporal structure.

2.3 Joint Temporal Contrastive Suppression

The framework fuses the outputs of the two branches:

• Grounded Branch ($z_{gnd}$): Produced by VFR, maximizing the use of video evidence.
• Prior-Biased Branch ($z_{prior}$): Produced by SEE, reflecting predictions driven by language priors.
• Fusion: The final logits ($\bar{z}$) are computed as $\bar{z} = (1 + \lambda)z_{gnd} - \lambda z_{prior}$. If a token is highly confident in both branches, it is likely a language prior and is suppressed. If it is confident only in the grounded branch, it is preserved or amplified.

3. Key Contributions

1. VADER Framework: A training-free, dual-branch framework for mitigating hallucinations in VideoLLMs that calibrates evidence grounding without parameter updates.
2. VFR (Visual Focus Reallocation): A novel mechanism that formulates visual enhancement as a video-conditioned intervention policy, automatically selecting intervention layers and response strength based on input-specific visual-to-text flow.
3. SEE (Selective Evidence Erasure): A decoder-internal, per-frame erasure technique that masks high-importance visual tokens to construct a prior-biased branch resistant to cross-frame compensation.
4. Performance: The method achieves significant improvements on event-level hallucination benchmarks, specifically reaching 72.60% accuracy on EventHallusion using LLaVA-Video-7B, surpassing training-based methods like TPO and RRPO.

4. Experimental Results

The authors evaluated VADER on three video hallucination benchmarks (VidHalluc, VideoHallucer, EventHallusion) and three general video understanding benchmarks (MVBench, VideoMME, TVBench) across three backbones (LLaVA-OneVision-7B, LLaVA-NeXT-Video-7B, Qwen3-VL-8B).

• Hallucination Mitigation: VADER yielded substantial gains, particularly on EventHallusion (improving LLaVA-Video-7B from 63.57% to 72.60%) and VideoHallucer. It outperformed representative training-free baselines (TCD, DINO-HEAL) and even surpassed training-based methods (TPO, RRPO) on EventHallusion without any retraining.
• General Understanding: Unlike some interventions that degrade general performance, VADER maintained or slightly improved scores on general video understanding benchmarks (e.g., +1.5 points on LLaVA-Video-7B average), indicating it improves faithfulness without sacrificing broad perception or reasoning.
• Ablation Studies:
  • Removing the adaptive policy (using fixed layers) resulted in lower gains, confirming the value of video-conditioned selection.
  • Removing SEE (VFR only) or VFR (SEE only) resulted in lower performance than the full model, demonstrating the complementarity of evidence reallocation and prior-biased branch construction.
  • Sensitivity analysis showed that moderate retention ratios for SEE (keeping ~30% of tokens) yield optimal results, balancing evidence removal with context preservation.

5. Significance and Claims

The paper claims that VADER addresses a critical gap in deploying VideoLLMs: the ability to mitigate hallucinations in a plug-and-play manner without the cost of retraining. By moving away from static, global interventions to adaptive, input-specific policies, VADER effectively handles the complex, redundant, and temporally dependent nature of video data.

The authors emphasize that their approach does not merely "fix" errors but fundamentally alters the decoding dynamics to distinguish between evidence-grounded predictions and language-prior-driven hallucinations. The results suggest that effective hallucination mitigation in video requires not just stronger visual attention, but also a mechanism to explicitly expose and suppress predictions that rely on priors when visual evidence is removed.

Limitations: The authors acknowledge that VADER incurs additional inference-time computation and memory overhead due to the gradient-based layer selection and multi-pass forward passes, suggesting efficient saliency estimation as a future direction.