Online Self-Calibration Against Hallucination in Vision-Language Models

1. Problem Statement

Large Vision-Language Models (LVLMs) frequently suffer from hallucinations, generating descriptions containing visual details absent from the input image (e.g., non-existent objects, incorrect attributes).

• Limitation of Current Methods: Recent approaches rely on offline preference alignment (e.g., RLHF, DPO) using supervision distilled from stronger teacher models (such as GPT-4) or human annotations.
• The Core Problem: Mismatch Between Supervision and Perception: The paper argues that forcing a weaker student model to align with the fine-grained, high-fidelity descriptions of a stronger teacher creates a mismatch. The student model lacks the perceptual capacity to verify these details and is forced to "guess" based on linguistic priors rather than "seeing" the image. Paradoxically, this leads to an increase in hallucination rates, as the model learns to mimic ungrounded details it cannot perceive.

2. Key Observations and Motivations

The authors identify two critical phenomena to address the problem:

1. Mismatch Between Supervision and Perception: Empirical experiments show that fine-tuning LLaVA-1.5 on data distilled from a stronger model (Qwen3-VL) actually degrades performance on hallucination benchmarks (e.g., POPE, CHAIR) compared to the baseline. The model learns to hallucinate to match the teacher's imperceptible details.
2. Gap Between Generation and Discrimination: LVLMs exhibit a significant performance gap between open-ended generation and discriminative verification.
   • Generation: The model is often subject to "linguistic inertia," hallucinating objects to meet prompt expectations.
   • Verification: When explicitly asked "Is object X in the image?", the same model demonstrates significantly higher accuracy in correctly identifying the absence of objects.
   • Implication: The model possesses a latent self-verification capability that remains unused during standard generation.

3. Methodology: OSCAR Framework

The authors propose Online Self-CAlibRation (OSCAR), a framework that leverages the model's own discriminative capabilities to generate high-quality training data online, thereby avoiding the mismatch between supervision and perception.

A. MCTS-Driven Generation

Instead of standard greedy search or beam search, OSCAR uses Monte-Carlo Tree Search (MCTS) to explore the generation space. This enables a "lookahead" that identifies tokens which appear locally safe but could lead to downstream hallucinations.

• State: Partial answers (sentences).
• Action: Generation of the next sentence.
• Search Phases: Selection (PUCT criterion), Expansion (sampling candidates), Evaluation (reward mechanism), and Backpropagation.

B. Dual-Granularity Reward Mechanism

To guide the MCTS, a novel reward system was designed to balance local fidelity and global quality:

1. Process Reward (Node-Level):
   • For each generated sentence, the model is prompted to act as a discriminator: "Does this sentence mention objects not present in the image?"
   • The reward is the probability with which the model answers "No" (indicating the sentence is free of hallucinations).
   • Goal: Suppression of early tokens that carry the risk of future hallucinations.
2. Gated Outcome Reward (Trajectory-Level):
   • Evaluates the complete answer only if it passes a strict fidelity check (comparing extracted objects with Ground Truth).
   • If the trajectory contains hallucinations, the reward is 0.
   • If it passes, a quality assessment (logical consistency, fluency, redundancy) is assigned.
   • Goal: Ensuring the final output is both factually grounded and of high quality.

C. Iterative Preference Learning (DPO)

• Data Creation: Preference pairs are extracted from the MCTS tree:
  • Global Path: Comparison of the trajectory with the highest value (Selected) vs. the trajectory with the lowest value (Rejected).
  • Sibling Comparison: Comparison of the optimal node at a specific depth with its worst-performing sibling node.
• Training: The model is updated via Direct Preference Optimization (DPO).
• Online Loop: The updated model generates new preference data in the next iteration, creating a positive feedback loop where the training distribution evolves alongside the model's improving perceptual boundaries.

4. Main Contributions

1. Identification of the Mismatch: Empirical evidence that offline distillation from stronger teachers can exacerbate hallucinations due to the mismatch between supervision and perception.
2. OSCAR Framework: Proposal of a novel training paradigm that exploits the gap between generation and discrimination. By integrating MCTS with a dual-granularity reward mechanism, it enables the model to self-correct and suppress hallucination-inducing tokens via lookahead.
3. State-of-the-Art Performance: Achievement of SOTA results on hallucination benchmarks while simultaneously improving general multimodal understanding capabilities.

5. Experimental Results

Experiments were conducted on LLaVA-1.5 (7B and 13B) across multiple benchmarks (Object-HalBench, AMBER, MM-VET, POPE).

• Reduction of Hallucinations:
  • For LLaVA-1.5-7B, OSCAR reduced CHAIRs from 49.0 to 27.6 and CHAIRi from 14.3 to 8.2.
  • For LLaVA-1.5-13B, the improvement was even more drastic, with CHAIRs reduced from 44.8 to 5.4 (an 87.9% reduction).
• General Capabilities: Unlike methods that sacrifice fluency for accuracy, OSCAR improved the MM-VET score (general multimodal understanding) for the 7B model from 32.5 to 34.6.
• Iterative Improvement: Performance consistently improved over 3 iterations, validating the online self-calibration loop.
• Ablation Studies: Confirmed that all three components (Process Reward, Gated Outcome Reward, and MCTS) are essential. Removing MCTS or the specific reward mechanisms led to significant performance drops.
• Comparison with SFT: SFT using OSCAR-constructed data performed significantly better than SFT with teacher-distilled data (which increased hallucinations) and SFT with the model's own raw generation.

6. Significance

This paper offers a paradigm shift in curbing LVLM hallucinations:

• From Offline to Online: It shifts focus from reliance on external, potentially mismatched supervision to self-supervised online learning.
• Respecting Perceptual Boundaries: It emphasizes that training data must respect the model's intrinsic perceptual limits. Forcing a model to describe what it cannot see leads to failure; allowing it to verify what it sees leads to success.
• Self-Correction: It demonstrates that LVLMs possess a latent ability to verify their own outputs, which can be unlocked through discriminative prompting and MCTS, enabling continuous self-improvement without external human or teacher labels.