Technical Summary: Vision-driven Preference Synthesis for Mitigating Hallucinations in VLMs

Problem Statement

Vision-Language Models (VLMs) have demonstrated significant capabilities in visual understanding but remain prone to hallucinations, where models generate content inconsistent with the input image, often inventing non-existent objects. This issue stems from an imbalance between language and vision modalities, where strong language priors cause models to rely on text statistics rather than visual evidence.

While Preference Alignment (specifically Direct Preference Optimization, or DPO) has emerged as a promising solution for improving visual faithfulness, its efficacy is heavily dependent on the quality of the constructed preference pairs (preferred vs. dispreferred responses). Existing methods for constructing these pairs suffer from two primary limitations:

1. Intervention-based methods (e.g., hallucination injection or correction) often introduce responses that deviate significantly from the model's natural policy distribution, creating a distribution mismatch.
2. Sampling-based methods (e.g., external-judged sampling or SFT-guided sampling) reduce this distribution mismatch but frequently underutilize visual information, relying instead on language priors to determine preference.

Methodology: ViPSy

The authors propose ViPSy (Vision-driven Preference Synthesis), a two-stage framework designed to construct preference data that is both policy-aligned (close to the model's response distribution) and visually grounded (relying on image evidence).

Stage 1: Self-Captioned Semantic Synthesis for Consistent Cue Extraction

To extract reliable visual cues without external bias, ViPSy leverages the policy model's own capabilities:

1. Self-Captioning: The policy model ($\pi_\theta$) generates $N_c$ captions for a source image $m$ using a standard captioning prompt.
2. Semantic Synthesis: These captions serve as prompts for a Text-to-Image (T2I) diffusion model ($P_\psi$) to synthesize $N_c$ semantically aligned image variants ($m'_j$).
3. Cue Extraction: A judge model ($\pi_\phi$) compares each synthetic variant $m'_j$ with the original image $m$ to extract pairwise object-level evidence ($c_j$) that is consistent between them.
4. Aggregation: The judge aggregates these pairwise cues into a single, consistent visual cue ($c$). The intuition is that objects recurring across diverse synthetic variations derived from the same source image are highly likely to be grounded in the original image.

Stage 2: Cue-Conditioned Self-Distillation for Preference Synthesis

Once the visual cue $c$ is established, the framework generates preference pairs:

1. Cue-Conditioned Rollouts: The policy model samples $N_s$ candidate responses ($y^{(s)}$) conditioned on the original image $m$, the prompt $x$, and the extracted visual cue $c$. This ensures candidates remain close to the policy distribution while being explicitly guided by visual evidence.
2. Preference Selection: The judge model evaluates the set of candidates $\{y^{(s)}\}$ against the original image $m$ to select a preferred response ($y_w$) (most visually grounded) and a dispreferred response ($y_l$) (least grounded).
3. Optimization: The resulting dataset $\mathcal{D}_{pref} = \{(x, m, y_w, y_l)\}$ is used to train the model via Direct Preference Optimization (DPO). Crucially, the visual cues and synthetic images are only used during data construction; the final optimization step conditions the policy solely on the original prompt-image pair, distilling the visual grounding capability into the model.

Key Contributions

1. ViPSy Framework: A novel vision-driven preference synthesis approach that constructs visually grounded preference pairs while maintaining alignment with the policy's response distribution.
2. Two-Stage Strategy:
   • Self-captioned semantic synthesis: Uses the policy's own captions to drive T2I synthesis, aggregating recurring object-level evidence to extract consistent visual cues, thereby reducing reliance on language priors.
   • Cue-conditioned self-distillation: Guides the policy's own rollouts with these cues to generate candidates that vary in visual grounding quality, enabling the selection of high-quality preference pairs.
3. State-of-the-Art Performance: The framework achieves new state-of-the-art results in hallucination mitigation and improves general visual grounding capabilities.

Experimental Results

The authors evaluated ViPSy on LLaVA-1.5 (7B and 13B) and various other policy models (Qwen2.5-VL, InternVL3.5, etc.), comparing against decoding methods, unlearning-based approaches, and existing preference alignment baselines.

• Hallucination Mitigation:
  • On LLaVA-1.5-7B, ViPSy reduced hallucination rates on AMBER by 35.7% and Object HalBench by 24.5% compared to the previous state-of-the-art (SENTINEL).
  • On LLaVA-1.5-13B, it outperformed all baselines across all hallucination benchmarks, showing significant improvements over the previous best methods.
• General Visual Grounding:
  • Unlike some baselines that degrade general performance, ViPSy improved scores on general benchmarks requiring visual grounding: MMStar (+6.0%), MMVP (+31.3%), and CV-Bench (+4.0%) on the 7B model.
• Vision Encoder Improvements:
  • Linear probing and segmentation evaluations on ImageNet-1k, ImageNet-O, and MS COCO showed gains in accuracy and recall. This suggests ViPSy enhances the underlying visual representations, not just the response-level generation.
• Generalization:
  • The method demonstrated consistent improvements across diverse policy models (Qwen, InternVL, mPLUG-Owl, Kimi), indicating it is not specific to a single architecture.
• Ablation Studies:
  • Self-rollouts were found superior to external VLM rollouts, confirming the importance of staying close to the policy distribution.
  • Diffusion-based synthesis outperformed simple image transformations (resize, crop) and video-based variations, highlighting the need for rich semantic diversity in cue extraction.
  • The method remained robust across different judge models (Qwen3-VL, GPT-5 mini).

Significance and Claims

The paper claims that ViPSy addresses the fundamental trade-off in preference construction: balancing policy alignment with visual grounding. By deriving visual cues from the intersection of the policy's own semantic understanding and synthetic image variations, ViPSy avoids the distribution shift of intervention-based methods and the visual neglect of standard sampling methods.

The authors assert that their approach not only sets a new state-of-the-art in reducing hallucinations but also enhances the model's general visual capabilities and the quality of its vision encoder representations. They emphasize that the framework effectively distills visual grounding into the policy without requiring complex reward modeling or online reinforcement learning, making it a practical and effective solution for improving VLM reliability in real-world applications.