💻 computer science

Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMs

The paper proposes AVES-DPO, a self-corrected preference learning framework that mitigates hallucinations in Large Vision-Language Models by generating in-distribution preference pairs through a consensus-based verification mechanism, thereby overcoming the distributional mismatch caused by reliance on proprietary models.

Original authors: Byeonggeuk Lim, JungMin Yun, Junehyoung Kwon, Kyeonghyun Kim, YoungBin Kim

Published 2026-04-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Byeonggeuk Lim, JungMin Yun, Junehyoung Kwon, Kyeonghyun Kim, YoungBin Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, artistic robot that loves to describe pictures. You show it a photo of a kitchen, and it says, "I see a brown cabinet, a wooden floor, and a person standing next to a dog." But wait—there's no dog in the picture! The robot just made it up. This is called a hallucination. It's like the robot is daydreaming while looking at the photo.

For a long time, scientists tried to fix this by hiring a "super-expert" (like a massive, expensive AI model called GPT-4V) to look at the robot's mistakes and tell it, "No, that dog isn't there. Fix it." They would then teach the robot using these corrections.

The Problem with the Old Way
The authors of this paper realized there was a flaw in this plan. Imagine teaching a student to paint by having them copy a masterpiece by a completely different artist. The student might learn the style of the master, but it won't match their own natural way of painting.

In the same way, when the "super-expert" corrects the robot, the correction sounds like the expert, not the robot. This creates a mismatch. The robot gets confused because the "right answer" sounds foreign to its own brain, making the learning process inefficient and requiring huge amounts of data.

The New Solution: "AVES-DPO" (The Self-Correcting Artist)
The authors propose a new method called AVES-DPO. Instead of hiring an outside expert, they teach the robot to check its own work and fix it.

Here is how their "Self-Correction" process works, broken down into simple steps:

The First Draft (The Mistake): The robot looks at the picture and writes a description. It might accidentally invent a dog or get the color of the cabinets wrong.
The Detective Work (Verification): Before the robot tries to fix it, they use a team of "detectives" (specialized, open-source AI tools) to scan the description.
- Did the robot say there's a dog? The detective checks the photo. "Nope, no dog."
- Did it say the cabinets are blue? The detective checks. "No, they are brown."
- Did it say the person is holding a cup? The detective checks. "No cup."
- If the detectives agree, the mistake is confirmed. If they are unsure, they ask a second opinion to be sure.
The Self-Correction (The Fix): Now, the robot looks at the list of confirmed mistakes. It rewrites its story.
- It removes the fake dog.
- It changes "blue cabinets" to "brown cabinets."
- Crucially, it doesn't just delete the bad parts; it also adds more real details it missed, like "the floor is shiny" or "there is a window."
The Lesson (Preference Learning): Now the robot has two versions of the story:
- Version A: The original, mistake-filled story (the "bad" answer).
- Version B: The new, self-corrected, detailed story (the "good" answer).
- The robot is trained to prefer Version B. Because it wrote Version B itself, the "good" answer sounds exactly like it. It fits perfectly into its own brain.

Why This is a Big Deal
The paper claims this method is a game-changer for three reasons:

It's Efficient: Because the robot is learning from its own "voice," it learns much faster. They only needed about 5,200 examples to teach the robot. Other methods needed 25 times more data (over 120,000 examples) to get similar results. It's like learning a language by talking to yourself in a mirror versus trying to mimic a foreign accent.
It Catches More Mistakes: Old methods mostly caught "big" mistakes, like inventing a whole object (a dog). This new method catches "tiny" mistakes too, like getting the relationship wrong (saying the dog is on the left when it's actually on the right) or the wrong color.
It's Cheaper: You don't need to pay for expensive "super-expert" AI models to do the correcting. The robot does the heavy lifting.

The Result
When they tested this new method, the robot became much better at describing pictures accurately. It stopped making up objects and got the details right, all while using a tiny amount of training data.

A Note on Limits
The authors are honest about what their method doesn't do yet.

It works best when looking at one object at a time. If a picture has a chaotic crowd of 50 people interacting in complex ways, the system might get a little confused.
For the biggest, most powerful robots (the 13-billion-parameter models), fixing the hallucinations made them slightly more "cautious." They became so good at not making things up that they sometimes forgot to mention general knowledge they used to know. It's a trade-off between being 100% factual and being 100% chatty.

In short, AVES-DPO is about teaching the robot to be its own best editor, ensuring that the "truth" it learns is the kind of truth that fits naturally into its own mind.

1. Problem Statement

Large Vision-Language Models (LVLMs) frequently suffer from hallucinations, generating content inconsistent with input images (e.g., non-existent objects, incorrect attributes, or wrong relationships). While Direct Preference Optimization (DPO) has emerged as a mainstream solution to mitigate this by aligning models with human preferences, existing approaches face two critical limitations:

Distributional Mismatch: Most methods rely on proprietary models (e.g., GPT-4V) to generate "preferred" (factual) responses. This introduces a distribution shift because the correction patterns of the external teacher model differ from the target LVLM's intrinsic distribution, hindering efficient alignment.
Bias Toward Object Existence: Existing datasets and training objectives primarily focus on object presence (binary existence), neglecting fine-grained hallucinations involving attributes (e.g., color, material) and relationships (e.g., spatial positioning).

2. Methodology: AVES-DPO

The authors propose AVES-DPO (Alignment via VErified Self-correction DPO), a framework that eliminates reliance on external proprietary models by leveraging the target model's intrinsic knowledge to generate in-distribution preference data. The framework consists of two distinct stages:

A. Hallucination Verification

This stage diagnoses errors in the model's initial response ( $y_{init}$ ) using a consensus-based verification mechanism:

Scene Graph Extraction: The response is parsed into a structured scene graph containing Objects ( $O$ ), Attributes ( $A$ ), and Relationships ( $R$ ).
Consensus Verification: Two independent verification models ( $V_1, V_2$ $V_{1}, V_{2}$ ) check the factuality of each element.
- Object Verification: Uses robust open-source detectors (YOLO, Grounding DINO). Ambiguous cases are resolved by a secondary verification using Qwen3-VL models.
- Attribute & Relation Verification: Only verified objects are checked for attributes and relationships using Qwen3-VL models.
- Consensus Rule: A label is finalized only if both models agree; otherwise, it is marked "Ambiguous."
Correction Candidates: If an element is hallucinated, the system retrieves a correct replacement from a pre-constructed vocabulary (derived from GQA and MS-COCO) based on semantic subtypes.

B. Self-Correction

Guided by the verification diagnoses, the LVLM refines its own output in two phases:

Factual Rectification: The model removes hallucinated elements or replaces them with verified correct content, producing a concise, factually accurate response.
Detailed Enrichment: To prevent the response from becoming too sparse, the model adds missing visual details grounded in the image.
- Iterative Filtering: A cyclic pipeline ensures quality. The enriched text is fed back into the verification module. If hallucinations are detected, it is re-generated. This loop runs up to 3 times (for 7B models) or 5 times (for 13B models).
- Negative Constraints: The model is explicitly instructed not to mention objects previously identified as hallucinated.

C. Preference Learning

The process generates a preference dataset $\mathcal{D}_{SC}$ consisting of pairs $(x, y^+, y^-)$ , where:

$y^-$ : The initial hallucinated response.
$y^+$ : The final self-corrected, enriched response.
Training Objective: The model is optimized using the standard DPO loss function to maximize the likelihood of $y^+$ over $y^-$ , effectively teaching the model to prefer its own verified, factual outputs.

3. Key Contributions

In-Distribution Alignment: AVES-DPO is the first framework to generate preference pairs strictly from the target model's intrinsic knowledge, resolving the distribution mismatch problem inherent in using external "teacher" models.
Comprehensive Hallucination Mitigation: Unlike prior works focusing mainly on object existence, AVES-DPO addresses a broad spectrum of hallucinations, including attributes and relationships, through its granular verification pipeline.
High Data Efficiency: The method achieves state-of-the-art performance using only 5.2k samples (for the 7B model), which is approximately 25 times less data than methods like LLaVA-RLHF, without requiring expensive proprietary API calls.
Robust Verification Pipeline: The introduction of a two-phase, consensus-based verification strategy (combining detectors and VLMs) ensures high-quality, noise-free preference data.

4. Experimental Results

The authors evaluated AVES-DPO on LLaVA-1.5 (7B and 13B) across multiple benchmarks:

Hallucination Benchmarks (Object-Hal, AMBER, MMHal-Bench):
- AVES-DPO achieved SOTA performance on over 60% of evaluation metrics for the 7B model.
- It significantly reduced CHAIR scores (hallucination rates) compared to baselines like POVID, oDPO, and SENTINEL.
- Fine-Grained Gains: The method showed substantial improvements in relationship and attribute categories, where other methods often struggle. For instance, on the AMBER benchmark, it achieved a 66.6 score in the "Relation" category for the 7B model, outperforming baselines that often degraded performance in this area.
General Multimodal Capabilities (MME Benchmark):
- The 7B model showed significant improvements in general perception and recognition tasks, confirming that hallucination mitigation did not degrade general capabilities.
- Note: The 13B model showed a slight trade-off in general benchmarks, suggesting a need for scale-aware strategies to balance factual grounding with broad knowledge retrieval.
Ablation Studies:
- Distribution Mismatch: Visualizing preference margins showed that self-correction aligns with the model's internal distribution (margin near zero/positive), whereas external GPT-4V supervision caused a negative shift (distribution mismatch).
- Data Efficiency: Performance gains were significant even with as few as 600 samples, with diminishing returns observed beyond 5.2k samples.

5. Significance

This paper presents a paradigm shift in hallucination mitigation for LVLMs. By demonstrating that self-correction is superior to external supervision for preference learning, AVES-DPO offers a cost-effective, scalable, and data-efficient solution. It removes the dependency on expensive proprietary APIs (like GPT-4V) while simultaneously addressing a wider range of hallucination types (objects, attributes, relations). The framework proves that LVLMs can effectively learn to align with their own internal knowledge bases to produce more factual and reliable outputs.