💬 NLP

PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment

This paper introduces Perception-Enhanced Alignment Direct Preference Optimization (PEA-DPO), a novel framework that addresses the visual insensitivity of multimodal large language models by explicitly leveraging visual preference signals, thereby significantly reducing hallucinations and improving multimodal alignment while preserving language capabilities.

Original authors: Jiawei Feng, Jiancan Wu, Xingyu Zhu, Junkang Wu, Xiang Wang, Xiangnan He

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiawei Feng, Jiancan Wu, Xingyu Zhu, Junkang Wu, Xiang Wang, Xiangnan He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a new generation of systems has emerged that can see and speak at the same time. These multimodal large language models are trained to look at an image and describe what they see, or to answer questions about a photograph. For these machines to be truly useful and safe, they must align with human values, meaning their answers need to be accurate, helpful, and grounded in the visual reality presented to them. A major challenge in teaching these systems has been a tendency to hallucinate, or to confidently describe objects or actions that simply are not there. To fix this, researchers have turned to a technique called direct preference optimization, a method that teaches the model by showing it pairs of answers—one good and one bad—and asking it to learn why the first is better. This approach has worked wonders for text-only models, but when applied to systems that also process images, a hidden flaw has been discovered that prevents them from truly "seeing" the difference between a clear photo and one where the most important details have been obscured.

A team of researchers at the University of Science and Technology of China and the National University of Singapore has identified this specific weakness and developed a new framework to cure it. They found that standard training methods often treat the image merely as a backdrop, a static setting for the conversation, rather than as the primary source of truth that the model must rely on. Because of this, the models became visually insensitive. In their experiments, the researchers showed that these models could not tell the difference between a picture of people having a picnic and a picture of people reading, even when the visual evidence for one was completely removed. The models would generate nearly identical responses for both, failing to notice that the key visual clues were missing. They also struggled to distinguish between specific objects within a single image, such as confusing a toilet brush with a generic tool, because they were not paying close enough attention to the critical visual cues that defined the object.

To solve this, the researchers created a method they call Perception-Enhanced Alignment. Instead of just teaching the model to choose the better text answer, they redesigned the training process to force the model to pay attention to the visual context itself. They began by taking a clear image and creating a series of altered versions where random parts were masked out. Using a separate, highly sensitive visual system, they identified which altered version had lost the most critical information, effectively creating a "rejected" image that was missing the key details needed to answer a question correctly. They then paired this missing-information image with the original, complete image. During training, the model was presented with the same question and the same text answer, but shown first with the complete image and then with the incomplete one. The model was taught that it should strongly prefer the answer when it had the full visual evidence, and that it should recognize the answer as less reliable when the key visual context was gone.

This dual approach taught the model two things at once. First, it learned to distinguish between a good text response and a bad one, just as before. Second, and more importantly, it learned to recognize when the visual evidence was insufficient to support a claim. The researchers tested this new method on two different sizes of image-text models, one with seven billion parameters and another with thirteen billion. The results were striking. On standard tests designed to measure how often these models make things up, the new method reduced hallucinations by significant margins. For the smaller model, the rate of made-up details dropped by nearly a third on one major benchmark, and by over ninety percent on another. The larger model showed similar improvements, often outperforming even some of the most advanced commercial systems available. The models became much better at describing exactly what was in the picture, such as correctly identifying a book title or a specific tool, rather than guessing based on general knowledge.

The researchers also checked to ensure that this new focus on visual accuracy did not make the models worse at other tasks. They found that the models retained their ability to follow instructions and answer general questions, and in some cases, their performance on these general tasks actually improved slightly. The only noticeable trade-off was that the models became slightly more conservative; they were less likely to guess when they were unsure, which meant they sometimes provided fewer details overall, but the details they did provide were far more likely to be true. By explicitly teaching the model to value the visual evidence as much as the text, the researchers demonstrated that it is possible to build artificial intelligence that does not just look at a picture, but truly sees it. This work suggests that for machines to be reliable partners in understanding the visual world, they must be trained to feel the weight of missing information, ensuring that their confidence is always matched by the clarity of what is actually before their eyes.

Technical Summary: PEA-DPO

Problem Statement

While Direct Preference Optimization (DPO) has proven effective for aligning text-only Large Language Models (LLMs) with human preferences, its direct application to Multimodal Large Language Models (MLLMs) reveals a critical limitation: visual insensitivity. The authors identify that standard multimodal DPO often treats images merely as static conditioning inputs rather than active targets of preference. Consequently, models fail to distinguish between an original image and one where critical visual context has been removed, leading to responses that are weakly grounded in visual evidence.

Through representational analysis and theoretical decomposition, the authors formalize this failure into two distinct manifestations:

Across-Image Insensitivity: The model assigns nearly identical preference scores to responses generated from distinct images (e.g., failing to differentiate between an image of an outdoor reading activity and a picnic), indicating a lack of discrimination between different visual contexts.
Within-Image Insensitivity: The model fails to distinguish semantically critical cues from irrelevant ones within the same image (e.g., confusing a toilet brush with a generic tool), leading to hallucinations even when the correct visual evidence is present.

Methodology: PEA-DPO

To address these limitations, the authors propose Perception-Enhanced Alignment DPO (PEA-DPO). This framework shifts the optimization paradigm from treating images as static context to jointly optimizing response quality and visual context preferences.

1. Construction of Perception-Enhanced Preference Data

The method generates high-quality visual preference pairs by explicitly removing critical visual context:

Perturbation: Starting with a chosen image ( $m_w$ ), the system applies random masks to generate a set of candidate perturbed images.
Selection via CLIP: Using CLIP embeddings, the system calculates semantic similarity between the original and perturbed images. The perturbed image with the lowest similarity score (indicating the removal of the most critical visual context) is selected as the rejected image ( $m_l$ ).
Data Structure: This creates a preference triplet $(x, m_w, m_l, y_w)$ , where the model must learn to prefer the response $y_w$ when conditioned on the complete image $m_w$ over the context-reduced image $m_l$ .

2. Joint Optimization Objective

PEA-DPO introduces a dual-objective loss function that simultaneously optimizes two complementary signals:

Response Quality Preference Optimization (RPO): The standard multimodal DPO term where the model learns to prefer a high-quality response ( $y_w$ ) over a lower-quality one ( $y_l$ ) given the same image ( $m_w$ ).
Visual Context Preference Optimization (VPO): A novel term where the model learns to prefer the response $y_w$ when conditioned on the complete image ( $m_w$ ) over the same response conditioned on the context-reduced image ( $m_l$ ).

The final objective combines these terms:
$\mathcal{L}_{PEA-DPO} = \mathcal{L}_{RPO} + \alpha \cdot \mathcal{L}_{VPO}$

To reduce computational overhead and improve controllability, the authors further refine this into mPEA-DPO by:

Replacing log-probability ratios with length-normalized log-probabilities (removing the need for a reference model).
Replacing the sigmoid function with ReLU to filter trivial data points.
Introducing target margins ( $\gamma_r, \gamma_m$ ) to enforce minimum separations between preferred and rejected pairs.

Theoretical Analysis

The paper provides a theoretical proof demonstrating that mPEA-DPO directly mitigates the identified failure modes:

Across-Image Insensitivity corresponds to a near-zero image-level margin ( $G_m$ ), representing the model's inability to distinguish between $m_w$ and $m_l$ . The $\mathcal{L}_{VPO}$ term explicitly enforces a positive margin here.
Within-Image Insensitivity corresponds to a near-zero response-level margin ( $G_r$ ), representing the model's inability to distinguish between $y_w$ and $y_l$ even with correct visual input. The $\mathcal{L}_{RPO}$ term enforces this margin.
The authors prove that the linear combination of these terms in mPEA-DPO provably enlarges both margins, thereby addressing both failure modes simultaneously.

Experimental Results

The authors evaluated mPEA-DPO on LLaVA-v1.5 models (7B and 13B parameters) across three hallucination benchmarks: MMHal-Bench, Object HalBench, and AMBER.

Hallucination Reduction: mPEA-DPO significantly reduced hallucination rates. For the 7B model, it reduced the MMHal-Bench hallucination rate by 33% compared to the base model. On Object HalBench, it achieved reductions of 92% (response-level) and 87% (object-level) in hallucination rates.
Comparison with SOTA: The method outperformed existing open-source RLHF/RLAIF-based baselines (e.g., LLaVA-RLHF, mDPO, OPA-DPO, DAMA) on the majority of hallucination metrics. While proprietary models like GPT-4V and Gemini-2.5-Pro achieved higher overall scores on MMHal-Bench and lower CHAIR scores on Object HalBench, mPEA-DPO demonstrated superior performance in reducing hallucination rates compared to other open-source methods. Notably, on the AMBER benchmark, mPEA-DPO (7B) achieved a hallucination rate of 10.3 and a cognition score of 0.6, significantly outperforming GPT-4V, which recorded a hallucination rate of 30.7 and a cognition score of 2.6.
Ablation Studies:
- Both RPO and VPO components were found to be individually effective, but their combination yielded the best performance.
- The proposed "Perception-Enhanced" strategy for constructing rejected images (removing key context via CLIP selection) outperformed alternative strategies like random block masking, rotation, cropping, or total blackness.
- Performance scaled positively with training data size, showing effectiveness even with smaller datasets (2.8k instances).
General Capabilities: Evaluations on general benchmarks (MMStar, AI2D, LLaVA-Bench, MMMU) indicated that mPEA-DPO preserved or slightly improved the model's instruction-following and general reasoning capabilities, suggesting the method does not degrade general performance.

Significance and Claims

The paper claims that PEA-DPO addresses a fundamental gap in current multimodal alignment: the tendency of models to ignore visual evidence when optimizing for text preferences. By explicitly modeling visual insensitivity and introducing a dual-preference learning framework, the authors demonstrate that:

Visual insensitivity is a measurable and correctable failure mode in multimodal DPO.
Explicitly leveraging visual preference signals (comparing complete vs. context-reduced images) is necessary to ground MLLM responses in visual evidence.
The proposed method achieves state-of-the-art performance among open-source methods in reducing hallucinations while maintaining the base model's general language capabilities, outperforming other open-source baselines and showing competitive performance against proprietary models on specific hallucination metrics.

The authors note limitations, including the additional computational cost of generating perception-enhanced data and a slight trade-off in coverage metrics (suggesting a more conservative generation strategy), but conclude that the framework offers a robust path toward more reliable and visually grounded MLLMs.