🤖 AI

Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization

This paper introduces Reasoning-Conditioned Direct Preference Optimization (RC-DPO), a novel training framework that mitigates hallucinations in Multimodal Large Reasoning Models by explicitly aligning answer generation with logically consistent, visually grounded reasoning chains rather than treating them as a monolithic output.

Original authors: Jiawei Kong, Hao Fang, Shunxiang Liao, Jinyu Li, Bin Chen, Hao Wu, Shu-Tao Xia, Min Zhang

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiawei Kong, Hao Fang, Shunxiang Liao, Jinyu Li, Bin Chen, Hao Wu, Shu-Tao Xia, Min Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Confident Liar"

Imagine a very smart student (the AI) who is taking a test about a picture. This student is great at solving hard problems, but they have a bad habit: they often hallucinate.

In the past, if the student got the final answer wrong, teachers would just tell them, "No, that answer is incorrect." But this paper discovered a new problem with "Reasoning Models" (AI that thinks out loud before answering).

These models don't just get the final answer wrong; they often lie while they are thinking.

The Scenario: The student looks at a picture of a truck in a desert.
The Lie: In their "thinking" steps (Chain-of-Thought), they invent a story: "I see a blue dining table in the foreground." (There is no table).
The Result: Because they convinced themselves there was a table, they confidently answer, "Yes, there is a dining table."

The paper argues that current training methods are like teachers who only grade the final answer. They see "Yes" and "No" and try to fix that, but they ignore the fact that the student's thinking process was full of lies. The student learns to guess the right answer by luck or by memorizing shortcuts, without actually learning to look at the picture correctly.

The Solution: RC-DPO (The "Thinking Coach")

The authors propose a new training method called RC-DPO (Reasoning-Conditioned Direct Preference Optimization).

Think of this as a new way for a coach to train an athlete. Instead of just saying, "You ran the race too slow, try again," the coach breaks it down:

The Condition: The coach says, "If you run with good form (a truthful thought process), you should get a gold medal. If you run with bad form (a lying thought process), you should get a penalty, even if you somehow crossed the finish line first."
The Goal: The AI learns that a "good answer" is only valid if it is supported by a "good thought process." It forces the AI to align its thinking with the visual evidence.

How They Built the Training Data (The "Search and Prune" Strategy)

To teach the AI this new lesson, the researchers needed examples of "Good Thinking" vs. "Bad Thinking." They didn't just ask humans to write these; they built a system to generate them automatically:

Finding the "Good Thinking" (MCTS):
Imagine a detective trying to solve a mystery. Instead of guessing one path, the detective tries many different paths (using a method called Monte Carlo Tree Search). They check each path to see: "Does this step match the photo? Is the logic sound?"
- They pick the path that is the most logical and visually accurate. This becomes the Positive Sample (the "Good Thinking" example).
Creating the "Bad Thinking" (Attention Pruning):
To teach the AI what not to do, they took a normal answer and deliberately broke it. They looked at which words in the "thinking" part were most connected to the image (like "red," "truck," "dust").
- They pruned (cut out) those important visual words.
- This created a "Bad Thinking" sample that sounds like it's thinking, but it's actually ignoring the picture. This becomes the Negative Sample.

The Result: A More Honest Thinker

By training the AI to prefer the "Good Thinking" path over the "Bad Thinking" path (even when the final answer is the same), the model learned to stop lying during its thought process.

Before: The AI would hallucinate a table, think about it, and answer "Yes."
After: The AI looks at the picture, sees no table, thinks "I see a truck and a shed, but no table," and answers "No."

Summary

The paper claims that to stop AI from hallucinating (making things up), we can't just fix the final answer. We have to fix the thinking process itself. Their new method, RC-DPO, acts like a strict coach that says, "You can't get the right answer if your reasoning is a lie." By training the model to link good answers strictly to good, evidence-based reasoning, they significantly reduced the number of hallucinations in these complex vision-language models.

Technical Summary: Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization

Problem Statement

Multimodal Large Reasoning Models (MLRMs) have demonstrated strong capabilities on complex vision-language tasks by explicitly generating intermediate reasoning steps (Chain-of-Thought, or CoT) before producing final answers. However, these models remain prone to hallucinations, generating content that appears plausible but is inconsistent with visual inputs. Unlike conventional Multimodal Large Language Models (MLLMs), where hallucinations typically occur in the final output, MLRM hallucinations can arise within the intermediate CoT and subsequently propagate to the final answer.

Existing training-based mitigation methods, such as Direct Preference Optimization (DPO), typically treat the CoT and the final answer as a monolithic output. The paper observes that this response-level optimization behaves similarly to answer-only optimization, primarily learning answer-level preferences while leaving CoT-level supervision insufficiently exploited. Consequently, residual hallucinations in the reasoning chain often persist, dominating the overall error rate of the full response.

Methodology: RC-DPO

To address the insufficiency of coarse-grained supervision, the authors propose Reasoning-Conditioned Direct Preference Optimization (RC-DPO). This framework explicitly models the relationship between reasoning chains and final answers by treating the CoT as a condition for answer generation.

1. Core Objective

Instead of jointly optimizing the entire response $(c, a)$ , RC-DPO contrasts the likelihood of generating the same preferred answer ( $a_w$ ) under different CoT conditions:

Preferred Condition ( $c_w$ ): A visually grounded and logically valid reasoning chain.
Rejected Condition ( $c_l$ ): A hallucinated or corrupted reasoning chain.

The objective function isolates the effect of the reasoning chain on answer generation. It maximizes the margin between the conditional probability of the preferred answer given a high-quality CoT versus a corrupted CoT:
$L_{RC} = -\log \sigma \left( \beta \left[ \log \frac{\pi_\theta(a_w|x, c_w)}{\pi_{ref}(a_w|x, c_w)} - \log \frac{\pi_\theta(a_w|x, c_l)}{\pi_{ref}(a_w|x, c_l)} \right] \right)$
The final training objective combines this reasoning-conditioned loss with standard response-level DPO:
$L = L_{DPO} + \lambda L_{RC}$

2. Preference Data Construction

RC-DPO requires constructing preference pairs $(x, c_w, a_w, c_l, a_l)$ without relying on external ground-truth annotations or stronger models. The authors introduce a pipeline using the model's own outputs:

Positive Sample Construction (MCTS): The authors employ Monte Carlo Tree Search (MCTS) to discover high-quality reasoning trajectories. Starting from a supervised fine-tuned (SFT) model, MCTS expands candidate reasoning steps and uses a verifier to score trajectories based on visual consistency, logical validity, and answer correctness. The highest-scoring trajectory is selected as the positive sample $(c_w, a_w)$ .
Negative Sample Construction (Attention-Guided Pruning): To create challenging negative samples, the authors employ an image-attention-guided token pruning strategy. They compute the attention scores from each CoT token to image tokens. Tokens with high visual relevance are identified and pruned (removed) from the original CoT. This disrupts the visual grounding of the reasoning chain, creating a corrupted CoT ( $c_l$ ) that serves as a negative condition for the same answer.

3. Training Pipeline

CoT Format Alignment: A warm-up SFT stage aligns the model with a structured CoT format (step-by-step reasoning followed by an answer) to facilitate MCTS operations.
Data Generation: MCTS and attention-guided pruning generate the preference dataset.
Optimization: The model is trained using the combined $L_{DPO} + \lambda L_{RC}$ objective.

Key Contributions

Analysis of Existing DPO: The paper reveals that standard response-level DPO is biased toward answer-level optimization, leaving CoT-level hallucinations insufficiently addressed.
RC-DPO Framework: The proposal of a fine-grained preference optimization framework that explicitly aligns reasoning chains by contrasting the same preferred answer under different CoT conditions.
Data Generation Strategy: The introduction of a self-contained preference data generation pipeline utilizing MCTS for positive sampling and attention-guided token pruning for negative construction.
Empirical Validation: Extensive experiments across four MLRMs and nine benchmarks demonstrating the effectiveness of the approach.

Experimental Results

The authors evaluated RC-DPO on four representative MLRMs: R1-Onevision-7B, MM-Eureka-7B, ThinkLite-VL-7B, and OpenVLThinker-7B.

Hallucination Mitigation: On the Object HalBench, RC-DPO consistently achieved the lowest CHAIRS and CHAIRI scores across all models, significantly outperforming baselines like VCD, SID, RLAIF-V, and OPA-DPO. For instance, on R1-Onevision, CHAIRS was reduced from 36.7 (Vanilla) to 21.7 (RC-DPO).
Object Existence: On the POPE benchmark, RC-DPO achieved the best accuracy and F1 scores across random, popular, and adversarial settings, indicating improved object existence recognition.
Open-Ended Generation: On MMHal-Bench and GPT-4 assisted evaluations, RC-DPO reduced hallucination rates while maintaining or improving descriptive quality and fluency.
General Capabilities: Evaluations on MME, MMBench, VMCBench, and MMVP showed that RC-DPO does not sacrifice general multimodal perception or reasoning abilities; in many cases, it improved performance on fine-grained visual tasks (e.g., OCR and position recognition).

Significance and Claims

The paper claims that explicit alignment of intermediate reasoning chains is crucial for building faithful and reliable MLRMs. By shifting from coarse-grained response-level optimization to fine-grained reasoning-conditioned optimization, RC-DPO addresses the root cause of hallucination propagation in reasoning models. The authors emphasize that their method effectively mitigates hallucinations in both the intermediate CoT and the final answer, leading to more trustworthy multimodal reasoning without compromising general capabilities. The work highlights that treating the CoT as a condition for answer generation provides a direct training signal for learning answer-supportive reasoning preferences.