🤖 AI

Unbiased Alignment for Large Language Models with Noisy Preferences

This paper introduces a theoretical framework featuring Unbiased Reward Model (URM) and Unbiased Direct Preference Optimization (UDPO) losses to enable large language models to achieve robust, noise-tolerant alignment directly from noisy preference datasets without requiring clean ground-truth supervision.

Original authors: Jialiang Wang, Xianming Liu, Xiong Zhou, Hui Liu, Haoliang Li

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jialiang Wang, Xianming Liu, Xiong Zhou, Hui Liu, Haoliang Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but inexperienced student (a Large Language Model) how to write good stories or answer questions. You have a stack of feedback cards from a group of people, telling the student which answers are "good" and which are "bad."

The problem? The people giving the feedback aren't perfect. Some are tired, some are confused, and some might even have personal biases. In the real world, about 20% to 40% of these feedback cards might be wrong. If you just blindly follow the cards, the student learns the wrong lessons and starts making mistakes.

This paper introduces a new way to teach the student that ignores the "noise" (the mistakes) in the feedback cards and focuses on the truth underneath.

The Core Idea: The "Noise-Canceling" Headphones

Think of the noisy feedback like a song playing with a lot of static interference. Standard teaching methods try to learn the song by listening to the whole mess, which results in a distorted melody.

The authors of this paper built a mathematical "noise-canceling" system. They realized that if you know how the noise happens (e.g., "20% of the time, people flip their answers"), you can mathematically reverse the process. It's like having a recipe that says, "If the cake tastes too salty, subtract exactly this much salt to get the perfect flavor back."

They created two specific tools for this:

URM (Unbiased Reward Model): This is for the first stage of teaching. Usually, the model learns to give a "score" to answers. If the feedback is noisy, the scoring system gets confused. URM acts like a filter that cleans the scores before the model even sees them, ensuring the model learns the right "good vs. bad" distinction.
UDPO (Unbiased Direct Preference Optimization): This is for the second stage, where the model actually learns to write. Instead of just following the noisy feedback directly, UDPO uses a special math formula to "undo" the confusion. It allows the model to learn the correct behavior even if the teacher is occasionally wrong.

How It Works: The "Magic Formula"

The paper claims that you don't need to know exactly which specific feedback cards are wrong. You just need to know the general "noise rate" (how messy the feedback is).

They use a variable called $a$ (which is based on the noise rate).

The Downward Compatibility Trick: Imagine you are guessing how messy the feedback is. If you guess it's very messy (a high noise rate), but it turns out to be only somewhat messy, your method still works perfectly. It's like wearing heavy-duty rain boots on a sunny day; you stay dry even if it's just drizzling. However, if you guess it's sunny but it's actually pouring rain, you get wet. The authors proved their method is safe even if you overestimate the noise.

The Results: Winning the Game

The researchers tested this on real-world datasets (like chat conversations and summarization tasks) and artificially added more noise to see how it held up.

The Baseline: Standard methods (like DPO) started to fail badly when the noise got high. They got confused and performed poorly.
The New Method: Their URM and UDPO methods stayed strong. Even when 40% of the feedback was flipped (made wrong), their models still learned the right way to behave.
The Proof: They showed mathematically that their method doesn't just "get lucky." It is proven to recover the "best possible" teacher (the Bayes optimal classifier) even from a noisy classroom.

In a Nutshell

This paper says: "Don't throw away your noisy feedback data. Instead, use our new math tools to clean it up as you learn."

They provide a "noise-canceling" loss function (a mathematical rule for learning) that lets AI models learn from messy, imperfect human feedback without getting confused. It's a robust, safer way to align AI with human values, ensuring that even if the humans making the feedback are tired or biased, the AI still learns the right lessons.

Technical Summary: Unbiased Alignment for Large Language Models with Noisy Preferences

Problem Statement

The alignment of Large Language Models (LLMs) with human preferences is a critical stage in the training pipeline, typically achieved via Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO). However, real-world preference datasets are inherently noisy, with annotation errors, lack of professional knowledge, and social biases introducing significant noise (often ranging from 20% to 40%).

Standard alignment methods, including RLHF and DPO, are vulnerable to this noise. They tend to overfit to erroneous preference pairs, leading to distorted reward signals or biased policy models. Existing approaches to mitigate this issue suffer from limitations: some fail to achieve full noise tolerance, while others (e.g., rDPO) require precise estimation of the noise rate, which is unfeasible in practical scenarios where ground-truth noise levels are unknown.

Methodology

The authors propose a unified theoretical framework for Unbiased Alignment that mathematically corrects the distortion induced by preference noise without requiring clean ground-truth supervision. The core insight is that preference noise follows a statistically modeled transition process. By inverting this process, the authors derive loss functions that allow for the direct training of unbiased models from noisy data.

The framework addresses two dominant paradigms:

1. Unbiased Reward Model (URM) for RLHF

In the RLHF paradigm, noise corrupts the reward model training stage. The authors characterize the relationship between the optimal noisy reward model ( $r^\eta$ ) and the optimal unbiased reward model ( $r^*$ ).

Theoretical Derivation: They establish a transformation relationship where the unbiased reward signal can be recovered from the noisy counterpart using a constant $a = \frac{\eta}{1-\eta}$ , where $\eta$ is the noise rate.
URM Loss: By substituting the noisy reward representation with the unbiased one, they derive the Unbiased Reward Model (URM) loss:
$L_{URM} = -\log \frac{\exp(\Delta) + a}{\exp(\Delta) + 1}$
where $\Delta = r_\phi(x, y_w) - r_\phi(x, y_l)$ . This loss enables training an unbiased reward model directly from noisy preference pairs.

2. Unbiased Direct Preference Optimization (UDPO) for DPO

For the DPO paradigm, which optimizes the policy directly without an explicit reward model, noise similarly biases the learned policy.

Theoretical Derivation: The authors demonstrate that the relationship between noisy and unbiased policies mirrors that of the reward models. They derive the transformation between the noisy policy ratio and the unbiased policy ratio.
UDPO Loss: This leads to the Unbiased Direct Preference Optimization (UDPO) loss:
$L_{UDPO} = -\log \frac{\left(\frac{\pi_\theta(y_w|x)\pi_{ref}(y_l|x)}{\pi_\theta(y_l|x)\pi_{ref}(y_w|x)}\right)^\beta + a}{\left(\frac{\pi_\theta(y_w|x)\pi_{ref}(y_l|x)}{\pi_\theta(y_l|x)\pi_{ref}(y_w|x)}\right)^\beta + 1}$
This allows for the direct optimization of an unbiased policy model while maintaining the simplicity of the DPO framework.

Unified Framework and Theoretical Properties

The authors show that both URM and UDPO share a unified margin-based form, preserving the fundamental equivalence between RLHF and DPO. Key theoretical properties established include:

Noise Tolerance: The methods are robust to noisy data by design.
Parameter Downward Compatibility: If the true noise rate $\eta^*$ is lower than or equal to the estimated parameter $\hat{\eta}$ (used to set $a$ ), the loss function still yields an unbiased optimal solution. This is crucial as exact noise rates are unknown; overestimating the noise parameter ensures robustness.
Classification Calibration: The proposed loss functions are proven to be classification-calibrated, ensuring that minimizing the surrogate loss leads to the Bayes-optimal classifier.
Excess Risk Bound: The paper establishes a bound relating the excess risk of the unbiased loss to the 0-1 loss, demonstrating that the method can recover the Bayes-optimal classifier.

Key Contributions

Theoretical Framework: A unified theoretical framework for unbiased alignment that mathematically inverts preference noise distortion for both RLHF and DPO.
Novel Loss Functions: The derivation of the URM loss for reward modeling and the UDPO loss for direct policy optimization, both capable of training unbiased models from noisy datasets.
Theoretical Guarantees: Rigorous proofs demonstrating noise tolerance, parameter downward compatibility, classification calibration, and excess risk bounds.
Empirical Validation: Comprehensive experiments across diverse datasets (Anthropic-HH, Reddit TL;DR, UltraFeedback Binarized) and model families (Llama-3, Qwen) showing consistent superiority over state-of-the-art baselines.

Experimental Results

The authors evaluated their methods against baselines including standard RM/DPO, label smoothing variants, and other robust methods (rDPO, Dr.DPO) under varying levels of manually injected noise (0%, 20%, 40%) and in naturally noisy real-world datasets.

Reward Model Training: URM and its normalized variant ( $\alpha$ -URM) consistently outperformed all baselines. On the TL;DR dataset with 40% noise, URM improved win rates by approximately 8% over standard baselines.
Supervised Preference Training: UDPO and $\alpha$ -UDPO achieved the highest average win rates across all datasets. The average win rate of UDPO exceeded that of normal DPO by roughly 15%, confirming that standard DPO overfits to noise while the unbiased framework recovers the true policy.
Scalability: The methods demonstrated effectiveness on larger models (Llama-3.1-8B, Qwen-3-8B, Qwen-3-14B) and complex datasets (UFB), maintaining robustness in closed-loop RLHF settings using GRPO.
Ablation: Experiments confirmed that the normalized variants ( $\alpha$ -URM/ $\alpha$ -UDPO) are less sensitive to hyperparameter choices and consistently achieve optimal performance when $a$ is set to cover potential noise levels.

Significance and Claims

The paper claims to offer a "concise, interpretable, and effective route to robust LLM alignment." By treating preference noise as a mathematical transition process and analytically inverting it, the authors provide a solution that does not rely on the impractical requirement of knowing the exact noise rate.

The significance lies in:

Practical Utility: The "parameter downward compatibility" property allows practitioners to set a conservative noise parameter to ensure robustness without needing precise noise estimation.
Theoretical Rigor: The work bridges the gap between noisy learning theory and LLM alignment, providing guarantees (calibration, risk bounds) that were previously lacking in robust alignment methods.
Unified Approach: It unifies the treatment of noise in both the reward modeling (RLHF) and direct optimization (DPO) paradigms, demonstrating that the fundamental RLHF-DPO equivalence holds even under the proposed unbiased objectives.

The authors conclude that their methods enable the training of LLMs that are resilient to the significant noise prevalent in real-world preference data, thereby improving the safety and reliability of aligned AI systems.