🤖 machine learning

The Differences Between Direct Alignment Algorithms are a Blur

This paper demonstrates that the ranking objective (pairwise vs. pointwise) is the primary determinant of alignment quality in Direct Alignment Algorithms, outweighing the specific scalar score optimized or the training stage configuration, when evaluated under a unified framework that standardizes SFT phases and hyperparameters.

Original authors: Alexey Gorbatovski, Boris Shaposhnikov, Viacheslav Sinii, Alexey Malakhov, Daniil Gavrilov

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexey Gorbatovski, Boris Shaposhnikov, Viacheslav Sinii, Alexey Malakhov, Daniil Gavrilov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart but slightly rebellious robot (a Large Language Model) how to be helpful and follow instructions. For a long time, the standard way to do this was like a three-step dance: first, teach it basics; second, hire a human judge to score its answers; and third, use a complex reward system to train it to get higher scores.

Recently, researchers invented "Direct Alignment Algorithms" (DAAs). These are shortcuts that skip the human judge and the complex reward system, trying to teach the robot directly using a single math formula. The problem? Everyone started inventing their own unique formula, claiming theirs was the "best," but nobody could agree on why one worked better than another. It was like everyone claiming their specific brand of paint was the best, without realizing they were all painting on different walls.

This paper is like a detective story that finally clears up the confusion. Here is what the authors found, explained simply:

1. The "Unified Workshop" Experiment

The authors realized that all these different algorithms were being tested in different "workshops" (training setups). Some started with a robot that had already learned the basics (SFT), while others tried to teach the basics and the alignment at the same time.

To make a fair comparison, they built a Unified Workshop. They forced every algorithm to:

First, learn the basics in a separate class (SFT).
Then, take the alignment class.
Use a common "temperature knob" (called $\beta$ ) to control how strict the training is.

The Result: Once they put everyone in the same workshop, the differences between the specific math formulas (the "scalars") mostly disappeared. It turned out that the specific math trick an algorithm used didn't matter as much as how it compared answers.

2. The Real Hero: "Pairwise" vs. "Pointwise"

The paper discovered that the most important factor isn't the specific math formula, but the strategy used to compare answers. They found two main strategies:

The "Pointwise" Strategy (The Solo Act): Imagine a teacher grading a student's essay by looking at it alone. "Is this essay good? Yes/No." Then they look at a bad essay. "Is this bad? Yes/No." They grade them independently.
The "Pairwise" Strategy (The Debate): Imagine a teacher looking at a good essay and a bad essay side-by-side. "Okay, given these two, which one is better?" They focus entirely on the difference between the two.

The Finding: The Pairwise strategy (the debate) consistently won. It was like a coach who focuses on comparing players against each other rather than judging them in isolation. The "Pointwise" strategy often got confused by the specific quirks of the questions (prompts), wasting energy trying to fix things that didn't need fixing.

3. Why Does This Happen? The "Bias" Analogy

The authors explain this with a clever analogy about bias and mental energy.

Imagine the robot has a limited amount of "mental energy" (capacity) to learn.

The Pointwise Problem: When the robot looks at a single answer in isolation, it tries to fix everything. If a question has a weird bias (like a trick question), the robot spends all its energy trying to "unlearn" that trick. It gets so busy fixing the trick that it forgets to learn how to actually solve the hard problems.
The Pairwise Advantage: When the robot compares two answers side-by-side, it ignores the weird tricks of the question. It only cares: "Is Answer A better than Answer B?" It doesn't waste energy trying to fix the question's bias; it just focuses on the ranking.

The Sweet Spot:

Easy Tasks: If the task is super easy, the robot has plenty of energy. Both strategies work fine.
Hard Tasks: If the task is super hard, the robot is so overwhelmed that neither strategy works well.
Medium Difficulty: This is where the magic happens. The robot has enough energy to learn, but not enough to fix every single bias. The Pairwise strategy wins here because it saves its energy for the hard parts, while the Pointwise strategy wastes its energy on the biases.

4. The "Data Efficiency" Surprise

Another cool finding is that you don't need a massive library of data to teach these robots. The authors found that using just 5% to 10% of the usual training data was enough for the robots to reach 95% of their potential performance. It's like realizing you don't need to read the entire encyclopedia to learn how to write a good story; a few key chapters are enough.

Summary

The paper concludes that the "best" algorithm isn't a specific brand of math formula. Instead, the winner is the Pairwise approach (comparing answers side-by-side) because it is smarter about how it uses the robot's limited brainpower. It avoids getting distracted by the quirks of individual questions, allowing it to focus on what actually matters: knowing which answer is better than the other.

The authors warn that previous claims of one algorithm being "superior" were often just because they were tested in different conditions or on tasks where the difference didn't matter. Once you level the playing field, the Pairwise strategy is the clear champion.

Technical Summary: The Differences Between Direct Alignment Algorithms are a Blur

Problem Statement

Direct Alignment Algorithms (DAAs) have emerged as a streamlined alternative to Reinforcement Learning from Human Feedback (RLHF), bypassing explicit reward modeling and reinforcement learning stages to directly optimize language model policies. Despite a proliferation of methods—ranging from the two-stage paradigm of Direct Preference Optimization (DPO) to the one-stage, odds-ratio-based approaches like Odds Ratio Preference Optimization (ORPO) and Aligned Supervised Fine-Tuning (ASFT)—the literature remains fragmented.

The core problem addressed is the difficulty in isolating which specific design choices actually drive improvements in alignment quality. Existing comparisons are often confounded by differing training protocols (e.g., one-stage vs. two-stage pipelines), the presence or absence of a reference policy, and varying hyperparameter spaces (specifically the $\beta$ parameter). Consequently, claims of algorithmic superiority are often based on marginal differences that may stem from setup artifacts rather than fundamental algorithmic advantages.

Methodology

To disentangle these factors, the authors propose a unified training framework that places all DAAs on equal footing. This framework involves two critical modifications to existing methods:

Standardization to a Two-Stage Pipeline: The authors convert originally one-stage methods (ORPO, ASFT) into a two-stage pipeline. This involves an explicit Supervised Fine-Tuning (SFT) phase to obtain a policy ( $\pi_{SFT}$ ), followed by an offline preference optimization phase. This isolates the alignment loss from the SFT loss, allowing for a controlled comparison of the alignment objectives themselves.
Introduction of the $\beta$ Parameter: The authors introduce a temperature/scaling parameter $\beta$ to one-stage odds-ratio methods (ORPO, ASFT), which traditionally lack this hyperparameter. This places all methods (including DPO, IPO, SimPO, etc.) into a common hyperparameter space, enabling a systematic sensitivity analysis.

Within this unified setting, the authors analyze DAAs along two primary axes:

Scalar Score Family: The choice of the underlying score function, specifically the policy–reference ratio ( $r^{ref}_\theta$ ) used in DPO versus the odds ratio ( $r^{odds}_\theta$ ) used in ORPO/ASFT.
Ranking Objective: The structural formulation of the loss, distinguishing between pairwise objectives (which optimize the relative gap between chosen and rejected responses, e.g., DPO, IPO, SimPO, ORPO) and pointwise objectives (which optimize the absolute likelihood of chosen and rejected responses independently, e.g., ASFT, NCA, Cal-DPO, APO-Zero).

The evaluation spans instruction-following tasks (AlpacaEval 2, ArenaHard) and math-reasoning benchmarks (GSM8K, MATH500, etc.) across multiple model scales (Llama 3.2 3B, Llama 3.1 8B, Mistral 7B, Qwen 2.5 7B/14B).

Key Contributions

Unified Protocol for DAAs: The paper establishes that converting one-stage methods into a two-stage pipeline with an explicit SFT phase and introducing the $\beta$ parameter is essential for maximizing performance, even for odds-ratio objectives.
Ranking Objective as the Primary Driver: Under the unified protocol, the authors find that the previously reported advantages of specific DAAs largely disappear after hyperparameter tuning. The primary determinant of alignment quality is the ranking type (pairwise vs. pointwise), not the scalar score choice (policy–reference vs. odds ratio). Pairwise methods consistently outperform pointwise methods, particularly at intermediate task difficulties and model scales.
Mechanism of Performance Gaps: The paper provides evidence that performance gaps arise from the interaction between the objective function and prompt-specific biases.
- Pairwise objectives are structurally invariant to uniform shifts in scores for a given prompt; they focus solely on relative ranking.
- Pointwise objectives generate gradient signals that incentivize the model to adjust the absolute score levels (the marginalized score) for each prompt. In capacity-limited regimes, this "unlearning" of prompt-specific bias competes with the primary ranking task, leading to suboptimal performance on harder tasks.
Data Efficiency: The study demonstrates that most DAAs are highly data-efficient, requiring only 5–10% of the SFT dataset to reach $\ge$ 95% of their full-data performance.

Results

Performance Convergence: When compared under the unified protocol, most DAAs perform similarly or worse than standard DPO. The distinct advantages claimed in prior literature for specific algorithms often vanish.
Pairwise Superiority: Across instruction-following and math-reasoning benchmarks, pairwise methods (DPO, IPO, SimPO, ORPO) consistently achieve higher peak scores than pointwise methods (ASFT, NCA, Cal-DPO, APO-Zero). This trend is most pronounced in the 8B model scale on the UltraFeedback dataset and in math reasoning tasks with Qwen 2.5.
Scalar Score Irrelevance: The choice between policy–reference ratios and odds ratios yields comparable results when the ranking objective and training protocol are controlled.
Sensitivity to Difficulty: The performance gap between pairwise and pointwise methods is negligible for very easy tasks (where capacity is sufficient to handle both ranking and bias removal) and very hard tasks (where capacity is insufficient for either). The gap emerges primarily at intermediate difficulty, where pointwise methods struggle to balance bias removal with ranking refinement.
$\beta$ Impact: Introducing the $\beta$ parameter to ORPO and ASFT significantly improves their alignment quality, confirming its role as an effective tempering mechanism.

Significance and Claims

The paper argues that the field of DAA research has been hindered by fragmented evaluation protocols that obscure the true drivers of performance. By unifying the training setup, the authors demonstrate that the choice of ranking objective (pairwise vs. pointwise) is the dominant structural factor influencing alignment quality, while specific score parameterizations play a secondary role.

The authors claim that previous assertions of "best" DAA often rely on underexplored details of setup and data bias rather than fundamental algorithmic superiority. Their findings underscore the need for nuanced, controlled evaluations to avoid oversimplified claims. Furthermore, the identification of the interaction between objectives and prompt-specific biases offers a mechanistic explanation for observed performance differences, suggesting that pairwise objectives are more robust in scenarios where model capacity is constrained relative to task difficulty.

The paper concludes that while odds-ratio DAAs (like ORPO) can be effective, they benefit significantly from the same structural adjustments (SFT separation and $\beta$ tuning) as other methods, and their performance is ultimately governed by the same pairwise vs. pointwise dynamics.