🤖 AI

Autoregressive Direct Preference Optimization

This paper proposes Autoregressive Direct Preference Optimization (ADPO), a novel variant that reformulates the DPO objective by explicitly applying autoregressive assumptions prior to the Bradley-Terry model, resulting in a shifted loss function and the identification of distinct token and feedback length measures for improved preference optimization.

Original authors: Masanari Oi, Mahiro Ukai, Masahiro Kaneko, Naoaki Okazaki, Nakamasa Inoue

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Masanari Oi, Mahiro Ukai, Masahiro Kaneko, Naoaki Okazaki, Nakamasa Inoue

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Robot to Write Better

Imagine you are teaching a robot to write stories. You have a "Gold Standard" book (the reference model) and a "Student" robot (the learnable model). You want the Student to write stories that humans prefer over the Gold Standard's stories.

For a long time, the best way to do this was DPO (Direct Preference Optimization). Think of DPO as a teacher who reads the entire story the student wrote, compares it to the Gold Standard's story, and gives a single grade: "Good" or "Bad."

The Problem:
The paper argues that this "all-or-nothing" grading system is a bit inefficient. Large Language Models (LLMs) don't write stories in one giant leap; they write them word by word (or token by token), like a person typing a sentence. However, the old DPO method waits until the very end to give feedback. It's like a coach waiting until the end of a marathon to tell a runner, "You ran well," without ever correcting their form while they were running.

The Solution: ADPO (Autoregressive DPO)

The authors propose a new method called ADPO. Instead of waiting until the story is finished to give a grade, ADPO gives feedback step-by-step as the story is being written.

The "Movie vs. Snapshot" Analogy

Old DPO (The Snapshot): Imagine taking a photo of a finished painting. You look at the whole picture and say, "This is better than that one." You don't know which brushstrokes made it better.
New ADPO (The Movie): Imagine watching the artist paint the picture in real-time. ADPO looks at the first brushstroke, then the second, then the third. It asks, "Was this specific stroke good? Was the next one better?" It learns to prefer the process of writing, not just the final result.

The Two "Length" Rules

One of the paper's most interesting discoveries is that there are actually two different ways to measure length when teaching these models, and the old method confused them.

Token Length (The Word Count): This is how many words the model actually types. (e.g., "The cat sat" = 3 words).
Feedback Length (The Grading Units): This is how many chunks the teacher decides to grade at once.

The Old Way: The teacher always graded the entire story as one single chunk (Feedback Length = 1), regardless of how many words were in it.
The New Way (ADPO): The teacher can choose to grade the story in chunks. They could grade every single word (Feedback Length = Word Count), or they could grade every 10 words, or every paragraph.

The paper shows that by breaking the story into smaller chunks (making the "Feedback Length" match the "Token Length"), the model learns much faster and writes better.

How It Works (The "Math" Made Simple)

In the old method, the math looked like this:

Take the whole story, calculate the difference, then apply a "sigmoid" (a smoothing function) to get a score.

In the new ADPO method, the math flips the order:

Calculate the difference for each step, apply the "sigmoid" to each step, and then add them all up.

The Analogy:

Old Way: You mix all the ingredients for a cake, bake it, taste the whole thing, and then decide if you need more sugar. (Too late to fix the batter).
New Way: You taste the batter after adding the flour, then after the eggs, then after the sugar. You adjust the recipe as you go.

The Results: Does It Work?

The authors tested this new method on two types of tasks:

Math Problems: They asked the models to solve complex math word problems.
Conversations: They asked the models to chat with humans.

The Findings:

Better Grades: In almost every test, the models trained with ADPO got higher scores than those trained with the old DPO method.
Finer is Better: The models performed best when the "Feedback Length" was very small (checking every single word). This proved that the models benefit from constant, granular feedback.
No Extra Cost: Even though checking every word sounds like it would take more time, the authors found the extra time required was negligible (less than 6% slower).

Summary

The paper introduces ADPO, a smarter way to train AI models. Instead of waiting until the AI finishes a task to give it a grade, ADPO gives feedback at every single step of the process. By treating the AI's writing as a sequence of small steps rather than one big block, the models learn to make better choices throughout the entire generation process, resulting in smarter math reasoning and better conversations.

Technical Summary: Autoregressive Direct Preference Optimization (ADPO)

Problem Statement

Direct Preference Optimization (DPO) has become a standard method for aligning Large Language Models (LLMs) with human preferences by analytically deriving the optimal policy from the Bradley-Terry (BT) model, eliminating the need for explicit reward modeling. However, the authors identify a theoretical limitation in the standard DPO formulation: while LLMs are inherently autoregressive (generating tokens sequentially), the standard DPO objective applies the BT model at the level of complete responses after assuming the autoregressive nature of the policy.

Specifically, standard DPO derives the objective function using the BT model over the output space $\mathcal{Y}$ , and only subsequently expands the log-probabilities into a sum of token-level terms. This results in a loss function where the summation of token-level log-ratios occurs inside the log-sigmoid function. The authors argue that this ordering fails to fully leverage the autoregressive assumption during the derivation of the preference model, potentially limiting the granularity and theoretical consistency of the optimization.

Methodology: Autoregressive DPO (ADPO)

The paper proposes Autoregressive DPO (ADPO), a novel variant that explicitly incorporates the autoregressive assumption prior to applying the Bradley-Terry model.

Theoretical Formulation

Prefix Closure and Energy Functions: ADPO extends the domain of the reward function from the complete output space $\mathcal{Y}$ to its prefix closure $\mathcal{Y}^*$ (the set of all incomplete responses). It defines "prefix-wise" energy functions $E^*_1$ and $E^*_2$ over these prefixes.
Prefix-wise Bradley-Terry Model: Instead of modeling preferences over full sequences, ADPO defines a prefix-wise BT model. The probability of a preferred sequence $y_w$ over a dispreferred sequence $y_l$ is formulated as the product of preference probabilities at each prefix step $i$ .
Derivation of the Loss: By applying the autoregressive assumption to the reference model $\pi_{ref}$ $π_{r e f}$ and the learnable model $\pi_\theta$ $π_{θ}$ within the prefix-wise BT framework, the authors derive a new loss function.
- Key Structural Difference: In standard DPO, the loss is $-\log \sigma(\sum_i \Delta_i)$ . In ADPO, the summation is shifted outside the log-sigmoid function:
  $\mathcal{L}_{ADPO} = -\mathbb{E} \left[ \sum_{i=1}^{T'} \log \sigma \left( \beta \log \frac{\pi_\theta(y_{w,i}|y_{w,<i}, x)}{\pi_{ref}(y_{w,i}|y_{w,<i}, x)} - \beta \log \frac{\pi_\theta(y_{l,i}|y_{l,<i}, x)}{\pi_{ref}(y_{l,i}|y_{l,<i}, x)} \right) \right]$
- This formulation treats the preference optimization as a sequence of token-level (or sub-sequence level) decisions rather than a single sequence-level decision.

Theoretical Analysis: Length Measures

The paper introduces a theoretical distinction between two length measures, which clarifies the relationship between DPO and ADPO:

Token Length ( $\mu$ ): The number of tokens in a sequence, inherent to the LLM's autoregressive nature.
Feedback Length ( $\mu'$ ): The granularity at which human feedback is applied.
- Standard DPO corresponds to the special case where $\mu'(y) = 1$ for all sequences (feedback is given only on the complete response).
- ADPO allows $\mu'$ to be independent of $\mu$ , enabling training at arbitrary granularities (e.g., token-level, sentence-level, or fixed-window levels).

The authors prove that any reward function consistent with the prefix-wise BT model can be reparameterized by an autoregressive model, ensuring ADPO remains theoretically consistent with the KL-constrained reward maximization problem.

Key Contributions

ADPO Formulation: The introduction of ADPO, which derives the preference optimization objective by extending the energy domain to the prefix closure $\mathcal{Y}^*$ , thereby explicitly integrating the autoregressive assumption into the BT model.
Loss Function Innovation: Deriving a loss function where the summation of token-level log-ratios is shifted outside the log-sigmoid function, providing a theoretically consistent way to facilitate finer-granularity training.
Theoretical Insight on Granularity: Identifying and distinguishing the token length measure ( $\mu$ ) and the feedback length measure ( $\mu'$ ). The paper demonstrates that DPO is a special case of ADPO where $\mu' \equiv 1$ , and that varying $\mu'$ allows for a family of algorithms ranging from full-response to token-level optimization.
Reparameterization Completeness: Proving that under mild assumptions, any reward function can be reparameterized by an autoregressive model within the prefix-wise framework.

Experimental Results

The authors evaluated ADPO on mathematical reasoning and conversational tasks across multiple LLM architectures (Llama-3, Qwen, Gemma, DeepSeek-Math).

Mathematical Reasoning (GSM8K, MATH500):
- ADPO variants consistently outperformed standard DPO and the token-level contrastive method cDPO across all models.
- The best results were achieved by cADPO (ADPO combined with critical token identification), with Qwen-3-8B reaching 91.74% accuracy on GSM8K.
- Granularity Analysis: Experiments with "Static" (fixed window size $k$ ) and "Adaptive" (fixed number of segments $m$ ) families showed that finer granularity generally improved performance. Specifically, token-level ( $k=1$ ) and high-segment adaptive settings ( $m \ge 128$ ) yielded superior results compared to full-response DPO.
Conversational Tasks (AlpacaEval 2, Arena-Hard, MT-Bench):
- ADPO outperformed DPO and performed comparably to or better than SimPO on all benchmarks.
- The adaptive family with $m=256$ achieved the highest Length-Controlled Win Rate (45.8%) on AlpacaEval 2.
Training Dynamics: Analysis of log-probabilities showed that ADPO more effectively amplifies the distinction between preferred and dispreferred sequences, particularly as granularity increases.
Efficiency: The additional prefix-wise computations introduced by ADPO resulted in a negligible training overhead (approximately 5.7% increase in wall-clock time).

Significance and Claims

The paper claims that ADPO provides a theoretically consistent extension of DPO that better aligns with the autoregressive nature of LLMs. By shifting the summation outside the log-sigmoid function, ADPO enables optimization at arbitrary granularities without violating the foundational principles of preference optimization.

The authors emphasize that their work is the first to explicitly distinguish between token length and feedback length measures, offering a unified framework that encompasses both full-response (DPO) and token-level approaches. The empirical results suggest that leveraging this finer granularity leads to more robust alignment, particularly in complex reasoning tasks, without significant computational cost. The paper concludes by noting that while ADPO relies on the original KL-constrained framework, future work could explore generalizing the divergence metrics or adapting the energy definitions for non-autoregressive models.