🤖 AI

A Regret Minimization Framework on Preference Learning in Large Language Models

This paper introduces Regret-based Preference Optimization (RePO), a novel framework that reframes reinforcement learning from human feedback through regret minimization to better capture the prospective and counterfactual nature of human preferences, resulting in consistent performance gains on reasoning and preference benchmarks.

Original authors: Suhwan Kim, Taehyun Cho, Geon-Hyeong Kim, Yu Jin Kim, Youngsoo Jang, Moontae Lee, Jungwoo Lee

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Suhwan Kim, Taehyun Cho, Geon-Hyeong Kim, Yu Jin Kim, Youngsoo Jang, Moontae Lee, Jungwoo Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to solve a math problem or write a helpful email. You want the robot to learn from human feedback, but humans don't usually say, "This sentence is worth exactly 7.5 points." Instead, we say, "I prefer this answer over that one."

For a long time, computer scientists taught robots to learn by trying to maximize a score. They treated every step the robot took as if it were earning immediate points, like collecting coins in a video game. If the robot got a step "right," it got a coin. If it got it "wrong," it lost a coin. The robot's goal was simply to grab as many coins as possible.

The Problem with the "Coin Collector" Approach
The paper argues that this "coin collector" mindset is actually a bad way to understand how humans think. Humans don't just look at the immediate step; we look ahead and we look back at what could have happened.

The authors use a great analogy: The "What If" Game.

Imagine you are watching a chess player make a move.

The Old Way (Reward Maximization): You look at that single move. Did it capture a piece? If yes, good! If no, bad. You judge the move in isolation.
The New Way (Regret Minimization): You look at the move and ask, "If they had made a different move right here, would the game have gone better?" Or, "If they keep playing like this, will they win in 10 moves?"

Humans are terrible at judging a step in isolation. We judge it based on prospective anticipation (what we think will happen next) and counterfactual comparison (what else they could have done).

The Paper's Solution: "Regret-Based Preference Optimization" (RePO)
The authors introduce a new method called RePO. Instead of trying to maximize points, RePO teaches the robot to minimize regret.

Think of "Regret" like the feeling you get when you say, "I should have taken the other road; I would have avoided the traffic."

In the old method, the robot only cares if the road it took was smooth right now.
In the new method (RePO), the robot asks: "Compared to the best possible road I could have taken, how much did I mess up?"

How It Works in Plain English

Looking Forward: When a human looks at a partial answer (like the first half of a math solution), they mentally finish the problem. If they think, "Oh, this path leads to a wrong answer eventually," they dislike the first half, even if the first half looks okay. RePO mimics this by simulating the future to see if the current step leads to a dead end.
Comparing Alternatives: Humans often judge an action by comparing it to a "better" version that didn't happen. RePO calculates the "distance" between what the robot did and what the "perfect" robot would have done. It tries to shrink that gap.
The "Regret" Score: Instead of a positive score for being "good," RePO calculates a "regret" score. The goal is to make this regret as close to zero as possible. If the regret is high, it means the robot made a choice that led to a worse outcome than it could have.

Why This Matters (According to the Paper)
The researchers tested this on math problems and general conversation tasks. They found that robots trained with RePO were better at:

Solving Math: They understood that a step is only "good" if it leads to the correct final answer, not just if the step itself looks logical.
Following Human Taste: They aligned better with human preferences because they stopped trying to "game" the system for immediate points and started thinking about the whole journey, just like humans do.

The "Magic Trick" (Sample Efficiency)
One of the coolest findings is that RePO is "smarter" about how it learns.

Old Method (DPO): If you show the robot a math problem where the answer is hidden (masked), the robot gets confused and performs poorly. It needs to be shown the full, correct answer many times to learn the pattern.
New Method (RePO): Because RePO is built on the idea of "regret" (thinking about what could have happened), it naturally understands that a hidden answer is suspicious. It doesn't need as much extra training to figure out that incomplete paths are bad. It "internalized" the lesson that "if you can't see the finish line, you probably took a wrong turn."

In Summary
The paper says: Stop teaching robots to be greedy coin collectors who only care about the next step. Instead, teach them to be reflective travelers who ask, "Did I take the best path possible, and how much did I regret my choices compared to the road not taken?" By doing this, the robots become better at reasoning and more aligned with how humans actually think.

Technical Summary: A Regret Minimization Framework on Preference Learning in Large Language Models

Problem Statement

While Reinforcement Learning with Verifiable Rewards (RLVR) has driven significant progress in reasoning-intensive tasks like mathematics and code generation by leveraging automated correctness signals, many realistic language tasks (e.g., safety-critical decisions, customer service, educational reasoning) lack reliable verifiers. Consequently, the field relies heavily on Reinforcement Learning from Human Feedback (RLHF).

The core problem addressed is the interpretation of human feedback within RLHF. Existing preference-based methods (e.g., DPO, IPO) typically interpret human preferences through the lens of reward maximization, treating preferences as proxies for immediate or accumulated utility assigned to observed trajectory segments. The authors argue this interpretation is fundamentally misaligned with human cognition. Human judgment is inherently prospective (anticipating future outcomes) and counterfactual (comparing realized outcomes against plausible alternatives). Current reward-based frameworks evaluate intermediate steps myopically, ignoring that human preferences over partial trajectories are shaped by the anticipated continuation of those trajectories and comparisons to what could have been done, rather than absolute, outcome-independent utility.

Methodology: Regret-based Preference Optimization (RePO)

The paper introduces Regret-based Preference Optimization (RePO), a framework that reframes preference learning through regret minimization rather than reward maximization.

Theoretical Foundation

Regret as the Objective: Instead of maximizing immediate rewards, RePO models human feedback as a relative assessment of suboptimality. Regret is defined as the difference between the value of the optimal policy ( $V^{\pi^*}$ ) and the Q-value of the behavior policy ( $Q^\mu$ ) at a given context.
KL-Regularized RL Framework: The authors formalize this within a step-level KL-regularized RL framework. They derive a closed-form expression for the Q-function associated with a behavior policy $\mu$ .
Decomposition of Regret: A key theoretical contribution (Theorem 5.4 and Lemma 5.5) establishes that the regret incurred by a behavior policy $\mu$ relative to an optimal policy $\pi^*$ can be decomposed into two components:
- A local preference term: $\alpha \log \frac{\pi^*(o|q_{<t})}{\pi_{ref}(o|q_{<t})}$ , which aligns with the standard Direct Preference Optimization (DPO) objective.
- A long-horizon term: A sequential forward KL divergence ( $\bar{D}_{KL}$ ) measuring the cumulative deviation of the behavior policy $\mu$ from the optimal policy $\pi^*$ along future rollouts.
This decomposition reveals that minimizing regret explicitly accounts for behavior-dependent deviations and future trajectory consistency, addressing the "likelihood mismatch" issues present in off-policy settings where DPO assumes on-policy data.

Practical Implementation

RePO: The standard algorithm utilizes the derived closed-form update, incorporating the sequential KL divergence term. It requires knowledge of the behavior policy $\mu$ that generated the data.
RePO_det: To handle scenarios where the behavior policy is unknown or computationally prohibitive to access (common in offline datasets), the authors propose RePO_det. This variant approximates the behavior policy as a deterministic distribution (Dirac delta) concentrated at the observed output. It replaces the sequential KL term with a sample-based estimator that averages the log-probability ratios over the observed trajectory, effectively treating the observed segment as a single rollout.

Key Contributions

Reframing Human Feedback: The paper argues that human feedback is more naturally understood as relative judgments conditioned on anticipated outcomes and implicit alternatives (regret) rather than signals for reward maximization.
Closed-Form Policy Update: The authors formalize this perspective within a KL-regularized RL framework, deriving a closed-form policy update (RePO) that is compatible with direct preference optimization but explicitly models behavior-dependent deviation.
Inductive Bias Internalization: Theoretical analysis (Lemma 6.1) suggests that regret minimization structurally internalizes an inductive bias present in human feedback: incomplete or partially observed successful trajectories are evaluated pessimistically due to uncertainty about future completion. This allows RePO to learn effectively without requiring explicit data augmentation to enforce such preferences.
Robustness to Behavior Policy: The introduction of RePO_det demonstrates that the framework remains effective even when behavior policy information is unavailable, making it applicable to heterogeneous offline datasets.

Experimental Results

The authors evaluated RePO on mathematical reasoning benchmarks (GSM8K, MATH, MATH500, AMC23, Minerva) and human preference alignment tasks (AlpacaEval 2, Arena-Hard, MT-Bench) using Qwen3 (1.7B/4B) and LLaMA3 models.

Performance Gains: RePO consistently outperformed existing baselines (DPO, IPO, RPO, KTO, TDPO) across both model scales and task types. For instance, on the 4B model, RePO achieved a 55.08% win rate on AlpacaEval 2 (vs. 32.89% for DPO) and 65.54% on GSM8K (vs. 87.87% for DPO, though RePO showed strong gains in reasoning tasks generally).
Offline Applicability: RePO_det achieved performance comparable to RePO within the Qwen family and significantly outperformed baselines in cross-model settings (e.g., training LLaMA models on data generated by Qwen models), demonstrating the robustness of the sample-based regret estimator.
Sample Efficiency: In ablation studies involving masked data augmentation (where successful trajectories were partially hidden to test the model's ability to judge incomplete paths), DPO required additional augmented data to improve performance, whereas RePO's performance remained stable and dominant. This supports the claim that RePO structurally encodes the inductive bias of human judgment regarding incomplete trajectories, leading to higher sample efficiency.

Significance and Claims

The paper claims that RePO offers a faithful abstraction of human feedback by aligning the learning objective with the prospective and counterfactual nature of human judgment. By shifting from reward maximization to regret minimization, the framework provides a principled interpretation of preferences over heterogeneous trajectories.

The authors conclude that regret minimization serves as a promising foundation for human-aligned post-training of Large Language Models. The results suggest that correctly modeling the relationship between behavior policies, future rollouts, and counterfactual alternatives is critical for effective preference learning, particularly in settings where explicit reward signals are absent or where data is collected from diverse sources. The work does not claim to solve all alignment problems but positions regret minimization as a necessary refinement to the current reward-centric paradigm in RLHF.