💬 NLP

Offline Preference Optimization via Maximum Marginal Likelihood Estimation

This paper introduces MMPO, a stable and simplified offline preference optimization method that aligns Large Language Models with human preferences by maximizing marginal log-likelihood, thereby eliminating the need for explicit reward models while outperforming baselines in alignment quality and general capability preservation.

Original authors: Saeed Najafi, Alona Fyshe

Published 2026-01-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Saeed Najafi, Alona Fyshe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Robot to Be "Polite"

Imagine you have a very smart robot (a Large Language Model) that has read almost everything on the internet. It can write stories, solve math problems, and answer questions. However, it's a bit like a brilliant but chaotic teenager: it knows how to speak, but it doesn't always know what humans actually want to hear. Sometimes it's rude, sometimes it's confusing, and sometimes it just makes things up.

To fix this, we need to "align" the robot with human preferences. The current standard way to do this is called RLHF (Reinforcement Learning from Human Feedback). Think of this like hiring a strict coach who watches the robot, gives it points for good answers, and takes points away for bad ones. But this process is messy, expensive, and the robot often gets confused by the coach's inconsistent rules.

Recently, a simpler method called DPO was invented. It's like skipping the coach and just showing the robot two answers (one good, one bad) and saying, "Pick the good one." This works better, but it still has some quirks and can be unstable if you tweak the settings too much.

This paper introduces a new method called MMPO. The authors say, "Let's try a completely different angle." Instead of trying to teach the robot to pick the winner, they treat the problem like a marginal likelihood estimation.

The Core Idea: The "Best Guess" Game

To understand MMPO, imagine you are trying to guess the ending of a mystery story. You know the beginning (the prompt), and you have two possible endings:

The Good Ending: The one humans prefer.
The Bad Ending: The one humans dislike.

The Old Way (DPO/RLHF):
The old methods are like a game of "Hot and Cold." They calculate a score for the Good Ending, a score for the Bad Ending, and then try to push the Good Ending up and the Bad Ending down. It's a bit like trying to balance a seesaw; if you push too hard on one side, the whole thing tips over (instability).

The New Way (MMPO):
The authors propose a different approach based on Maximum Marginal Likelihood (MML).

Imagine you are a detective trying to figure out how likely it is that a specific "Good Ending" was the true ending of the story, given that you only have a few clues (the two endings you are looking at).

Instead of comparing the two endings against each other directly, MMPO asks: "If we look at all the possible ways this story could end, how likely is it that the 'Good Ending' is the one that actually happened?"

To do this, the math looks at both the Good Ending and the Bad Ending as "samples" (clues) to estimate that likelihood.

It doesn't need a separate "Coach" (Reward Model) to tell it what is good.
It doesn't need to force the robot to be "creative" (Entropy Maximization) just for the sake of it.

The Magic Trick: How It Works Without a Coach

The paper claims that by using this "Best Guess" math, the robot implicitly learns to prefer the good answer without being explicitly told to do so.

Here is the analogy:
Imagine you are teaching a dog to sit.

RLHF: You have a trainer with a clicker and treats. You click when the dog sits, and you don't when it doesn't.
DPO: You show the dog a picture of it sitting and a picture of it standing, and you say, "I like the sitting picture more."
MMPO: You simply look at the "sitting" picture and calculate the probability that this is the correct behavior based on the evidence you have.

The paper proves mathematically that when you do this calculation, the math naturally pushes the robot to choose the "Good Ending" more often. It's like the math itself acts as the reward system. The robot learns to boost the probability of the good answer and slightly lower the bad one, all in one smooth motion.

Why Is This Better? (The Results)

The authors tested this on robots of different sizes (from small "toy" robots to large "brainy" ones). Here is what they found:

It's More Stable:
Think of tuning a radio. With the old methods (DPO), if you turn the dial (a setting called $\beta$ ) just a tiny bit, the music might turn into static or cut out completely. With MMPO, the radio stays clear no matter how you turn the dial. It's much harder to "break" the training.
It Keeps the Robot Smart:
Sometimes, when you teach a robot to be polite, it forgets how to do math or tell jokes. It becomes a "yes-man" that is safe but boring.
The paper shows that MMPO is better at keeping the robot's original smarts. It learns to be polite without forgetting how to be a general-purpose assistant.
It Wins More Often:
When they pitted their new method against the old ones in a competition (AlpacaEval), MMPO consistently won or tied for first place, especially on the smaller, more efficient models.

The "Secret Sauce" Ingredients

The paper also ran experiments to see what parts of their recipe were essential. They found two key ingredients that made the magic work:

In-Batch Normalization: This is like making sure everyone in a classroom is graded on the same scale. If one student gets an A+ and another gets an F, the teacher adjusts the scores so the comparison is fair. This prevents one weird example from messing up the whole lesson.
No "Extra" Entropy: They found that trying to force the robot to be "diverse" (entropy maximization) actually made it worse. They let the math do the work naturally.

Summary

In short, this paper says: "Stop trying to build a complex reward system to teach robots what humans like. Instead, use a simple statistical trick that treats human preferences as clues. This trick naturally teaches the robot to be helpful, keeps it stable, and ensures it doesn't forget how to be smart."

They call this new method MMPO (Maximum Marginal Likelihood Preference Optimization), and they believe it's a simpler, more reliable way to align AI with human values.

Technical Summary: Offline Preference Optimization via Maximum Marginal Likelihood Estimation

Problem Statement

Aligning Large Language Models (LLMs) with human preferences is a critical step for real-world deployment, traditionally achieved through Reinforcement Learning from Human Feedback (RLHF). However, the standard RLHF pipeline is complex and unstable, requiring the training of a separate reward model and the use of sensitive policy gradient algorithms like Proximal Policy Optimization (PPO). While Direct Preference Optimization (DPO) simplified this by reframing alignment as a classification problem, its loss function is still derived from the RLHF objective, which implicitly encourages optimizing the entropy of the language model. The authors investigate whether effective alignment can be achieved under even more relaxed assumptions, specifically by removing the explicit need for both a reward function and the entropy maximization term inherent in standard RLHF objectives.

Methodology: MMPO

The paper proposes Maximum Marginal Likelihood Preference Optimization (MMPO), a novel offline preference alignment algorithm.

Theoretical Foundation

MMPO recasts the alignment problem through the lens of Maximum Marginal Likelihood (MML) estimation. Instead of directly optimizing for preferences, the method maximizes the marginal log-likelihood of a preferred text output $y$ given an input $x$ .

Formulation: The target is to maximize $P_\theta(y|x) = \sum_z \pi_\theta(z|x) \times P^*(y|x, z)$ , where $z$ represents possible text completions and $P^*$ is the true underlying distribution.
Approximation: Since the summation over all possible $z$ is intractable, the method approximates the marginalization using the provided preference pair $(z_w, z_l)$ (chosen and rejected responses) as samples.
Stability: To ensure numerical stability during training, the objective is reformulated using the log-sum-exp operation.

Theoretical Connection to Preference Optimization

The authors demonstrate (Theorem 3.2) that optimizing the approximate MML objective on a preference dataset implicitly performs preference optimization.

Implicit Gradient: The resulting gradient update is shown to be equivalent to a weighted sum of gradients for the chosen ( $z_w$ ) and rejected ( $z_l$ ) responses.
Weighting Mechanism: The update naturally up-weights the chosen response relative to the rejected one, modulated by a sigmoid function of their score difference ( $\sigma(s_w - s_l)$ ).
Key Distinction: This process achieves preference alignment without explicitly formulating a reward function or an entropy-maximizing term.

Algorithm Implementation

The MMPO loss function incorporates several specific design choices:

Score Definition: Scores $s_w$ and $s_l$ are defined based on log-probabilities and a reward term (approximated via reference model log-probabilities and a hyperparameter $\beta$ ).
In-Batch Normalization: To stabilize training and prevent outliers from dominating gradients, rewards are normalized within a mini-batch. This ensures relative scores are on a consistent scale.
Auxiliary Objective: An auxiliary term, $\log \sigma(s_w - s_l)$ , is added to explicitly reinforce the preference constraint ( $s_w \ge s_l$ ), providing a direct supervisory signal similar to DPO but operating on normalized scores rather than log-probability ratios relative to an SFT model.

Key Contributions

Algorithm Introduction: The proposal of MMPO, a new offline preference optimization algorithm that treats preference pairs as samples for MML estimation.
Theoretical Analysis: A proof showing that MMPO on preference pairs is equivalent to a weighted gradient update that inherently favors preferred responses without explicit reward modeling or entropy terms.
Empirical Stability: Demonstration that MMPO is significantly more stable with respect to the key hyperparameter $\beta$ compared to DPO and SimPO.
Capability Preservation: Evidence that MMPO better preserves the general language capabilities (reasoning and understanding) of base instruction-tuned models compared to existing baselines.
Benchmark Performance: Achievement of competitive or superior win-rates on the AlpacaEval-2 benchmark across models ranging from 135M to 8B parameters.

Experimental Results

The authors evaluated MMPO on four model sizes (SmolLM2-135M, SmolLM2-360M, Llama3.2-1B, and Llama3-8B) using the Orca DPO Pairs and UltraFeedback-Binarized datasets.

Stability: Unlike DPO and SimPO, which showed significant sensitivity to the hyperparameter $\beta$ , MMPO maintained consistent performance across a range of $\beta$ values (0.01, 0.05, 0.5).
Preference Alignment: MMPO achieved strong win-rates on AlpacaEval-2. For instance, on SmolLM2-135M, MMPO improved win-rates monotonically to 67.3%, whereas DPO yielded very low win-rates and SimPO degraded significantly over training epochs.
General Capability Retention: Evaluated on the LM Harness benchmark (10 diverse language understanding tasks), MMPO caused minimal degradation in general language capabilities. In contrast, DPO and SimPO often showed more significant drops in accuracy, particularly when tuned for high preference alignment.
Pareto Optimality: MMPO demonstrated a more favorable trade-off profile, delivering higher preference win-rates with minimal sacrifice to general reasoning capabilities compared to baselines.

Ablation Studies

Experiments on the SmolLM2-360M model revealed:

Length Normalization: Improved general accuracy but did not significantly boost preference win-rates.
Entropy Maximization: Explicitly adding an entropy term degraded performance on both benchmarks, suggesting that the implicit optimization in MMPO is sufficient and that explicit diversity encouragement is detrimental in this context.
Auxiliary Loss & Normalization: Removing the DPO-style auxiliary loss or in-batch normalization slightly improved win-rates but degraded peak general accuracy, indicating a trade-off where the core log-sum-exp term drives performance while the auxiliary components help preserve general capabilities.

Significance and Claims

The paper claims that MMPO offers a simpler, more stable alternative to current alignment paradigms. By recasting alignment as an MML estimation problem, it eliminates the need for explicit reward models and entropy regularization while implicitly achieving the desired preference optimization. The authors emphasize that MMPO's stability is an inherent property of its formulation (stemming from in-batch normalization and the log-sum-exp operation) rather than a result of complex calibration mechanisms.

The work suggests that effective alignment does not strictly require the complex machinery of RLHF or the specific constraints of DPO. Instead, maximizing the marginal likelihood of preferred outputs using preference pairs as samples provides a robust path to alignment that preserves the model's foundational language capabilities. The authors note that while the current evaluation is limited to offline settings and models up to 8B parameters (due to resource constraints), the method represents a promising direction for simplifying the LLM alignment pipeline.