Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?

1. Problem Statement

Preference optimization methods like DPO (Direct Preference Optimization) and KTO are standard for aligning Large Language Models (LLMs) with human preferences. However, the specific properties of preference data that drive improvements in reasoning capabilities remain poorly understood.

While recent work (Geng et al., 2025) introduced the "Delta Learning Hypothesis"—suggesting that the relative quality gap between two weak model responses is sufficient to train a stronger student, even without ground-truth correctness—this hypothesis has not been rigorously tested in the context of verifiable reasoning tasks (math, STEM, code).

The paper addresses two core questions:

1. What aspects of a preference pair (chosen vs. rejected) actually improve a reasoning model's performance?
2. How do different definitions of "quality delta" (generator-level vs. sample-level) impact downstream reasoning gains?

2. Methodology

The authors propose a framework decomposing the "delta" (quality gap) in preference pairs into two distinct notions:

A. Two Notions of Delta

1. Generator-Level Delta: The capability gap between the models generating the chosen and rejected responses. This is a macro-level design choice (e.g., pairing a 49B model's output against a 4B model's output).
2. Sample-Level Delta: The quality difference within a single preference pair, measured along specific reasoning dimensions. This is a micro-level characteristic of the data itself.

B. Experimental Setup

• Base Model: Nemotron-8B.
• Data Source: OpenR1-Math-220k (220k math problems with traces generated by DeepSeek-R1).
• Generators: The authors synthesized reasoning traces using various models from two families: Nemotron (4B–49B) and s1 (3B–32B).
• Evaluation:
  • In-Domain: Mathematical reasoning (Easy: MATH-500, GSM8K; Hard: Minerva, AIME).
  • Out-of-Domain: STEM knowledge (MMLU-Pro, TheoremQA) and Code generation (LiveCodeBench).
• Quality Measurement: An LLM-as-a-judge (GPT-OSS-120b) rated reasoning traces on five dimensions:
  1. Factuality: Grounding to problem constraints (no hallucination).
  2. Strategy Coherence: Reasonableness of the overall plan.
  3. Step Coherence: Logical flow between individual reasoning steps.
  4. Computational Precision: Absence of arithmetic/sign errors.
  5. Signal-to-Noise: Relevance of calculations vs. redundancy/verbosity.

C. Experimental Design

1. Scaling Generator-Level Delta: Fixed the rejected model (s1-3B) and scaled the chosen model (from s1-7B up to Nemotron-49B) to observe performance changes.
2. Testing Outcome Correctness: Constructed preference pairs across four correctness combinations (Correct-vs-Incorrect, Correct-vs-Correct, Incorrect-vs-Incorrect, Incorrect-vs-Correct) using a single generator (Nemotron-4B) to isolate the effect of answer correctness.
3. Sample-Level Data Selection: Selected subsets of the top-k pairs with the largest sample-level deltas in each of the five quality dimensions to test data efficiency.

3. Key Contributions & Results

Contribution 1: Validation of Delta Learning in Reasoning

The study confirms that the Delta Learning Hypothesis holds for reasoning tasks. Training on preference pairs generated by weaker models (e.g., s1-7B vs. s1-3B) yields downstream performance comparable to training on pairs from a much stronger model (DeepSeek-R1), provided a meaningful quality gap exists.

• Result: DPO on weak-model pairs outperforms the base model and matches DPO on strong-model pairs, even without ground-truth verification.

Contribution 2: The Scaling Law of Generator-Level Delta

The authors investigated how increasing the capability gap between generators affects performance.

• In-Domain (Math): Performance gains plateaued as the generator capability increased. The base model was already proficient, limiting further gains.
• Out-of-Domain (STEM/Code): Performance improved steadily as the generator-level delta increased.
• Mechanism: As the chosen model scaled up, the sample-level delta (the quality gap in reasoning dimensions) widened significantly across all five measured dimensions. This widening gap explains the improved generalization to out-of-domain tasks.

Contribution 3: Outcome Correctness is Not the Dominant Signal

Contrary to the assumption that "Correct vs. Incorrect" is the primary driver of learning:

• Result: All four correctness combinations (including Incorrect-vs-Incorrect and Incorrect-vs-Correct) improved performance over the base model.
• Finding: While Correct-vs-Incorrect performed best, the gap between configurations was modest. Crucially, Incorrect-vs-Incorrect pairs still yielded gains, proving that the learning signal resides in the reasoning process quality, not just the final answer correctness.

Contribution 4: Step Coherence as the Optimal Selection Criterion

By filtering data based on the magnitude of sample-level delta in specific dimensions:

• Data Efficiency: Training on as few as 5,000 high-delta pairs matched or exceeded the performance of the full 16,500-pair dataset.
• Best Dimension: Step Coherence (the logical flow between steps) was identified as the most consistent and effective criterion for selecting high-value training examples.
• Implication: Models learn most from the structural integrity of the reasoning chain, not just the correctness of the result.

4. Significance and Practical Implications

The paper provides a "twofold recipe" for improving reasoning performance through preference optimization:

1. Maximize Generator-Level Delta: When constructing preference data, pair outputs from models with the largest possible capability gap. This is particularly effective for improving generalization to out-of-domain tasks (STEM, Code).
2. Exploit Sample-Level Delta: Do not rely solely on answer correctness for data selection. Instead, filter for pairs with large differences in Step Coherence. This allows for highly data-efficient training, achieving state-of-the-art results with a fraction of the data.

Broader Impact:

• Cost Reduction: The findings suggest that expensive, ground-truth verified datasets are not strictly necessary for reasoning alignment; high-quality preference pairs can be constructed from weaker models if the quality gap is managed correctly.
• Data Efficiency: Selecting for specific reasoning qualities (like step coherence) can reduce the computational cost of training by up to 70% (using 5k vs. 16.5k pairs) without performance loss.
• Theoretical Shift: The work shifts the focus of preference optimization from "outcome supervision" to "process supervision," highlighting that the structure of reasoning is more critical than the result.