RIFT: Repurposing Negative Samples via Reward-Informed Fine-Tuning

1. Problem Statement

Large Language Model (LLM) alignment typically relies on Supervised Fine-Tuning (SFT) using high-quality expert data or Rejection Sampling Fine-Tuning (RFT), which filters self-generated samples based on a quality threshold. Both approaches suffer from significant inefficiencies:

• Data Scarcity & Cost: SFT requires expensive, curated expert demonstrations.
• Data Waste: RFT discards all "negative" (sub-threshold) samples, wasting the computational resources used to generate them and losing valuable information about model failure modes.
• Training Instability: Naive attempts to incorporate negative samples into the loss function (e.g., multiplying log-likelihood by a negative reward) lead to gradient explosion. As the model suppresses a negative sample (probability $\to 0$), the gradient magnitude approaches infinity, causing training collapse and catastrophic forgetting.

2. Methodology: Reward-Informed Fine-Tuning (RIFT)

RIFT is a framework designed to utilize all self-generated trajectories (both positive and negative) by reweighting the loss function based on scalar rewards, while ensuring numerical stability.

Core Concept

Instead of discarding negative samples (as RFT does) or ignoring them (as standard SFT does), RIFT assigns a scalar reward $r$ to every generated response $y$ given input $x$:

• Positive Reward ($r > 0$): Encourages the model to increase the likelihood of the response.
• Negative Reward ($r < 0$): Encourages the model to decrease the likelihood of the response.

Theoretical Innovation: Stabilized Loss Formulation

The paper identifies that a naive signed-weighted loss $L_{naive} = -E[r \cdot \log \pi_\theta(y|x)]$ is unbounded from below for negative rewards. As $\pi_\theta(y|x) \to 0$, the gradient $\frac{\partial L}{\partial \pi} \to \infty$, leading to instability.

To solve this, RIFT introduces a hybrid loss function that decouples positive and negative samples:

1. For Positive Samples ($D^+$): Retains the standard Negative Log-Likelihood (NLL) objective to maximize likelihood.
   $$-E_{D^+}[r \cdot \log \pi_\theta(y|x)]$$
2. For Negative Samples ($D^-$): Replaces the logarithmic term with a Linear Probability Approximation (based on the first-order Taylor expansion of $\log u \approx u-1$).
   $$-E_{D^-}[r \cdot \pi_\theta(y|x)]$$
   Note: Since $r < 0$, the term $-r$ is positive. Minimizing this term forces $\pi_\theta(y|x)$ toward 0, but the gradient remains constant ($\nabla \pi = -r$) rather than exploding.

Final RIFT Loss:
$$L_{RIFT}(\theta) = -\sum_{(x,y) \in D^+} r(x,y) \log \pi_\theta(y|x) - \sum_{(x,y) \in D^-} r(x,y) \pi_\theta(y|x)$$

This formulation ensures:

• Boundedness: The loss contribution from negative samples is bounded within $[r, 0]$, preventing divergence to $-\infty$.
• Stability: Gradients for negative samples remain finite even as probabilities approach zero.
• Optimization: It effectively maximizes a surrogate lower bound of the expected reward.

3. Key Contributions

• Novel Framework: Proposes RIFT, a simple extension of SFT that repurposes rejected negative samples via reward-informed reweighting, eliminating the need for hard thresholding.
• Theoretical Stability: Provides a principled loss formulation that resolves the gradient explosion problem inherent in naive reward integration, ensuring robust training without reference models.
• Data Efficiency: Demonstrates that learning from mixed-quality data (both correct and incorrect trajectories) yields better generalization than filtering data, effectively turning "failures" into learning signals.
• Resource Efficiency: Unlike Direct Preference Optimization (DPO) or other RL-based methods, RIFT does not require a reference model, significantly reducing peak memory usage.

4. Experimental Results

The authors evaluated RIFT on mathematical reasoning benchmarks (GSM8K, MATH, Olympiad, AIME, etc.) using various base models (Qwen2.5-Math, Qwen3, DeepSeek-R1-Distill).

• Performance: RIFT consistently outperformed strong baselines (SFT, RFT, DFT, DPO, SimPO, KTO) across all model scales.
  • On Qwen2.5-Math-1.5B (trained on MATH), RIFT achieved 37.0% Mean@8 accuracy, surpassing DPO (31.0%) and RFT (25.6%) by significant margins.
  • On DeepSeek-R1-Distill-Qwen-1.5B, RIFT achieved the highest gains, improving Mean@8 by +6.1% over RFT and +1.2% over DPO.
• Generalization: RIFT showed superior Out-of-Distribution (OOD) generalization compared to SFT and DFT, which often degraded on harder benchmarks.
• Efficiency:
  • Memory: RIFT reduced peak VRAM usage by nearly 50% compared to DPO (e.g., ~20GB vs. ~43GB) because it does not load a reference model.
  • Throughput: Computational cost is comparable to SFT and RFT.
• Robustness: RIFT remained stable even with moderately capable base models where RFT failed to generate sufficient high-quality data for self-improvement.

5. Significance

• Paradigm Shift: RIFT challenges the standard "filter-and-discard" approach of RFT, proving that negative samples contain critical information for refining model reasoning and error correction.
• Scalability: By removing the dependency on reference models and expensive preference pairs, RIFT offers a highly scalable and cost-effective path for aligning LLMs, particularly in domains with verifiable rewards (like math and coding).
• Practicality: The method is simple to implement (a modification to the loss function) and requires no complex RL infrastructure, making it accessible for widespread adoption in post-training pipelines.

In conclusion, RIFT represents a robust, data-efficient, and computationally lightweight alternative to current alignment methods, enabling models to learn effectively from the full spectrum of their own generated outputs.