IRIS: Interpolative Rényi Iterative Self-play for Large Language Model Fine-Tuning

1. Problem Statement

Large Language Models (LLMs) are typically aligned with human preferences using Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO). However, these methods rely on expensive, curated preference datasets that are difficult to scale. Self-Play Fine-Tuning (SPIN) offers a solution by allowing models to improve without new human annotations, framing fine-tuning as a game where the current model learns to distinguish human responses from its own previous generations.

Despite its promise, existing SPIN methods suffer from suboptimal learning dynamics due to their reliance on a fixed divergence regime (e.g., KL-divergence, Jensen-Shannon, or $\chi^2$-divergence).

• KL-divergence: Provides smooth gradients early in training but loses sensitivity as distributions converge, leading to vanishing gradients.
• $\chi^2$-divergence: Offers fine-grained discrimination near convergence but can over-penalize outliers early on, causing instability.
• Jensen-Shannon (JS): Ensures stability via independent classification but lacks sensitivity in later stages.

No single fixed divergence provides optimal performance across all training stages. The core problem is the lack of a unified framework that can adapt the divergence geometry to the evolving distributional gap between the model and the target data.

2. Methodology: IRIS

The authors propose IRIS (Interpolative Rényi Iterative Self-play), a unified framework based on Rényi divergence ($D_\alpha$), which continuously interpolates between different divergence regimes via an order parameter $\alpha$.

A. The IRIS Objective

IRIS constructs a loss function that decomposes into two independent terms over annotated (real) and synthetic (model-generated) data, utilizing the tilted risk principle. The objective function $L_{IRIS}(\theta; \alpha)$ is defined as:

$$L_{IRIS}(\theta; \alpha) = -\frac{1}{\alpha-1} \log \mathbb{E}_{y \sim p_{data}} \left[ e^{(\alpha-1)r(x,y)} \right] + \frac{1}{\alpha} \log \mathbb{E}_{y' \sim p_{\theta_t}} \left[ e^{\alpha r(x,y')} \right]$$

Where:

• $r(x, \hat{y}) = \log p_\theta(\hat{y}|x) - \log p_{\theta_t}(\hat{y}|x)$ is the log-ratio reward.
• The first term is a negated tilted risk on real data (encouraging high rewards for human responses).
• The second term is a tilted risk on synthetic data (penalizing high rewards for model-generated responses).
• $\alpha > 0, \alpha \neq 1$ is the Rényi order parameter.

Key Properties:

1. Fixed Point: The global minimum is attained when $p_\theta = p_{data}$, ensuring convergence.
2. Gradient Structure: The gradient involves exponential importance weights ($\tilde{w} \propto \exp((\alpha-1)r)$). This acts as a differentiable attention mechanism, concentrating learning on informative samples.
3. Unification: By varying $\alpha$, IRIS recovers existing methods as special cases:
   • $\alpha \to 1$: Recovers KL-divergence (SPIN, T-SPIN, DRIFT).
   • $\alpha = 2$: Recovers $\chi^2$-divergence (SPIF, SPPO).
   • $\alpha = 1/2$: Recovers Hellinger distance (a new regime emphasizing mode-covering).
   • $\alpha = 1$ (via NCE limit): Recovers Jensen-Shannon (SPACE, SGALM).

B. Adaptive Order Schedule

Instead of fixing $\alpha$, IRIS introduces an adaptive schedule $\alpha_t$ that adjusts the divergence geometry based on the current distributional gap.

• Distributional Gap Feedback: $\alpha_t = 1 + c \cdot \hat{D}_t$, where $\hat{D}_t$ is the expected reward gap between real and synthetic data.
  • Early Training (Large Gap): $\alpha_t$ is large (e.g., $\approx 3$), promoting mode-covering exploration and concentrating weights on high-gap samples.
  • Late Training (Small Gap): $\alpha_t$ decreases toward 1, shifting to smooth refinement with uniform gradients (KL regime) to avoid instability near convergence.
• Geometric Annealing: A simpler alternative interpolating from $\alpha_{max}$ to $\alpha_{min}$ over iterations.

3. Key Contributions

1. Theoretical Unification: The paper establishes that diverse self-play objectives (SPIN, SPACE, SPIF) are merely specific regimes of the Rényi divergence family, providing a unified theoretical perspective.
2. IRIS Framework: Proposes a novel self-play framework with a continuously adjustable objective and an adaptive order schedule that dynamically selects the optimal divergence for the current training stage.
3. Theoretical Analysis: Proves the fixed-point property of IRIS and analyzes how $\alpha$ controls gradient concentration and importance weighting.
4. Empirical Superiority: Demonstrates that IRIS outperforms strong baselines across multiple benchmarks and model sizes, maintaining stable improvements across iterations where others plateau or degrade.

4. Experimental Results

Experiments were conducted on Zephyr-7B and Qwen2.5-3B-Instruct across ten benchmarks (including GSM8K, MATH, MMLU, IFEval, etc.).

• Performance: IRIS achieved the highest average scores. On Zephyr-7B, it reached 44.57% at Iteration 4, surpassing T-SPIN (43.56%), SPACE (42.40%), and SPIN (41.17%).
• Stability: Unlike SPIN and SPIF, which showed performance degradation or plateauing in later iterations, IRIS maintained monotonic improvement.
• Data Efficiency: A critical finding is that IRIS trained on only 26k annotated samples (for 2 iterations) surpassed standard Supervised Fine-Tuning (SFT) trained on the full 200k dataset.
• Ablation Studies:
  • Fixed $\alpha$ values (e.g., 0.5, 1.5, 2.0) resulted in trade-offs between stability and aggressiveness.
  • The Distributional Gap Feedback schedule outperformed geometric annealing, confirming that adapting to the actual training signal is superior to a predetermined trajectory.
  • Training dynamics showed that IRIS successfully increases rewards on real data while decreasing them on synthetic data, avoiding the "reward collapse" seen in gap-based methods.

5. Significance

IRIS represents a significant advancement in LLM alignment by addressing the fundamental limitation of fixed-divergence self-play methods.

• Theoretical Insight: It bridges the gap between disparate self-play algorithms, showing they are points on a continuum controlled by the Rényi order.
• Practical Impact: By dynamically adjusting the learning objective, IRIS enables models to extract more value from limited annotated data, reducing the cost of alignment.
• Scalability: The method is compatible with existing self-play codebases, requiring only a substitution of the loss module, making it easily adoptable for future large-scale training.

In summary, IRIS solves the "one-size-fits-all" problem in self-play fine-tuning by introducing a mathematically grounded, adaptive mechanism that optimizes the learning trajectory from exploration to refinement.