Technical Summary: The Mirage of Optimizing Training Policies (MIPU)

1. Problem Statement: Objective Misalignment in LLM RL

Reinforcement Learning (RL) has become a critical paradigm for post-training Large Language Models (LLMs), particularly for enhancing reasoning and instruction-following capabilities. However, modern LLM RL pipelines often suffer from training-inference mismatch. In practice, LLMs utilize separate engines for generation (inference, e.g., vLLM, SGLang) and gradient computation (training, e.g., FSDP, Megatron). Even with synchronized model parameters, differences in precision (e.g., FP8 vs. FP16), decoding implementations, and serving backends cause the training policy ($\pi$) and the inference policy ($\mu$) to assign different probabilities to the same trajectories.

Existing works attempt to mitigate this mismatch by correcting sampler-side ratios, filtering unstable samples, or decaying learning rates. However, the authors argue that these approaches address the symptom rather than the root cause: objective misalignment. Standard RL algorithms optimize the training policy ($\pi$), assuming that an improvement in $\pi$ ($J(\pi_{k+1}) - J(\pi_k) \ge 0$) guarantees an improvement in the deployed inference policy ($\mu$). The paper demonstrates that under mismatch conditions, this implication does not hold:
$$J(\pi_{k+1}) - J(\pi_k) \ge 0 \nRightarrow J(\mu_{k+1}) - J(\mu_k) \ge 0$$
Consequently, a synchronized update that appears beneficial on the training side may actually degrade the performance of the inference policy used in deployment.

2. Methodology: Monotonic Inference Policy Update (MIPU)

To address this, the authors propose a new optimization principle, Monotonic Inference Policy Improvement (MIPI), which shifts the objective from improving the training policy to ensuring the monotonic improvement of the inference policy ($J(\mu_{k+1}) - J(\mu_k) \ge 0$). They realize this principle through a two-step framework called Monotonic Inference Policy Update (MIPU).

The core of MIPU is the decomposition of the inference policy improvement into three terms:
$$J(\mu_{k+1}) - J(\mu_k) = \underbrace{[J(\mu_{k+1}) - J(\pi_{k+1})]}_{\text{Term 1: Post-update Inference Gap}} + \underbrace{[J(\pi_{k+1}) - J(\pi_k)]}_{\text{Term 2: Training-side Update}} + \underbrace{[J(\pi_k) - J(\mu_k)]}_{\text{Term 3: Pre-update Inference Gap}}$$

Step 1: Sampler-Referenced Policy Update

This step optimizes Terms 2 and 3 combined ($J(\pi_{k+1}) - J(\mu_k)$).

• Mechanism: Instead of using the standard training policy $\pi_k$ as the reference for the surrogate objective (as in standard GRPO/PPO), MIPU uses the inference policy $\mu_k$ (the sampler) as the reference.
• Implementation: The authors adopt a Truncated Importance Sampling (TIS) approach. They calculate a correction weight $w_i^k = \pi_k(y_i|x) / \mu_k(y_i|x)$ to account for the pre-update mismatch. To prevent high variance, this weight is truncated. The clipping constraint is then applied only to the current update ratio $r_i(\theta) = \pi_\theta(y_i|x) / \pi_k(y_i|x)$.
• Goal: This generates a candidate training policy $\pi_{k+1}$ that is theoretically aligned with the sampler $\mu_k$, ensuring the "halfway" monotonicity $J(\pi_{k+1}) - J(\mu_k) \ge 0$.

Step 2: Inference-Gap-Aware Update Acceptance

This step validates Term 1 ($J(\mu_{k+1}) - J(\pi_{k+1})$), which represents the gap between the candidate training policy and its synchronized inference realization.

• Mechanism: After synchronizing $\pi_{k+1}$ to the inference engine to create $\mu_{k+1}$, the system evaluates the post-update inference gap.
• Proxy Estimation: Since the exact advantage under $\pi_{k+1}$ is unavailable, MIPU uses a reverse identity to estimate the gap using validation rollouts from $\mu_{k+1}$. It computes a proxy $\hat{T}_{post} = -\mathbb{E}[\rho_i \hat{A}_{\mu_{k+1}}]$, where $\rho_i$ is a length-normalized importance weight correcting for the distribution shift from $\mu_{k+1}$ to $\pi_{k+1}$.
• Acceptance Criterion: The update is accepted only if $\hat{T}_{post} \ge -c$, where $c$ is a tolerance parameter. If the gap is too negative (indicating the inference policy underperforms the training candidate), the update is rejected, and the system rolls back to the previous checkpoint.

3. Key Contributions

1. MIPI Principle: The paper identifies and formalizes the objective misalignment in LLM RL, proposing that the optimization target must be the inference policy trajectory, not the training surrogate.
2. MIPU Framework: A novel two-step RL framework that decouples candidate construction (Step 1) from inference-gap validation (Step 2). It allows for flexible implementation of sampler-referenced updates and inference-aware acceptance.
3. Empirical Validation: Demonstrates that MIPU improves both reasoning performance and training stability in high-mismatch settings (specifically FP8-quantized rollouts) compared to standard GRPO and other stabilization baselines.

4. Experimental Results

The authors evaluated MIPU on Qwen3-1.7B and Qwen3-4B models using FP8-quantized rollouts to induce significant training-inference mismatch. Training was conducted on mathematical reasoning datasets (DAPO-Math-17, DeepMath-103K) and evaluated on five benchmarks (MATH-500, AIME24, AMC23, Minerva, OlympiadBench).

• Performance: MIPU achieved the highest average accuracy across both model scales (66.71% on Qwen3-4B and 53.97% on Qwen3-1.7B), outperforming baselines like standard GRPO, MIS (mismatch filtering), and LR-decay.
• Stability: While baselines often exhibited sharp performance degradation or collapse after reaching intermediate peaks, MIPU maintained stable training dynamics throughout the process.
• Ablation Studies:
  • Step 1 only: Improved candidate quality but failed to prevent accumulation of mismatch errors, leading to instability.
  • Step 2 only: Prevented collapse by rejecting bad updates but could not improve performance if the underlying baseline candidates were biased.
  • Full MIPU: The combination was necessary; Step 1 corrected the update direction, while Step 2 filtered unreliable candidates, resulting in both higher peak performance and long-term stability.
• Analysis of Step 2: The authors verified that Step 2's success is not merely due to rejecting more updates (a "random rollback" control failed despite rejecting more updates). Instead, its effectiveness stems from using the inference-gap signal to distinguish between harmful and benign synchronized candidates.

5. Significance and Claims

The paper claims that training-inference mismatch should not be viewed solely as a low-level system discrepancy to be minimized, but as a fundamental shift in the objective of policy improvement. By shifting the focus from the training policy to the inference policy, MIPU addresses the "mirage" where training-side improvements do not translate to deployment gains.

The authors modestly note that their experiments are currently limited to moderate-scale models due to computational constraints and that Step 2 represents a flexible design space rather than a fixed implementation. They suggest future work could explore more efficient estimators for the post-update gap and apply these principles to larger models and diverse RL systems. The core contribution is a paradigm shift: ensuring that the policy used for deployment is the one being optimized, rather than assuming training-side optimization suffices.