Technical Summary: GAC – Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training

1. Problem Statement

Large Language Models (LLMs) are typically post-trained using a hybrid paradigm combining Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). SFT stabilizes model output format using expert demonstrations, while RL improves reward-seeking behavior via on-policy rollouts. However, these objectives cannot be fully decoupled without mutual degradation (Niu et al., 2026).

Current hybrid methods often rely on fixed mixing schedules or heuristic rules to balance SFT and RL gradients. These static approaches fail to adapt when the relative noise of the two signals evolves during training. Specifically, they cannot account for:

• Evolving policy drift: Changes in the model's distribution over time.
• Reward noise: Variability in RL advantage estimates.
• Gradient disagreement: Conflicts between SFT and RL objectives.

This lack of adaptability can induce entropy collapse, late-stage over-imitation, or reward hacking. While recent works (e.g., CHORD, SRFT, HPT) have introduced dynamic weighting or conflict-aware coupling, none explicitly model both gradient noise variance and SFT–RL disagreement simultaneously to derive an optimal mixing weight.

2. Methodology: GAC (Guided Adaptive Controller)

The proposed method, GAC, reframes the mixing problem as a noise-aware control task. It consists of a theoretical closed-form estimator and a practical guided controller.

2.1 Theoretical Foundation: Closed-Form $\mu^*$

The core contribution is the derivation of an optimal mixing weight $\mu^*$ by minimizing the Mean Squared Error (MSE) between the mixed stochastic gradient and a target gradient.

• Assumptions: Let $\hat{g}_s$ and $\hat{g}_r$ be gradient estimators for SFT and RL, composed of noise-free gradients ($g^*_s, g^*_r$) and zero-mean noise ($\varepsilon_s, \varepsilon_r$) with variances $\sigma^2_s$ and $\sigma^2_r$.
• Objective: Minimize $E(\mu) = E[\|\mu\hat{g}_s + (1-\mu)\hat{g}_r - g^*\|^2]$, where $g^*$ is a target gradient defined by a desired trade-off ratio $\alpha_{tgt}$.
• Result: Under independence assumptions, the unique minimizer is:
  $$\mu^* = \frac{\alpha_{tgt} \Delta g^2 + \sigma^2_r}{\Delta g^2 + \sigma^2_s + \sigma^2_r}$$
  where $\Delta g^2 = \|g^*_s - g^*_r\|^2$ represents gradient disagreement.
• Interpretation: This formula embodies a bias–variance trade-off. When disagreement ($\Delta g^2$) is low, the weight approaches inverse-variance weighting. When disagreement is high, the weight defaults to the user-specified preference $\alpha_{tgt}$.

2.2 Practical Implementation: Proxy Signals & Online Estimation

Since true gradient statistics are expensive to compute at every step, GAC employs computationally tractable proxy signals estimated online from existing training tensors:

1. RL Uncertainty ($\sigma^2_r$): Estimated via sequence-level advantage dispersion.
2. SFT Uncertainty ($\sigma^2_s$): Estimated via length-normalized, trimmed Negative Log-Likelihood (NLL) variance.
3. Disagreement Proxy ($\Delta \tilde{g}^2$): Estimated via the variance of z-normalized coefficient differences between RL and SFT gradients.

These proxies are validated to have high Pearson correlations ($r \approx 0.76\text{--}0.84$) with true gradient statistics.

2.3 Stability-Motivated Design Guidelines

To prevent abrupt shifts and ensure training stability, the raw estimator $\mu^*$ is processed through a Guided Adaptive Controller with three guarded steps:

1. EMA Smoothing: $\mu_{ada} = \beta \mu_{t-1} + (1-\beta)\mu^*$ to dampen high-frequency noise.
2. Schedule Prior Blending: $\mu_{blend} = (1-\lambda)\mu_{prior} + \lambda \mu_{ada}$, where $\mu_{prior}$ is a warmup/cosine schedule, ensuring the controller does not deviate too far from a reasonable baseline.
3. Capped Updates: The step change $\delta_t$ is clipped to a small bound $\bar{c}$ (e.g., 0.01) to prevent destabilizing jumps.

Additionally, the target ratio $\alpha_{tgt}$ is dynamically adjusted to $\alpha_{ctrl}$ using a KL-controlled hysteresis rule to prevent policy drift.

3. Key Contributions

1. Noise-Aware Mixing Controller (C1): A closed-form solution for the optimal mixing weight derived from MSE minimization, balancing gradient noise variance and objective disagreement. It is instantiated via z-normalized proxies and wrapped in a stability-guaranteed controller with <1% computational overhead.
2. Token-wise Reweighting Integration (C2): The system adopts the token-wise reweighting function $\phi(p) = p(1-p)$ from CHORD (Zhang et al., 2025). The authors demonstrate this component is orthogonal to the global controller, providing an additional +0.4–1.4pp gain.
3. Systematic Evaluation (C3): Comprehensive evaluation across math, code, science, and logic benchmarks at 1.5B, 7B, and 14B parameter scales.

4. Experimental Results

Experiments were conducted on the Qwen2.5-7B-Instruct model (and scaled to 1.5B/14B) using OpenR1-Math-220k and other benchmarks (MBPP, HumanEval, GPQA, SciBench, BBH).

• Performance Gains:
  • Math (AMC): GAC + Token-φ achieves 67.2%, outperforming the best hybrid baseline (HPT, 63.4%) by +3.8pp and the noise-aware controller alone (GAC w/o φ) by +3.0pp over rule-based controllers.
  • Code: GAC achieves 78.8% on MBPP (+3.4pp over CHORD) and 83.5% on HumanEval.
  • Science/Logic: GAC improves GPQA by +3.9pp and BBH logical subsets by +3.1pp over HPT.
• Stability Improvements:
  • KL-Drift: Reduced KL-drift area by 28% compared to constant mixing.
  • Large Shifts: Reduced large $|\Delta \mu|$ events by >70%.
  • Training Dynamics: GAC maintains higher rollout accuracy, moderate response lengths (avoiding reward hacking spikes), and faster entropy stabilization compared to fixed schedules.
• Scalability: Gains scale with model size, showing +2.2pp at 1.5B, +3.8pp at 7B, and +3.3pp at 14B.
• Efficiency: The method adds only 3 EMA scalars and incurs <1% wall-time overhead with no measurable memory increase.

5. Significance and Claims

The paper claims that GAC successfully reframes SFT–RL mixing as a noise-aware control problem rather than a heuristic scheduling task.

• Primary Driver: Causal attribution analysis confirms that the noise-aware estimator (the MSE-derived signal) is the dominant contributor to performance gains (+3.7pp), while EMA smoothing and schedule priors provide necessary stability but smaller incremental gains.
• Robustness: The controller is robust to weak or misspecified prior schedules; even with "bad" constant priors, GAC corrects the mixing weight effectively.
• Limitations: The authors note that the stability analysis relies on idealized assumptions (L-smooth losses, KL-bounded updates) and provides design guidelines rather than strict convergence guarantees. The method is currently validated on tasks with verifiable rewards; extension to open-ended alignment with learned reward models remains future work.

In summary, GAC provides a theoretically grounded, low-overhead mechanism to dynamically balance SFT and RL based on real-time uncertainty estimates, significantly outperforming fixed schedules and heuristic controllers across diverse reasoning tasks.