How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum

1. Problem Statement

The paper addresses two critical failure modes in training reasoning models (models that generate latent chains of thought before answering) using only output-level supervision:

1. Cold-Start Stalling: When the initial probability of success ($p_0$) is very low, standard Reinforcement Learning from Verifiable Rewards (RLVR) and its variants (like GRPO or Rao-Blackwellized REINFORCE) often fail to make progress. The gradient signal is too weak to escape the "cold start" regime.
2. Noise Memorization: Conversely, methods that aggressively push the model to fit the training distribution (like standard Supervised Fine-Tuning or Maximum Likelihood Estimation) tend to memorize label errors and noise, reducing generalization and self-consistency.

Existing approaches treat these as separate problems. The authors argue that these are two ends of a spectrum governed by the model's "commitment" to the supervision signal. They propose a unified theoretical framework to balance the speed of escaping cold starts against the robustness to noise.

2. Methodology: The $J_Q$ Loss Continuum

The core contribution is the definition of a loss family, $J_Q$, derived from the Tsallis $q$-logarithm. This family interpolates between two poles:

• Exploitation Pole ($q=0$): Corresponds to maximizing expected accuracy (equivalent to RLVR with exact-match rewards). The loss is bounded ($1 - P_\theta$).
• Density-Estimation Pole ($q=1$): Corresponds to maximizing the log-marginal-likelihood over latent trajectories (equivalent to standard SFT/ML). The loss is unbounded ($-\log P_\theta$).

Key Theoretical Insight: Gradient Amplification

All members of the $J_Q$ family share the same per-example gradient direction, differing only by a scalar amplification factor:
$$\nabla_\theta \ell_q = P_\theta^{-q} \nabla_\theta \ell_0 = P_\theta^{1-q} \nabla_\theta \ell_1$$
Where $P_\theta = p_\theta(y^*|x^*)$ is the model's conditional success probability.

• The Parameter $q$ (Commitment): Controls how much the gradient is amplified for "hard" examples (low $P_\theta$).
  • Low $q$ (near 0): Minimal amplification. The model focuses on what it already knows (noise-robust) but escapes cold start slowly ($\Omega(1/p_0)$ time).
  • High $q$ (near 1): Massive amplification for low-probability examples. The model aggressively explores unfamiliar supervision, escaping cold start quickly ($\Theta(\log(1/p_0))$ time) but risks memorizing noise.

Gradient Estimators

Since $P_\theta$ is intractable (it involves summing over all latent trajectories), the authors derive two Monte Carlo estimators based on the dual factorization of the gradient:

1. GARL (Gradient-Amplified RL):

   • Mechanism: Samples trajectories from the prior $p_\theta(z|x^*)$ and amplifies the standard RL gradient by $(\bar{w}_M)^{-q}$.
   • Characteristics: Lower variance, generalizes RB-REINFORCE ($q=0$) and IWAE ($q=1$).
   • Use Case: Essential for cold-start scenarios where posterior sampling yields no valid trajectories.

2. PAFT (Posterior-Attenuated Fine-Tuning):

   • Mechanism: Importance-resamples trajectories from the posterior (those agreeing with the answer) and attenuates the SFT gradient by $(\bar{w}_M)^{1-q}$.
   • Characteristics: Produces semantically coherent gradients (only updates on valid rationales) but has higher variance due to resampling noise. Generalizes the E-step of EM ($q=1$).
   • Use Case: More stable during warm-start training, preventing the "collapse" seen in GARL on complex datasets.

3. Key Contributions

1. Unified Theoretical Framework: The paper provides the first theoretical account unifying cold-start stalling and noise memorization as a trade-off controlled by the parameter $q$.
2. Escape Speed Analysis: Theoretical proofs (Theorems 5.1 & 5.2) demonstrate that the time to escape cold start scales as $\Omega(1/p_0)$ for $q=0$ but improves to $\Theta(\log(1/p_0))$ for $q=1$. This explains why standard RLVR fails when $p_0$ is small.
3. Dual Estimators (GARL & PAFT): The derivation of two practical, drop-in estimators that allow practitioners to tune the "commitment" level without changing the model architecture.
4. Bias-Variance Trade-off: Analysis showing that while high $q$ accelerates learning, it introduces estimator bias ($O(q/MP_\theta^{q+1})$), necessitating intermediate values of $q$ for optimal performance.

4. Empirical Results

The authors evaluated GARL and PAFT on three reasoning benchmarks (FinQA, HotPotQA, MuSiQue) using a Qwen 3 0.6B model.

• Cold-Start Performance:

  • Standard GRPO and $q=0$ (RB-RLOO) failed completely (0% accuracy) on cold-start tasks.
  • GARL with $q=0.75$ successfully escaped cold start, achieving significant accuracy (e.g., 38.3% maj@16 on FinQA vs. 0% for GRPO).
  • $q=1$ escaped cold start but suffered from higher noise memorization, resulting in lower final accuracy than $q=0.75$.

• Warm-Start Performance:

  • FinQA: Training was stable across all $q$. GARL at $q=0.25$ achieved the best results (38.7% maj@16), outperforming GRPO by +11.8 points.
  • HotPotQA & MuSiQue: GARL collapsed (validation accuracy dropped to zero) during training at all tested $q$ values, likely due to pathwise term corruption or overfitting.
  • PAFT Stability: PAFT did not collapse on any benchmark. PAFT at $q=0.75$ achieved the best stable results on HotPotQA (47.9% maj@16, +14.4 over GRPO) and MuSiQue (22.4% maj@16).

• Prompt Independence: Remarkably, the cold-start GARL model (trained without task-specific prompts) outperformed the warm-start GRPO model (trained with prompts) on all metrics, suggesting that strong commitment ($q \approx 0.75$) allows the model to discover task structure directly from input-output pairs.

5. Significance and Practical Implications

• Solving the Cold-Start Bottleneck: The paper proves that the bottleneck in RLVR is not just gradient variance (addressed by Rao-Blackwellization) but the lack of gradient amplification for low-probability examples. Increasing $q$ solves this.
• Dynamic Training Strategy: The authors propose a practical recipe:
  • Cold Start: Use GARL with high $q$ ($\approx 0.75$) to force the model out of the zero-reward regime.
  • Warm Start: If training is stable (e.g., FinQA), use GARL with low $q$ ($\approx 0.25$) to minimize bias. If training is unstable (e.g., HotPotQA), switch to PAFT with high $q$ ($\approx 0.75$) to maintain stability via coherent gradients.
• Beyond RLVR: This framework offers a principled way to interpolate between Reinforcement Learning and Supervised Fine-Tuning, moving beyond the rigid "SFT then RL" pipeline to a continuous spectrum of commitment.

In summary, the paper demonstrates that commitment to supervision is a tunable hyperparameter. By adjusting $q$, practitioners can control the trade-off between the speed of learning (escaping cold starts) and the robustness of learning (avoiding noise memorization), leading to state-of-the-art performance on complex reasoning tasks.