Technical Summary: Not Only Where, But When: Temporal Scheduling for RLVR

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become a dominant paradigm for post-training Large Language Models (LLMs), particularly for reasoning tasks. However, standard RLVR approaches, such as Group Relative Policy Optimization (GRPO), typically broadcast a single scalar reward across the entire generated sequence. This results in a uniform optimization signal where all sampled tokens participate equally in policy updates, overlooking the heterogeneous policy behaviors exhibited along a trajectory (e.g., reasoning scaffolding, logical connections, reflection, and answer convergence).

Existing solutions attempt to address this by introducing credit allocation mechanisms to determine where learning signals should be distributed. These include:

• Process Reward Models (PRMs): Providing step-level feedback (high annotation cost).
• Token-level Proxies: Using metrics like token entropy, likelihood, or KL divergence to reweight advantages or select sparse subsets of tokens for optimization.

The Core Limitation: While these methods identify specific tokens to prioritize, the allocation criteria remain stagnant throughout the entire training process. Once a proxy (e.g., high entropy) is selected, the corresponding policy behavior is continuously emphasized. The authors argue that this static approach limits resilient policy evolution and can lead to the sacrifice of policy entropy when trying to accommodate diverse behaviors simultaneously. The paper posits that the temporal dimension—specifically when learning signals are scheduled—has been underexplored and is as critical as where they are allocated.

Methodology

The paper proposes Temporal Scheduling, a framework that dynamically adjusts credit allocation criteria over the course of training rather than fixing them.

1. Trajectory Percentiles as a Natural Perspective

Instead of relying solely on complex policy proxies, the authors introduce the Trajectory Percentile Score (TP-Score) to distinguish policy behaviors based on token position.

• Analysis: Empirical analysis reveals distinct token distributions and entropy profiles across trajectory positions.
  • Early Percentiles: Associated with reasoning scaffolding and constrained patterns; exhibit lower entropy.
  • Middle Percentiles: Associated with reflection and critical reasoning; exhibit the highest entropy and exploratory behavior.
  • Later Percentiles: Associated with answer realization and summarization; exhibit intermediate entropy.
• Insight: Trajectory percentiles provide a natural, simple anchor for distinguishing heterogeneous behaviors without requiring complex proxy calculations.

2. The Temporal Scheduling Mechanism

The core innovation is scheduling the credit allocation factor, $\hat{f}_t(\tau)$, as a function of training progress $\tau$.
$$\hat{f}_t(\tau) = S(\tau) \hat{f}_t$$
Where:

• $\hat{f}_t$ is the original credit allocation factor derived from existing methods (e.g., entropy-based reweighting or sparse selection).
• $S(\tau)$ is a monotonically decreasing schedule function ranging from 1 to 0.
• Strategy: The optimization begins by prioritizing the most targeted tokens (e.g., those identified by the specific proxy or later trajectory positions) to reinforce specific behaviors. As training progresses, the criteria gradually attenuate, expanding the optimization scope to include more general tokens.

3. Integration with Existing Paradigms

The method is designed as a "drop-in" replacement for existing credit allocation strategies:

• Continuous Advantage Reweighting: The weight $\psi(\mu_t)$ is scaled by $S(\tau)$.
• Discrete Sparse Optimization: The selection threshold $\epsilon$ is dynamically relaxed: $\mu_t \geq S(\tau) \cdot \epsilon$. This gradually includes more tokens into the optimization set over time.

4. Trajectory Percentile Scheduling (TP-Schedule)

The authors propose a specific instantiation using trajectory percentiles directly:

• Initialization: Focus exclusively on later tokens (high percentiles) responsible for reliable continuation.
• Progression: Gradually incorporate earlier tokens (lower percentiles) responsible for reasoning scaffolding.
• Formula: $\hat{f}_t(\tau) = \mathbb{I}[t/T \geq S(\tau) \cdot 1]$, where $t/T$ is the normalized position. This creates a structured shift where optimization propagates from the end of the sequence backward to the beginning.

Key Contributions

1. Identification of the Temporal Dimension: The paper argues that the timing of credit allocation is a critical, underexplored optimization dimension. It challenges the stagnant allocation paradigm, proposing that scheduling criteria over training steps leads to more resilient policy evolution.
2. Temporal Scheduling Framework: A general method to complement existing credit allocation techniques (both continuous reweighting and sparse optimization) by prioritizing targeted tokens early and gradually attenuating toward general optimization.
3. Trajectory Percentiles as an Effective Proxy: The authors identify that simple trajectory percentiles serve as a natural and effective criteria for distinguishing policy behaviors, working synergistically with temporal scheduling to avoid the entropy collapse seen in standard methods.
4. Empirical Validation: Demonstrated consistent improvements across multiple model scales (Qwen3-4B, Qwen3-8B), RL algorithms (GRPO, GSPO), and benchmarks (mathematical and general reasoning).

Experimental Results

The authors evaluated their method on Qwen3-4B and Qwen3-8B models using datasets from OpenMathReasoning and DeepMath-103K. Benchmarks included in-distribution math tasks (AIME, HMMT, Minerva) and out-of-distribution reasoning tasks (Winogrande, GPQA, MuSR).

• Performance Gains:
  • TP-Schedule (using trajectory percentiles) outperformed vanilla GRPO by 2.2% on mathematical benchmarks and 2.7% on general reasoning benchmarks for the 4B model.
  • Applying temporal scheduling to existing stagnant methods (e.g., Entropy-Adv, Forking-Tok) yielded consistent improvements, with discrete scheduling (expanding token subsets) showing particularly strong potential.
  • Improvements were observed across different model sizes and RL algorithms (including GSPO).
• Training Dynamics:
  • Entropy Preservation: Standard optimization often sacrifices policy entropy to accommodate heterogeneous behaviors. Temporal scheduling preserved significantly higher policy entropy (over 33.9% higher in some comparisons) throughout training, leading to healthier dynamics.
  • Structured Alignment: KL divergence analysis showed that temporal scheduling induces a progressive shift in policy updates, starting from later trajectory percentiles and gradually propagating to earlier ones, rather than updating uniformly.

Significance and Claims

The paper claims that temporal scheduling constitutes a promising optimization dimension for RLVR. Its significance lies in:

• Shifting the Paradigm: Moving from a static "where" (token selection) focus to a dynamic "when" (scheduling) focus.
• Mitigating Entropy Collapse: By avoiding the simultaneous accommodation of all heterogeneous behaviors, the method prevents the rapid collapse of policy entropy often seen in standard RLVR.
• Simplicity and Compatibility: The approach relies on simple trajectory percentiles and is compatible with existing credit allocation strategies, requiring no complex new reward models or annotation costs.

The authors conclude that while the framework is currently applied to specific allocation criteria, future work involves orchestrating diverse credit allocation criteria within this temporal framework. The results suggest that the timing of learning signals is as crucial as their spatial distribution across tokens.