Technical Summary: Efficient Reinforcement Finetuning via Adaptive Curriculum Learning

Problem Statement

Reinforcement Finetuning (RFT) has emerged as a critical technique for aligning Large Language Models (LLMs) with task-specific goals, particularly in domains like mathematical reasoning where correctness is well-defined. However, standard RFT approaches (e.g., PPO, GRPO) suffer from significant sample and computational inefficiency. They often require extensive training involving repeated rollout generation and policy updates, making them costly and difficult to scale.

Existing attempts to improve efficiency face specific trade-offs:

• Algorithmic simplifications (e.g., RAFT, ReMax) may increase variance or limit stability.
• Data-centric strategies (e.g., LIMO, LIMR) often rely on manual curation or static scoring, which can be brittle, computationally heavy, or model-specific.
• Curriculum Learning (CL) approaches in prior work typically use fixed, manually defined stages or online filtering schemes that require repeated rollouts to prune data. These methods lack true adaptivity to the model's evolving skill level or incur significant computational overhead.

Consequently, there is a need for a method that dynamically adjusts training difficulty to match the model's current capabilities without modifying the underlying RL algorithm or requiring extensive manual data curation.

Methodology: ADARFT

The authors propose ADARFT (Adaptive Curriculum Reinforcement Finetuning), a lightweight, model-agnostic framework that integrates adaptive curriculum learning into standard RFT pipelines (compatible with PPO, GRPO, REINFORCE++).

Core Mechanism

ADARFT operates by maintaining a target difficulty level ($T$) and dynamically adjusting it based on the model's recent reward signals. The core intuition is that learning is most effective when tasks are neither too easy nor too hard (the "zone of proximal development").

The algorithm proceeds as follows:

1. Offline Difficulty Estimation: Each problem in the dataset $D$ is pre-annotated with a difficulty score $d_i$. These scores can be derived from human annotations, model-based rollouts (pass@k), or LLM judges (e.g., GPT-4o).
2. Dynamic Curriculum Sampling: At each training step, the algorithm computes the absolute difference $\Delta_i = |d_i - T|$ between the target difficulty and every problem's difficulty. It selects a batch of $B$ problems with the smallest $\Delta_i$, ensuring the model trains on tasks closest to its current capability.
3. Policy Update: The selected batch is used to update the policy $\pi_\theta$ using a standard RL algorithm (e.g., PPO). A binary reward signal ($R_i \in \{0, 1\}$) is computed based on the correctness of the response.
4. Target Difficulty Update: The target difficulty $T$ is updated based on the average reward ($R_{avg}$) of the batch. The update rule is defined as:
   $$T' = \text{clip}(T + \eta \cdot \tanh(\alpha \cdot (R_{avg} - \beta)), d_{min}, d_{max})$$
   • $\beta$: The target reward level (set to 0.5), representing the desired success rate.
   • $\eta$: Step size for adjusting difficulty.
   • $\alpha$: Sensitivity parameter.
   • $\text{clip}$: Constrains $T$ within the valid difficulty range $[d_{min}, d_{max}]$.

If the model performs well ($R_{avg} > \beta$), $T$ increases, making subsequent problems harder. If performance is poor ($R_{avg} < \beta$), $T$ decreases.

Theoretical Justification

The authors provide a theoretical justification for setting the target reward $\beta = 0.5$. In entropy-regularized RL, the gradient magnitude for policy updates is proportional to the reward variance. For binary rewards, variance is maximized when the success rate is 0.5. Therefore, training on problems where the model succeeds roughly half the time provides the strongest learning signal, avoiding the vanishing gradients associated with trivially easy tasks or the noise of prohibitively hard ones.

Key Contributions

1. Adaptive Curriculum Framework: ADARFT introduces a dynamic, feedback-driven curriculum that adjusts task difficulty in real-time based on model performance, eliminating the need for fixed schedules or repeated rollouts for data filtering.
2. Lightweight and General: The method requires only a minimal extension to standard RL algorithms (PPO, GRPO, etc.) and does not alter the reward function or model architecture. It is compatible with various RL algorithms and model sizes.
3. Offline Difficulty Estimation: The approach relies on static, pre-computed difficulty scores (derived from rollouts or LLM judges), avoiding the computational overhead of recomputing difficulty online during training.
4. Comprehensive Evaluation: The method is evaluated across diverse data distributions (uniform, skew-easy, skew-difficult) and model scales (Qwen 2.5 1.5B and 7B) on competition-level math benchmarks.

Results

Experiments on competition-level math datasets (AMC, AIME, IMO-style) demonstrate that ADARFT significantly improves both training efficiency and final reasoning performance compared to standard PPO, PPO with filtered data, and PPO with fixed curriculum schedules.

• Training Efficiency: ADARFT reduces training time by up to 2×. Models trained with ADARFT require significantly fewer steps to reach the performance levels of baseline methods. For instance, on the Qwen 2.5 1.5B model with a skew-difficult distribution, standard PPO required 71.7% more steps to match ADARFT's performance at step 60.
• Final Accuracy: ADARFT consistently achieves higher final accuracy. On the skew-difficult distribution, the Qwen 2.5 1.5B model improved from 37.41% (PPO) to 40.48% (ADARFT). Similar gains were observed for the 7B model (44.17% to 46.83%).
• Robustness: The method is particularly effective in imbalanced data regimes (e.g., skew-difficult), where static sampling often fails to provide sufficient learning signals early in training.
• Algorithm Agnosticism: Ablation studies confirm that ADARFT improves convergence and accuracy when instantiated with GRPO and REINFORCE++, not just PPO.
• Difficulty Estimation: The paper shows that difficulty estimates derived from LLM judges (GPT-4o) correlate well with empirical success rates, offering a scalable alternative to expensive rollout-based estimation.

Significance and Claims

The paper claims that ADARFT offers a practical and scalable path to more efficient RFT in structured reasoning tasks. Its significance lies in:

• Eliminating Manual Curation: Unlike methods requiring hand-designed curricula or extensive data filtering, ADARFT automatically adapts to the model's learning dynamics.
• Optimizing the Learning Signal: By targeting a 50% success rate, the method ensures the model is consistently trained on "solvable but challenging" tasks, maximizing the information gain per update.
• Scalability: The approach is lightweight and model-agnostic, making it suitable for large-scale training pipelines where computational resources are constrained.

The authors note that the benefits of curriculum learning are most pronounced in imbalanced training distributions and for models with limited capacity, where standard RFT struggles to bootstrap capabilities. While manual curation could theoretically achieve similar results, ADARFT provides an automated, unified solution that generalizes across model architectures and skill levels without requiring re-tuning for each new model.