Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training

1. Problem Statement

Reinforcement Learning (RL) post-training of Large Language Models (LLMs) relies heavily on selecting training problems from massive, heterogeneous datasets. The choice of data significantly impacts training stability, sample efficiency, and final performance.

• Challenges:
  • Scale & Diversity: Datasets are too large for manual curation or tabular statistics.
  • Non-Stationarity: The utility of a problem changes as the policy (actor) improves; a problem that is "hard" for a weak model may be "easy" for a strong one.
  • Partial Feedback: The system only observes rewards for selected problems, not the entire dataset, creating a classic exploration-exploitation trade-off.
  • Brittleness of Existing Methods: Traditional curriculum learning relies on manual difficulty labels, static buckets, or heuristic signals (e.g., absolute mean advantage), which fail to generalize or adapt to evolving policy dynamics.

The core problem is to design an automated, adaptive data selection strategy that dynamically chooses which problems an LLM should train on at each iteration to maximize overall post-training performance without human intervention.

2. Methodology: ACTOR-CURATOR

The authors propose ACTOR-CURATOR (AC), a fully automated framework that jointly trains an Actor (the LLM being optimized) and a Curator (a neural network that selects training data).

Core Concept

The framework treats problem selection as a non-stationary stochastic bandit problem. The Curator learns to assign probabilities to problems based on their expected contribution to policy improvement, rather than static difficulty metrics.

Key Components

1. Policy-Improvement Learning Signal:

   • Instead of using heuristics like "absolute advantage," AC derives a principled objective based on the Performance Difference Identity (Kakade & Langford, 2002).
   • The utility of a problem $x$ is defined as its marginal contribution to the expected reward under the evaluation distribution:
     $$u_x^t = p_X(x) \mathbb{E}_{y \sim \pi_t} \left[ \frac{\pi_{t+1}(y|x)}{\pi_t(y|x)} A_{\pi_t}(y|x) \right]$$
   • This signal is actor-aware, meaning it directly measures how much a specific problem helps the current policy improve.

2. Online Stochastic Mirror Descent (OSMD):

   • The Curator is optimized using OSMD, a bandit algorithm designed for non-stationary environments with partial feedback.
   • Objective: Minimize regret against the best sequence of problem-selection distributions in hindsight.
   • Update Rule: The Curator updates its distribution $p_t$ based on bandit feedback (estimated utility) using a negative-entropy regularizer:
     $$p_{t+1} \leftarrow \arg \min_{p} \{ -\eta \langle p, \hat{U}_t \rangle + \text{KL}(p \| p_t) \}$$

3. Two-Stage Sampling for Scalability:

   • Directly sampling from a distribution over millions of problems is infeasible.
   • Stage 1: A candidate batch $\tilde{X}_t$ is sampled from a fixed proposal distribution (e.g., uniform).
   • Stage 2: The Curator reweights this smaller candidate batch to select the final training batch $X_t$.
   • An unbiased estimator is used to correct for the two-stage sampling probabilities, ensuring the Curator learns from the true utility of problems.

4. Stabilization (PPO-style Clipping):

   • To prevent unstable updates when training the neural Curator, the authors introduce a clipped surrogate objective (similar to PPO). This constrains the importance ratio between the new and old Curator policies, ensuring stable online learning.

Training Loop

1. Selection: The Curator selects a training batch from a candidate pool.
2. Rollout: The Actor generates solutions for these problems.
3. Update: The Actor is updated (e.g., via GSPO or GRPO).
4. Feedback: The system estimates the per-problem policy improvement ($\hat{U}_x$) using the pre- and post-update policies.
5. Curator Update: The Curator is updated using the OSMD objective with the estimated bandit feedback.

3. Key Contributions

1. Automated Framework: Introduces the first fully automated, scalable curriculum learning framework for LLM post-training that requires no human annotations, difficulty labels, or manual dataset structuring.
2. Theoretical Formulation: Formulates data curation as a non-stationary stochastic bandit problem with partial feedback. It derives a learning signal grounded in policy improvement theory and provides regret guarantees ($O(T^{2/3}V^{1/3})$) under non-stationarity.
3. Scalable Architecture: Combines neural function approximation with OSMD and two-stage sampling to handle datasets with $>10,000$ problems, overcoming the limitations of tabular methods.
4. Co-Adaptive Learning: The Actor and Curator are trained jointly in an on-policy manner, allowing the curriculum to evolve dynamically as the Actor's capabilities improve.

4. Experimental Results

The authors evaluated ACTOR-CURATOR on five reasoning benchmarks: Countdown, Zebra, ARC-1D, MATH500, and AIME24.

• Performance Gains:
  • Outperformed uniform sampling and strong baselines (SEC, PCL) across all benchmarks.
  • ARC-1D: Achieved a 30.5% relative improvement over the strongest baseline.
  • AIME24: Achieved a 28.6% relative improvement.
  • Hard Subsets: Gains were most pronounced on difficult subsets (e.g., +58.97% on ARC-Hard), demonstrating effectiveness in challenging regimes.
• Efficiency:
  • Achieved up to 80% speedup in convergence time compared to uniform sampling.
  • Reached comparable or higher peak performance with significantly fewer training steps.
• Stability:
  • Exhibited more stable optimization dynamics, avoiding the plateaus often seen in baseline methods.
  • Showed larger and more sustained actor gradient norms, indicating more informative learning signals.
• Ablation Studies:
  • Confirmed that both the policy-improvement signal and the OSMD bandit objective are critical; replacing them with regression losses or absolute advantage metrics significantly degraded performance.
  • Demonstrated robustness across different actor update algorithms (GRPO, GSPO) and model sizes.

5. Significance and Impact

• Paradigm Shift: Moves away from treating training data as a static resource. Instead, it treats data selection as an optimizable component of the learning process that adapts in tandem with the policy.
• Scalability: Provides a practical solution for scaling RL post-training to massive, heterogeneous datasets without the bottleneck of manual curation.
• Theoretical Rigor: Bridges the gap between theoretical bandit algorithms and practical deep RL, offering regret guarantees in a non-stationary, function-approximation setting.
• Generalizability: The framework is agnostic to the specific actor update rule, making it a versatile foundation for future RL post-training pipelines in mathematics, logic, code, and other domains with verifiable rewards.

In summary, ACTOR-CURATOR demonstrates that learning to curate data via policy-improvement-driven bandits is a superior strategy for scaling RL post-training, offering significant gains in both final performance and training efficiency.