Technical Summary: Internalizing Curriculum Judgment for LLM Reinforcement Fine-Tuning

Problem Statement

In Large Language Model (LLM) Reinforcement Fine-Tuning (RFT), learning efficiency and final performance are heavily dependent on curriculum learning—the strategy of selecting which prompts to train on. Effective learning requires focusing on problems near the "zone of proximal development," where the policy is neither consistently succeeding nor consistently failing. However, current curriculum methods suffer from a critical misalignment: they externalize the judgment of prompt informativeness.

Existing approaches rely on:

1. Handcrafted heuristics: Static rules based on offline difficulty or domain categories.
2. Auxiliary models: Separate learned models (scorers) that estimate prompt difficulty.
3. Post-hoc filtering: Selecting prompts after rollouts based on statistics.

These methods treat the policy as a passive recipient of an externally defined curriculum. Because the policy's competence frontier shifts dynamically as it learns, external selectors often drift out of alignment with the model's current state, leading to suboptimal compute allocation. Furthermore, many existing heuristics (e.g., pass-rate thresholds) are restricted to binary rewards and fail to generalize to continuous reward settings like code generation or agentic tasks.

Methodology: METIS

The authors propose METIS (METacognitive Internalized Self-judgment), a framework that internalizes curriculum judgment as a native capability of the policy itself. METIS reframes curriculum learning not as data selection, but as training the model to decide what to learn next.

Core Mechanism

METIS operates on the observation that within-prompt reward variance is a general, policy-dependent signal of prompt informativeness.

• Low Variance: The policy consistently solves or consistently fails a prompt (low learning signal).
• High Variance: The policy produces diverse outcomes for the same prompt, indicating the prompt lies at the competence frontier (high learning signal).

METIS implements a two-fold design to internalize this judgment:

1. In-Context Self-Judgment (Pre-Rollout):
   Instead of using an external scorer, the policy $\pi_\theta$ itself predicts the expected reward variance for candidate prompts. It does this via In-Context Learning (ICL).

   • The policy is provided with a "calibration memory" ($\mathcal{M}_t$) containing recent pairs of prompts and their realized reward variances from the current training history.
   • For each candidate prompt, the policy generates a structured response predicting the variance $\hat{v}_\theta(x)$.
   • Prompts are ranked by these predictions, and the top-$B$ most informative prompts are selected for rollout.

2. Joint Optimization (Post-Rollout):
   METIS closes the loop by jointly optimizing two objectives within the same policy:

   • Task Loss ($\mathcal{L}_{policy}$): Standard group-relative RFT loss (e.g., GRPO) to improve problem-solving capabilities.
   • Self-Judgment Loss ($\mathcal{L}_{judge}$): A calibration reward that penalizes the difference between the predicted variance $\hat{v}_\theta(x)$ and the realized variance $v(x)$ obtained from the rollouts.

   The total objective is $\mathcal{L}_{total} = \mathcal{L}_{policy} + \lambda \mathcal{L}_{judge}$. This forces the policy to not only solve tasks but also accurately assess its own learning state, effectively learning a metacognitive skill.

Generalization

Unlike prior work restricted to binary rewards (pass/fail), METIS formalizes informativeness using the variance of rewards $v_\theta(x) = \text{Var}_{y \sim \pi_\theta} [r(x, y)]$. This formulation naturally extends to continuous rewards (e.g., partial credit in code generation, trajectory scores in agentic tasks) without requiring task-specific difficulty definitions.

Key Contributions

1. Problem Formulation: The authors formalize curriculum learning for LLM RFT as a policy-dependent competence-frontier tracking problem, identifying within-prompt reward variance as a unified signal for both discrete and continuous rewards.
2. METIS Framework: They introduce the first LLM RFT framework to internalize curriculum judgment. By jointly optimizing task performance and self-judgment accuracy, METIS transforms curriculum selection from an external mechanism into a learned capability of the policy.
3. Empirical Validation: METIS is validated across diverse benchmarks (mathematical reasoning, code generation, agentic function-calling) and model backbones (Qwen, Llama, DeepSeek).

Experimental Results

The paper reports extensive experiments on discrete (math) and continuous (code, agentic) reward tasks.

• Performance: METIS consistently achieves superior final accuracy compared to baselines (No Curriculum, Category Scheduling, Handcrafted Heuristics, and Auxiliary Model selectors).
• Efficiency: METIS accelerates convergence significantly, achieving up to 67% reduction in wall-clock time to reach peak performance compared to uniform sampling.
• Overhead: The method introduces minimal computational overhead (approx. 3.9% per-step) because the self-judgment is a short ICL prediction, which is far cheaper than generating full task rollouts.
• Convergence Dynamics: METIS maintains a higher mean group-relative advantage throughout training, indicating it successfully concentrates compute on prompts with the highest learning signal.
• Ablation Studies:
  • Removing in-context examples ($K=0$) causes the policy to fail to parse the task, treating prompts as standard problems rather than variance predictions.
  • Removing the judgment loss ($\lambda=0$) results in uncalibrated predictions that do not track the moving frontier as effectively, though ICL alone still provides some benefit.
  • The method works effectively on continuous reward tasks where pass-rate-based baselines fail or require ad-hoc extensions.

Significance and Claims

The paper claims that METIS establishes a simple, closed-loop, and highly efficient paradigm for LLM reinforcement fine-tuning. By bypassing handcrafted heuristics and auxiliary models, METIS demonstrates that an LLM can learn to monitor its own learning state and select its own curriculum.

The authors position this as a shift from "selecting data" to "training the model to decide what to learn." They note that while METIS relies on the base policy's in-context learning capacity, it removes the need for external tuning or separate selector models. The work concludes that prompt-level curriculum judgment can be reliably absorbed into the policy itself, offering a metacognitive framework that improves both the efficiency and effectiveness of RFT across diverse domains.

Limitations Acknowledged: The authors modestly note that METIS depends on the base policy's ICL capabilities, which may vary across weaker backbones. They also state that while training dynamics appear stable, they do not formally quantify training stability in this work.