Technical Summary: Transfer-Aware Curriculum for Multi-Domain RLVR

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has successfully enhanced the reasoning capabilities of Large Language Models (LLMs) in single domains like mathematics and coding. However, extending this to multi-domain reasoning suites (spanning math, programming, logic, simulation, tables, and STEM) presents a critical challenge: reasoning skills transfer unevenly across domains.

Current approaches to multi-domain training typically rely on fixed sampling schedules or hand-tuned curricula. While recent "learnability-based" curricula (e.g., SEC) adapt sampling based on which domain the policy is currently improving on (using on-policy advantages), they suffer from a fundamental blind spot: they do not account for cross-domain transferability. A domain may be highly learnable locally (providing strong immediate gradients) but its updates may not benefit, or may even hinder, performance on other domains. Conversely, a less obviously learnable domain might yield gradient updates that broadly improve the policy across the entire suite. Existing methods risk over-committing to locally rich but globally narrow domains, leaving significant cross-domain improvement potential untapped.

Methodology: Transfer-Aware Curriculum (TAC)

The authors propose Transfer-Aware Curriculum (TAC), an automated, bandit-style online curriculum that integrates cross-domain transferability as a first-class signal for domain selection. TAC operates within the Group Relative Policy Optimization (GRPO) framework and formulates domain selection as a multi-armed bandit problem.

Core Components

1. Bandit Formulation:

   • Each reasoning domain is treated as an "arm."
   • At each training step, the curriculum samples a domain based on a value estimate $Q_m$ derived from a feedback signal $S_m$.
   • The sampling distribution is updated using a Boltzmann distribution over Upper Confidence Bound (UCB) augmented values to balance exploration and exploitation.

2. Dual Feedback Signal ($S_m$):
   The feedback signal for each domain is a weighted combination of two terms:

   • Local Learnability ($\hat{L}_m$): Derived from the mean absolute GRPO advantage. This captures the immediate learning signal (active improvement) within the domain. It is normalized via a running z-score to handle scale variations.
   • Cross-Domain Transferability ($T_m$): A novel, self-supervised signal estimating how well a gradient step on domain $m$ aligns with the gradient directions of other domains.
     • Mechanism: Instead of computing full high-dimensional gradients, TAC uses projected gradients (via random projections, e.g., TRAK-style sketches) to represent the update direction in a low-dimensional space.
     • State Tracking: Each domain maintains an Exponential Moving Average (EMA) of its projected gradient vectors.
     • Alignment Metric: Periodically, the pairwise cosine similarities between a domain's gradient state and the states of all other active domains are computed. The transferability score $T_m$ is the mean cosine similarity, normalized via min-max scaling to a $[0, 1]$ range.
     • Efficiency: This signal is computed entirely from gradients already generated during standard RL training, requiring no extra rollouts, held-out probes, or oracle annotations. The overhead is negligible (<1% wall-clock time).

3. Signal Integration:
   The final score is $S_m = \beta \hat{L}_m + (1-\beta) T_m$. The coefficient $\beta$ balances the two signals. The method dynamically updates the value estimates for both sampled and unsampled arms (using cached learnability and fresh transferability estimates), allowing the curriculum to shift focus even when a domain is not currently being sampled.

Key Contributions

1. Transfer-Aware Bandit Curriculum: The first curriculum framework for multi-domain RL that explicitly optimizes for cross-domain transferability alongside local learnability. It addresses the limitation of prior methods that only consider "what the policy can learn now" rather than "what helps the policy learn everywhere."
2. Self-Supervised Gradient-Geometry Signal: A computationally efficient method to estimate transferability using projected gradient cosines. This signal is adaptive (evolving with the policy) and relative (ranking domains based on current alignment rather than absolute magnitude), avoiding the need for expensive external validation.
3. Empirical Validation: Demonstrated on a six-domain reasoning suite (Math, Codegen, Logic, Simulation, Table, STEM) using Qwen3-1.7B and Llama3.2-3B models.

Experimental Results

The paper evaluates TAC against four baselines:

• Base: No RL training.
• Random: Proportional random sampling (uniform or data-size weighted).
• M2O (Math-to-Others): A hand-designed two-stage curriculum (train on math, then others).
• SEC: A learnability-only bandit curriculum ($\beta=1$).

Key Findings:

• Superior Performance: TAC achieves the best macro-averaged accuracy on both model backbones. On Qwen3-1.7B, it improves over the strongest baseline (SEC) by +1.8 percentage points (+5.6% relative). On Llama3.2-3B, it improves over M2O by +1.6 percentage points (+5.4% relative).
• Robustness to Imbalance: In experiments with skewed data budgets (where Math and STEM have 3x the data of other domains), TAC outperforms SEC by +1.5 points. The learnability-only baseline (SEC) over-commits to the dominant, high-data domains (Math/STEM) even when their updates cease to benefit other domains, whereas TAC's transferability signal redirects sampling to under-represented but highly transferable domains (like Table and Logic).
• Curriculum Dynamics: Analysis shows that the most learnable domain (often STEM or Math) is frequently not the most transferable. SEC tends to "anchor" on the most learnable domain early in training, while TAC dynamically shifts sampling toward domains with high gradient alignment (e.g., Table), leading to broader generalization.
• Ablations: Removing the transferability term ($\beta=1$) causes a sharp performance drop, confirming its necessity. The method is robust to hyperparameter variations, particularly the mixing coefficient $\beta$, which peaks around 0.2 (indicating a strong need for transferability signals even when learnability is present).

Significance and Claims

The paper claims that cross-domain transferability is a key, previously underutilized signal for curriculum design in multi-domain RLVR.

• Paradigm Shift: The work moves beyond optimizing for immediate local gains (learnability) to optimizing for the geometric alignment of updates across the entire training suite.
• Efficiency: It demonstrates that high-quality curriculum signals can be derived "for free" from existing training gradients, making it a practical addition to existing RLVR pipelines without significant computational cost.
• Generalization: The results suggest that a single policy can reason competently across heterogeneous tasks if the training curriculum explicitly accounts for how skills transfer, rather than treating domains as independent silos or relying on static schedules.
• Limitations: The authors note that TAC targets the sampling distribution and is complementary to, not a substitute for, advances in optimizers, reward models, or architectures. Furthermore, the current implementation relies on verifiable rewards (RLVR); extending these signals to non-verifiable domains (relying on LLM judges) is left for future work.

In summary, TAC establishes that for multi-domain reasoning, knowing where the model is improving is insufficient; one must also know where those improvements will propagate.