Technical Summary: When Domains Interact in GRPO for Reasoning

Problem Statement

While Group Relative Policy Optimization (GRPO) has become a cornerstone technique for enhancing reasoning capabilities in Large Language Models (LLMs), its behavior under multi-domain training remains poorly understood. Existing research has largely focused on single-domain applications (e.g., mathematics or code) or independent studies of specific domains (e.g., logic or puzzles). Consequently, the interactions between different reasoning domains during training—specifically how sequential versus mixed-domain strategies affect performance, transfer, and interference—are not systematically characterized. This paper addresses the gap in understanding how domain-specific signals interact, whether training order matters, and if a universal training strategy exists for multi-domain reasoning.

Methodology

The authors conduct a systematic empirical study using the Qwen3-4B-Base model trained with standard GRPO. The experimental framework involves four representative reasoning domains:

1. Mathematics: Sourced from Skywork-OR1-RL-Data.
2. Science: Sourced from OpenScienceReasoning-2 (STEM subset).
3. Logic: Sourced from the Knights-and-Knaves dataset.
4. Puzzle: Sourced from Enigmata-Data.

The study is organized into three analytical phases:

1. Single-Domain Generalization: Models are trained on one domain and evaluated on others to measure cross-domain transfer capabilities.
2. Cross-Domain Interaction: The authors investigate bidirectional effects by training models sequentially (e.g., Domain A $\rightarrow$ Domain B) and measuring performance changes on both domains compared to single-domain baselines. This quantifies "forgetting" (interference) and "facilitation" (transfer).
3. Multi-Domain Training Strategies: The paper compares sequential training (training on domains one after another in various orders) against mixed training (training on a combined dataset of multiple domains simultaneously).

Evaluation is performed on standard benchmarks: MATH500 for math, GPQA for science, and specific test splits for logic and puzzles.

Key Contributions and Results

1. Asymmetric Single-Domain Generalization

The study reveals a highly asymmetric transfer pattern:

• Mathematics: Exhibits strong cross-domain transfer. Training on science, logic, or puzzle data improves mathematical reasoning accuracy by approximately 18–25%.
• Science: Shows moderate transferability, benefiting from other domains but to a lesser extent than math.
• Logic and Puzzle: These domains are largely domain-specific. Training on other domains yields negligible gains (often <4%), and high performance in these areas is only achieved through in-domain training.

2. Order-Sensitive Cross-Domain Interactions

The sequence of training significantly impacts final performance, with interactions that are not reciprocal:

• Math $\leftrightarrow$ Science: The order is critical. Training Math $\rightarrow$ Science achieves high accuracy on both (83% Math / 41% Science). Reversing the order to Science $\rightarrow$ Math causes substantial degradation (77% Math / 25% Science).
• Logic $\leftrightarrow$ Science: Science training consistently interferes with logical reasoning, regardless of order. Training on science after logic causes a ~5% drop in logic accuracy; training on logic after science causes a ~7.6% drop.
• Puzzle: The puzzle domain is largely insensitive to interference from other domains.

3. No Universal Optimal Strategy

The paper demonstrates that no single training paradigm is optimal for all domains:

• Sequential Training: Favors Mathematics. The optimal sequence (e.g., Logic $\rightarrow$ Math $\rightarrow$ Science) can push Math accuracy to 84%, outperforming single-domain training. However, poor ordering (e.g., Science $\rightarrow$ Math $\rightarrow$ Logic) can result in average accuracies as low as 56%.
• Mixed Training: Favors Science, Logic, and Puzzle. For these domains, mixed training often outperforms the best sequential strategies and single-domain baselines. Specifically, mixed training on Logic and Science effectively mitigates the interference observed in sequential settings.

Significance and Claims

The paper claims that GRPO in multi-domain settings exhibits pronounced asymmetry, order sensitivity, and strategy dependence. The authors emphasize that:

• Training Order is Critical: Ignoring the sequence of domains can introduce substantial bias, leading to performance gaps of up to 28 percentage points (e.g., 84% vs. 56% average accuracy depending on the strategy).
• Domain-Aware Design is Necessary: A "one-size-fits-all" approach is suboptimal. Effective multi-domain training requires tailoring the strategy to the specific domains involved (e.g., using sequential training for Math-heavy goals and mixed training for Science/Logic goals).
• Foundation for Future Work: This study provides the first systematic investigation of these cross-domain effects, laying the groundwork for principled multi-domain reinforcement learning in reasoning models.

The authors conclude that understanding and explicitly managing these cross-domain interactions is essential for building robust, generalizable reasoning models as the field shifts from single-domain optimization to complex multi-domain training regimes.