IRIS: Interleaved Reinforcement with Incremental Staged Curriculum for Cross-Lingual Mathematical Reasoning

1. Problem Statement

Large Language Models (LLMs) struggle with complex mathematical reasoning, particularly in multilingual and low-resource settings (e.g., Indian languages like Hindi and Marathi). Existing training paradigms face two main limitations:

• Reward Sparsity: Training that rewards only the final answer fails to supervise the intermediate reasoning steps, leading to hallucinated logic chains.
• Curriculum Gaps: Standard curriculum learning (progressing from easy to hard problems) often fails to capture where reasoning breaks down. Conversely, step-level reinforcement methods (like Reverse Curriculum) often lack difficulty scheduling, leading to unstable learning on sparse rewards.
• Cross-Lingual Transfer: Low-resource languages often lack sufficient data to independently sustain high-quality reasoning, and direct translation of reasoning traces often degrades coherence.

2. Methodology: The IRIS Framework

The authors propose IRIS (Interleaved Reinforcement with Incremental Staged Curriculum), a two-axis training framework designed to decouple skill acquisition from reasoning refinement.

A. The Two-Axis Curriculum

1. Vertical Axis (Problem-Level Curriculum):

   • Goal: Gradually increase task difficulty.
   • Process: The model undergoes Supervised Fine-Tuning (SFT) in stages: Easy $\to$ Easy+Medium $\to$ Easy+Medium+Hard.
   • Mechanism: The model learns full reasoning traces for problems of increasing complexity, establishing a baseline of reasoning patterns before reinforcement learning begins.

2. Horizontal Axis (Step-Level Continuation Curriculum):

   • Goal: Reduce reliance on provided reasoning prefixes and encourage independent planning.
   • Process: A Reverse Curriculum approach. For a problem with $n$ steps, the model is trained on $n-1$ stages.
     • Stage 0 (Easiest): The model is given $n-2$ steps and must generate only the final 2 steps.
     • Stage $k$: The model is given $n-k-2$ steps and must generate the remainder.
     • Stage $n-2$ (Hardest): The model is given only the question and must generate the full solution from scratch.
   • Mechanism: This forces the model to transition from completing partial chains to generating full reasoning trajectories.

B. Composite Reward Design (RLVR)

To optimize the Horizontal Axis, the authors use Group Relative Policy Optimization (GRPO) with a novel composite reward function $R_k$:
$$R_k = r_{final} + \lambda_k \cdot r_{cos} + r_{cont} + r_{int}$$

• Correctness ($r_{final}$): Binary reward (+2) for exact answer match.
• Cosine Alignment ($r_{cos}$): Measures semantic similarity between the generated suffix and the ground-truth reasoning steps. Crucially, this weight $\lambda_k$ decays linearly as the curriculum stage increases. It acts as a stabilizer in early stages but is eliminated in hard stages to encourage diverse, valid reasoning paths.
• Continuity ($r_{cont}$): Rewards correct step numbering and logical flow (e.g., continuing from where the prefix left off).
• Integer Incentive ($r_{int}$): Small bonus for producing numeric outputs.

C. Multilingual Strategy: English as a Reasoning Anchor

The framework introduces a bilingual training setup (e.g., English + Hindi).

• Hypothesis: English acts as a "reasoning anchor." Since English has high-quality reasoning data, it stabilizes the learning dynamic.
• Mechanism: The model learns the logic of reasoning from English and transfers this capability to low-resource languages (Hindi/Marathi), which focus primarily on expressing that logic in a new script. Experiments show that even a 20% English mix significantly prevents reward stagnation in monolingual low-resource training.

3. Key Contributions

1. IRIS Framework: A novel two-axis curriculum combining difficulty-based SFT (Vertical) with reverse-step RL (Horizontal) to stabilize long-horizon reasoning.
2. Composite Reward Function: A multi-component reward signal that balances answer correctness with step-wise alignment and structural continuity, optimized via GRPO.
3. CL-Math Dataset: A new 29k-example multilingual dataset containing math problems with step-level annotations in English, Hindi, and Marathi. It includes 2,184 easy, 2,212 medium, and 5,588 hard samples per language, with strict train/test splits to prevent data leakage.
4. Cross-Lingual Insights: Empirical evidence that bilingual training (English + Low-Resource) acts as a necessary regularizer, enabling stable RL post-training where monolingual approaches fail.

4. Results

The authors evaluated IRIS on the Qwen2.5-Math-7B and WizardMath-7B models.

• Performance on CL-Math:
  • IRIS improved reasoning accuracy by up to +15.8% on the hardest Hindi problems compared to baselines.
  • Bilingual settings (English-Hindi/Marathi) showed substantial gains over monolingual settings, confirming the "reasoning anchor" hypothesis.
• Generalization to Standard Benchmarks:
  • SVAMP: 90.6% (vs. 70.4% base).
  • GSM8K: 83.9% (vs. 70.6% base).
  • MATH: 64.6% (vs. 29.5% base).
  • IRIS outperformed other specialized 7B models (MetaMath, DeepSeek-Math) and approached GPT-4 performance on specific benchmarks.
• Ablation Studies:
  • Removing the vertical SFT warm-up or the curriculum progression significantly degraded performance.
  • Simplifying the reward to correctness-only reduced accuracy, highlighting the importance of the composite reward structure.
  • A 20% English mix in Marathi training yielded the best balance, acting as a regularizer.

5. Significance and Impact

• Bridging the Low-Resource Gap: IRIS demonstrates a practical pathway to enhance mathematical reasoning in underrepresented languages (Hindi, Marathi) by leveraging high-resource English data without requiring massive native datasets.
• Stable RL for Reasoning: The combination of staged curricula and decaying alignment rewards solves the instability issues common in Reinforcement Learning for reasoning tasks, allowing models to learn complex, multi-step logic without collapsing.
• Open Science: The release of CL-Math and the codebase provides a critical resource for the community to further research multilingual reasoning and curriculum learning.
• Cognitive Alignment: The framework mirrors human cognitive development (learning structure first, then scaling complexity), offering a more robust training paradigm than end-to-end black-box optimization.

In conclusion, IRIS establishes that structured curriculum design combined with interleaved reinforcement is a highly effective strategy for unlocking deep reasoning capabilities in small language models, particularly in multilingual contexts.