Technical Summary: DARC (Decoupled Asymmetric Reasoning Curriculum)

1. Problem Statement

The paper addresses the instability inherent in self-play frameworks for Large Language Model (LLM) self-evolution. While self-play (a co-evolutionary loop between a Questioner that generates tasks and a Solver that solves them) is a promising paradigm for achieving self-improving AI without human intervention, existing coupled systems suffer from two critical issues:

1. Non-Stationary Optimization Targets: The Questioner is optimized against a moving target determined by the Solver's evolving performance. As the Solver improves, the Questioner's learned difficulty signal becomes stale or misaligned, leading to oscillatory shifts in task difficulty and unstable dynamics.
2. Bootstrapping Errors and Self-Confirmation Bias: The Solver is trained on pseudo-labels generated by the Questioner. These labels are inevitably noisy; errors propagate and amplify across iterations, causing performance collapse or premature plateaus.

Current methods often rely on tightly coupled co-evolution, which the authors argue is structurally prone to these instabilities.

2. Methodology: The DARC Framework

The authors propose DARC (Decoupled Asymmetric Reasoning Curriculum), a two-stage framework that restructures self-evolution into sequential, decoupled phases to stabilize training.

Stage 1: Questioner Training (Difficulty-Calibrated Generation)

Instead of optimizing the Questioner against a live Solver, DARC trains the Questioner using explicit difficulty levels and an external corpus.

• Input: A document $d$ from the corpus and a target difficulty scalar $\tau \in [0, 1]$.
• Policy: The Questioner $Q_\theta$ generates a question $q$ conditioned on $d$ and $\tau$.
• Optimization: The model is trained using Group Relative Policy Optimization (GRPO) with a reward function $r_Q(q)$ that enforces two constraints:
  1. Grounding: The question must be grounded in the source document $d$ (verified by an LLM-as-a-Judge). Ungrounded questions receive a negative reward.
  2. Difficulty Alignment: The empirical difficulty of the question, estimated by a fixed Solver's success rate, must match the target $\tau$.
• Outcome: This decouples question generation from the Solver's real-time performance, creating a stable curriculum of questions with known difficulty levels.

Stage 2: Solver Training (Asymmetric Self-Distillation)

The trained Questioner generates an offline dataset of questions. The Solver is then trained on this curriculum using an asymmetric self-distillation mechanism.

• Privileged Teacher: A "Teacher" Solver has access to both the question $q$ and the source document $d$. It generates multiple candidate answers, and a majority vote determines the pseudo-label $a^*$. Samples with low voting agreement are discarded to reduce noise.
• Student Solver: A "Student" Solver is trained to predict the answer $a^*$ using only the question $q$ (without access to $d$).
• Optimization: The Student is optimized via GRPO using a correctness reward $r_S(a) = \mathbb{I}[a = a^*]$.
• Significance of Asymmetry: By forcing the student to solve problems without the document, the framework prevents trivial copying and mitigates self-confirmation bias, as the student must learn reasoning patterns rather than memorizing document-content associations.

Theoretical Analysis

The paper provides a theoretical analysis (Theorem 1) demonstrating that in coupled self-play, a Solver update can reverse the gradient ascent direction of the Questioner's objective, leading to instability. In contrast, DARC's decoupled objective (matching a fixed difficulty $\tau$) is independent of the time-varying Solver, ensuring that gradient updates remain valid and stable.

3. Key Contributions

1. Decoupled Architecture: The paper introduces a two-stage framework that separates question generation from solver training, eliminating the non-stationary target problem found in coupled self-play.
2. Asymmetric Self-Distillation: A novel training scheme where a document-augmented teacher generates high-quality pseudo-labels for a document-free student, reducing label noise and confirmation bias.
3. Difficulty-Calibrated Curriculum: The ability to explicitly control the difficulty of generated questions, enabling a structured, progressive learning path (Easy $\to$ Medium $\to$ Hard).
4. Model Agnosticism: The framework is designed to be independent of the specific backbone model, allowing the curriculum to generalize across different architectures.

4. Experimental Results

The authors evaluated DARC on nine reasoning benchmarks (mathematical and general) using three backbone models (Qwen3-4B/8B and OctoThinker-8B).

• Performance Gains: DARC yielded an average improvement of 10.9 points across all benchmarks and backbones compared to base models.
• Comparison to Baselines:
  • DARC outperformed all label-free self-evolving baselines (R-Zero, Absolute-Zero) and corpus-grounded methods (SPICE).
  • It remained competitive with weakly supervised methods (R-Few) that utilize 1% human-labeled data, despite DARC using no human annotations.
  • On the Qwen3-8B backbone, DARC approached the performance of General-Reasoner, a model trained on 232K human-annotated samples.
• Stability: Unlike previous self-play systems that exhibited performance collapse or oscillatory behavior, DARC demonstrated steady validation reward improvements without degradation.
• Generalization:
  • Cross-Solver: A Questioner trained on one backbone (Qwen3-4B) successfully generated a curriculum that improved the performance of both smaller (1.7B) and larger (8B) Solver backbones.
  • Beyond Memorization: DARC outperformed standard fine-tuning (Vanilla FT) on the same corpus, suggesting it induces transferable reasoning abilities rather than mere pattern memorization.

5. Significance and Claims

The paper claims that DARC represents a significant step toward reliable and efficient self-evolution mechanisms that operate independently of human data. By decoupling the generation of tasks from the learning of solutions, the framework resolves the structural instability of coupled self-play.

The authors emphasize that DARC enables models to autonomously refine their capabilities, approaching the performance of fully supervised models without the bottleneck of human annotation scarcity. The work suggests that with sufficient base model scale, self-evolution via a decoupled curriculum can achieve competitive reasoning performance.

Limitations Acknowledged:
The paper notes that DARC still relies on an external corpus (not fully data-free), that pseudo-labels remain inherently noisy, and that the current framework is primarily designed for domains with verifiable answers, limiting its immediate applicability to open-ended tasks.