Technical Summary: Agon – Competitive Cross-Model RL with Implicit Rival Grading

1. Problem Statement

Current Reinforcement Learning from Verifiable Rewards (RLVR), particularly methods like Group Relative Policy Optimization (GRPO), relies solely on outcome-based rewards (i.e., correctness of the final answer). While effective for tasks with clear ground truth, this approach fails to grade the reasoning process (the "trace") itself. On difficult problems, this creates a strong incentive for models to increase the quantity of reasoning (length) rather than the quality. Models learn to "write more" by adding hedging, backtracking, and redundant attempts to increase the probability of stumbling upon the correct answer, leading to a "length pathology" where token count grows significantly while accuracy improves only marginally.

Directly grading reasoning is intractable because there is no ground truth for "good thinking" (no labels for specific insightful steps), and learned process reward models are expensive, brittle, and unverifiable. The paper posits that a second model can supply the missing signal: if a competing policy attempts the same problem and is rewarded for out-reasoning the first, the reasoning trace is graded implicitly without explicit process labels.

2. Methodology: Agon

Agon (Greek for "contest") is a training framework that jointly optimizes two distinct policies, $A$ and $B$, in a competitive head-to-head setting.

Core Mechanism

The training process operates in alternating roles (draft-and-challenge) with per-step rotation:

1. Drafting: One model (the drafter) generates $N$ rollouts from a plain problem prompt. It receives a standard GRPO update based solely on the correctness of its final answer.
2. Challenging: The other model (the challenger) reads a summary of the drafter's solution (the post-reasoning recap, excluding the raw trace and the final answer). The challenger then generates a new rollout attempting to solve the problem, potentially correcting or improving upon the drafter's work.
3. Reward: The challenger receives a competitive reward that depends on both its own correctness and the drafter's outcome.
   • Correctness: Standard binary reward for a correct answer.
   • Conversion Bonus: An additional reward is granted if the challenger solves the problem correctly and the drafter failed ($c(b_i) \cdot (1 - c(a_i))$). This incentivizes the challenger to identify and fix the drafter's errors.
   • Optional Length Tiebreak: A bonus can be applied if the challenger is correct and uses fewer tokens than a correct drafter, encouraging brevity without penalizing correctness.

Architecture and Efficiency

• Dual-Adapter Instantiation: To maintain efficiency, both policies are instantiated as Low-Rank Adaptation (LoRA) adapters over a single frozen base model. This keeps memory overhead low (~2% of the base model) and ensures the policies operate in compatible representation spaces.
• Divergence: The two adapters are initialized differently (one with standard zero-init, the other with small Gaussian noise) and receive different gradient streams due to role rotation. This structural divergence ensures they develop different "blind spots," preventing the system from collapsing into self-play or simple distillation.
• Inference: At deployment, the pair operates as a two-stage cascade: one model drafts a solution, and the second reads the summary to produce the final answer.

Theoretical Justification

The paper argues that single-model self-improvement (self-play) often plateaus because the model re-audits its own work using the same biases that caused the errors. By introducing a second model with different failure modes, Agon breaks this closed loop. The "grader" is a distinct agent, and because both models are optimized, the difficulty of the "rival" increases over time, pushing both policies beyond the ceiling of single-model RL.

3. Key Contributions

1. Reframing Reasoning Quality: The paper reframes reasoning quality as an implicit reward supplied by a competing peer, transforming the unlabeled "good thinking" problem into a head-to-head game.
2. The Agon Recipe: A novel training recipe using competitive GRPO with draft-and-challenge rollouts, per-step role rotation, and a competitive reward structure (correctness + conversion bonus). It runs on standard GRPO trainers with no core optimization changes.
3. Efficient Implementation: A shared-base, dual-adapter instantiation that makes training two competing policies nearly as cheap as training a single model.
4. Empirical Validation: A 2x2 experimental design (crossing competition vs. cooperation and information exchange vs. no exchange) that isolates the value of adversarial pressure and cross-model information.

4. Experimental Results

Experiments were conducted on the DeepMath-hard split (using Qwen3 models) and a competitive programming subset of CodeContests.

• Performance Gains: On the DeepMath-hard split with Qwen3-0.6B:
  • Vanilla GRPO: 30% pass@1.
  • Cooperative Exchange (no competition): 46% pass@1.
  • Agon (Competition + Exchange): 61% pass@1.
  • This represents a doubling of GRPO's performance and an ~8x gain over an untrained Mixture-of-Agents (MoA) baseline.
• Scaling: The gains persist across model families (Qwen3.5, Gemma 4) and sizes. Notably, the Agon cascade on a 0.6B base (61%) outperformed a vanilla GRPO-trained 4B base (59%).
• Trace Length: Agon significantly reduces reasoning length. The final-stage trace averaged 3.5k tokens compared to 8.1k tokens for vanilla GRPO, while achieving higher accuracy. This shortening is an emergent property of the competition, not an explicit penalty.
• Ablations:
  • Competition: Removing the competitive bonus (cooperative only) drops performance to 46%.
  • Exchange: Removing visible information exchange (shared opponent hidden from context) drops performance to 32% (near vanilla GRPO), confirming that the competitive gradient requires visible context variance to function.
  • Role Rotation: Fixing roles (no rotation) reduces performance to 52%, highlighting the importance of both models learning both drafting and challenging.

5. Significance and Claims

The paper claims that Agon demonstrates that cross-model information exchange and adversarial pressure can significantly improve reasoning capabilities without requiring process labels or a separate reward model.

• Implicit Grading: It proves that a rival model can effectively grade reasoning traces implicitly, solving the "unlabeled thinking" problem.
• Beyond Self-Play: It establishes that a pair of divergent policies can co-improve past the plateau of single-model self-refinement.
• Efficiency: It shows that high-performance reasoning can be achieved with a two-stage cascade that is computationally efficient (using adapters) and produces shorter, denser reasoning traces.
• Modesty: The authors note that the current implementation relies on text exchange, which bounds bandwidth. They suggest future work could move to latent-space exchange (e.g., KV-cache injection) to remove serialization bottlenecks. They also acknowledge that the method requires a clean verifier and that the divergence between models is maintained heuristically rather than guaranteed.

In summary, Agon shifts the paradigm from "grading the answer" to "grading the reasoning via competition," enabling models to learn to think better, not just write more.