💬 NLP

OpenSIR: Open-Ended Self-Improving Reasoner

OpenSIR is a self-play framework that enables large language models to continuously improve their reasoning capabilities by alternately acting as teacher and student to generate and solve novel problems without external annotations, achieving consistent performance gains across math and general reasoning benchmarks.

Original authors: Wai-Chung Kwan, Aryo Pradipta Gema, Joshua Ong Jun Leang, Pasquale Minervini

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wai-Chung Kwan, Aryo Pradipta Gema, Joshua Ong Jun Leang, Pasquale Minervini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant student who has already memorized every math textbook in the library. They are great at solving standard problems, but they've hit a ceiling. They can't get any better because they've stopped learning new things; they just keep practicing the same old exercises.

This is the problem the paper OpenSIR tries to solve.

The Problem: The "Echo Chamber" of Learning

Current AI models (like the ones powering DeepSeek-R1 or OpenAI's o1) are usually trained using Reinforcement Learning with Verifiable Rewards (RLVR). Think of this like a student taking a standardized test where a human teacher grades every answer.

The Bottleneck: You need thousands of human-annotated examples (teachers grading papers) to do this.
The Limit: Once the student has mastered the human-curated tests, they stop improving. They can't go beyond "human level" because they are only learning from what humans have already written down.

Some researchers tried Self-Play, where the AI plays against itself to learn. Imagine a chess player playing against a clone of themselves.

The Failure: Previous self-play methods failed with advanced models. Why? Because the AI kept generating the same types of problems it already knew. It was like a student who only practices multiplication tables over and over, never trying algebra or calculus. They got stuck in a loop of "familiar concepts."

The Solution: OpenSIR (The "Curious Teacher-Student")

The authors introduce OpenSIR (Open-Ended Self-Improving Reasoner). It's a framework where a single AI model plays two roles simultaneously: The Teacher and The Student.

Think of it as a person who is both the gym coach and the athlete, but with a special twist: they are constantly trying to invent new workouts that are just hard enough to be challenging, but not so hard that they are impossible.

Here is how the cycle works, step-by-step:

1. The Seed (The Spark)

You start with one incredibly simple problem, like "What is 1 + 1?" This is the only human input you need.

2. The Teacher's Job (Inventing New Challenges)

The AI (acting as the Teacher) looks at the simple seed and tries to invent a new math problem.

The Trap: Without guidance, the Teacher would just invent "What is 2 + 2?" and "What is 3 + 3?" over and over.
The Fix (Diversity Reward): OpenSIR gives the Teacher a "bonus point" for creating problems that are different from everything it has seen before. It's like a chef who gets a bonus for inventing a dish with ingredients they've never used together. This forces the AI to explore new mathematical concepts (like calculus, probability, or optimization) rather than repeating arithmetic.

3. The Student's Job (Solving the Challenge)

The AI (acting as the Student) tries to solve the new problem the Teacher invented.

The Trap: If the Teacher invents a problem that is too hard or broken (e.g., missing numbers), the Student can't solve it, and no learning happens.
The Fix (Difficulty Calibration): OpenSIR checks if the problem is "just right." It uses a "Solvability Score." If the problem is too easy, it's boring. If it's too hard or broken, it's useless. The system only rewards problems that are challenging but solvable.

4. The Loop (Co-Evolution)

The Teacher and Student learn together.

As the Student gets better at solving problems, the Teacher is forced to invent harder and more complex problems to keep getting those "bonus points."
As the Teacher gets better at inventing complex problems, the Student learns to solve them.
They push each other forward, creating a never-ending cycle of learning without needing a human to grade a single paper.

Why It's a Big Deal

The paper tested this on seven different math benchmarks and three general reasoning tests. Here is what happened:

Beating the "Human-Graded" Baseline: OpenSIR started with just one trivial problem ("1+1"). Despite having zero human-annotated training data, it outperformed models trained on over 7,000 human-graded examples.
Fixing the "Stuck" Models: Previous self-play methods actually made advanced models worse or didn't help them at all. OpenSIR consistently improved them.
Generalizing: The skills learned weren't just for math. Because the AI was forced to explore diverse concepts, it got better at general reasoning tasks (like logic puzzles) too.

The Secret Sauce

The paper found that two things were absolutely critical:

Diversity: If you remove the "bonus for being different," the AI stops learning new things and performance drops.
Calibration: If you don't check if the problems are solvable, the AI starts generating nonsense, and learning stops.

In a Nutshell

OpenSIR is like a self-driving car that doesn't just drive on the same road it learned from a map. Instead, it invents new roads, checks if they are drivable, and learns to drive on them, eventually becoming a better driver than any human instructor could teach it to be—all without a single human ever telling it where to turn.

Technical Summary: OpenSIR: Open-Ended Self-Improving Reasoner

Problem Statement

Recent advances in Large Language Model (LLM) reasoning, such as DeepSeek-R1 and OpenAI o1, rely heavily on Reinforcement Learning with Verifiable Rewards (RLVR) using extensive human-annotated datasets. This dependence creates a scalability bottleneck and potentially caps model performance at human levels. While self-play offers a path to surpass human performance by learning from unambiguous reward signals (as seen in games like Go and Poker), applying it to mathematical reasoning faces a fundamental obstacle: generated problems lack ground-truth answers for feedback.

Existing self-play methods attempt to circumvent this by using external verifiers (e.g., compilers) or heuristic signals (e.g., majority voting). However, when applied to models that have already undergone significant post-training on human-curated data, these methods yield marginal or even negative gains. The paper identifies a deeper limitation: prior approaches fail to support open-ended learning. Without explicit incentives for novelty, the "teacher" component of self-play collapses into familiar problem templates, ceasing to generate new learning signals. Consequently, these methods cannot push well-trained models beyond their current capabilities.

Methodology: OpenSIR

The authors propose Open-Ended Self-Improving Reasoner (OpenSIR), a self-play framework where a single policy $\pi_\theta$ alternates between teacher (problem generation) and student (problem solving) roles without external supervision or annotated data. The framework bootstraps from a single trivial seed problem (e.g., "What is 1+1?").

Each training iteration consists of four phases:

Problem Generation: The teacher generates $G$ new problems conditioned on reference problems sampled from an accumulated pool.
Solution Sampling: The student attempts multiple solutions per problem. A reference answer is determined via majority voting, and a "solve rate" is calculated to measure reliability.
Scoring:
- Teacher Scoring (Novelty): Problems are scored based on Difficulty and Diversity.
  - Difficulty: Measured via a solvability score (triangular function peaking at an optimal solve rate, e.g., 0.5–0.9) and solution length (encouraging multi-step reasoning).
  - Diversity: Measured via embedding distance (cosine distance) between the new problem and the existing problem pool to ensure exploration of semantically new concepts.
- Student Scoring (Correctness): Based on the accuracy of the parsed answer against the reference answer and proper formatting.
Model Update: The policy is updated using a Group Relative Policy Optimization (GRPO)-style objective. High-quality problem-solution pairs are selected based on novelty scores, with half the batch allocated to teacher training and half to student training.

Crucially, OpenSIR employs dual-role training, where both the teacher and student co-evolve. This allows the teacher to calibrate problem difficulty dynamically as the student's capabilities improve, preventing the generation of problems that are either too easy or too hard (and thus invalid).

Key Contributions

Identification of Failure Modes: The paper demonstrates that existing self-play methods fail on well post-trained models because they lack mechanisms to sustain novelty, leading to repetitive problem generation and performance degradation.
OpenSIR Framework: The introduction of the first self-play framework capable of open-ended self-improvement from a single trivial seed. It jointly optimizes for problem difficulty (calibration) and diversity (exploration) without external verifiers.
Empirical Superiority: OpenSIR consistently improves performance across instruction-tuned and reasoning-specialized models, outperforming baselines trained on thousands of human-annotated examples.

Results

The authors evaluated OpenSIR on seven mathematical benchmarks (GSM8K, MATH-500, Minerva, OlympiadBench, College Math, AIME 2024/2025) and three general reasoning benchmarks (BBEH, MMLU-Pro, SuperGPQA).

Instruction Models: On Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct, OpenSIR achieved average gains of +3.99 and +3.27 on mathematical reasoning, respectively. This surpassed GRPO baselines trained on >7,000 human-annotated examples (which gained +2.69 and +1.91) and other self-play methods (Absolute Zero, R-Zero), which showed negligible or negative gains.
Reasoning Models: On heavily post-trained reasoning models (DeepSeek-R1-Distill-Llama-8B and Qwen3-8B), OpenSIR improved mathematical reasoning by +3.71 and +2.41, respectively. In contrast, data-based GRPO baselines offered minimal gains, and other self-play methods often degraded performance.
General Reasoning Transfer: OpenSIR was the only method to consistently improve general reasoning capabilities, with gains of +4.79 and +4.41 on the reasoning models. This suggests that training on diverse, self-generated novel problems develops transferable reasoning skills, not just mathematical proficiency.
Ablation Studies:
- Diversity Reward: Removing the diversity reward reduced math accuracy by ~2.7 points and general reasoning by ~2.5 points, while halving the number of unique concepts explored.
- Dual-Role Training: Training only the student (with a fixed teacher) resulted in a significant performance drop and unstable problem difficulty, confirming the necessity of co-evolution.
- Seed Robustness: The method performed consistently regardless of whether the initial seed was a trivial arithmetic problem, a geometry problem, or a competition-level problem.

Significance

The paper claims that OpenSIR represents a compelling paradigm for open-ended autonomous mathematical reasoning development. By combining difficulty calibration and diversity rewards, it enables models to recursively expand their capabilities beyond the boundaries of human-annotated data.

The authors emphasize that this approach eliminates the dependence on extensive curated datasets, which are a bottleneck for scalability. The results suggest that models can effectively bootstrap mathematical reasoning through recursive self-improvement, provided the system is explicitly incentivized to explore novel concepts rather than repeating familiar ones. While the paper notes that performance eventually plateaus after extended training, it establishes that novelty-driven self-play is a viable and necessary path for pushing LLMs beyond human-level performance in reasoning tasks.