💬 NLP

SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks

SCOPE is a data-free self-play framework that co-evolves a Challenger and a Solver with a frozen self-judge to generate and grade document-grounded tasks, significantly improving open-ended and short-form QA performance across multiple models without relying on external supervision or curated prompts.

Original authors: Wai-Chung Kwan, Aryo Pradipta Gema, Joshua Ong Jun Leang, Pasquale Minervini

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wai-Chung Kwan, Aryo Pradipta Gema, Joshua Ong Jun Leang, Pasquale Minervini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a student how to write a great research paper, but you don't have a teacher, a textbook, or even a list of questions to ask. You only have a massive library of books (the internet) and the student's own brain.

This is the problem the paper SCOPE tries to solve. Usually, to train AI to do complex, open-ended tasks (like deep research or creative writing), humans have to write the questions and grade the answers. But humans are slow, expensive, and limited.

SCOPE is a new way to train AI without any human help. It uses a "self-play" method, similar to how two chess players might play against each other to get better, but with a twist: they are playing a game of Hide and Seek with Information.

Here is how it works, broken down into simple roles:

1. The Three Characters

The system creates three versions of the same AI model to play different roles:

The Challenger (The Quiz Master): This AI's job is to create a tricky question based on a specific document it just read. It has to make the question hard enough that the other AI can't just guess the answer from its memory, but not so hard that it's impossible.
The Solver (The Detective): This AI's job is to answer the Challenger's question. To do this, it has to go back to the library, search for clues, read documents, and piece together the answer. It can't just rely on what it already knows; it must find new evidence.
The Judge (The Strict Grader): This is a frozen, unchanging copy of the original AI. It never learns or changes. Its only job is to look at the source document and the question, then write a specific "grading rubric" (a checklist of what a good answer looks like). It then grades the Solver's answer against that checklist.

2. The Game Loop: How They Get Better

The magic happens in a cycle of three steps:

The Challenge: The Challenger reads a document and invents a question. It tries to find a "sweet spot": a question that is just hard enough to stump the current Solver. If the question is too easy, the Solver gets an A instantly; if it's too hard, the Solver fails completely. The Challenger wants the Solver to struggle a little bit so it can learn.
The Investigation: The Solver gets the question. It doesn't know the answer. It has to search the library, read multiple documents, and synthesize the information to write a response.
The Grading: The Judge looks at the source document and the question, then writes a specific checklist (rubric). For example, if the question is about a shipwreck, the rubric might say: "The answer must mention the captain's decision to sail into the storm" and "The answer must mention the safety management failures." The Judge then grades the Solver.

The "Aha!" Moment:
The key trick is that the Challenger and the Judge see the source document, but the Solver does not. The Solver only sees the question. This forces the Solver to learn how to search effectively to find the missing information.

If the Solver gets better at searching, the Challenger has to get smarter about asking harder questions. If the Challenger gets better at asking hard questions, the Solver has to get better at searching. They "co-evolve," pushing each other to higher levels of skill, just like two athletes training together.

3. Why This is a Big Deal

No Human Teachers Needed: Most AI training relies on humans writing thousands of questions and answers. SCOPE generates all its own training data from raw documents.
Good at "Fuzzy" Tasks: Previous self-play methods only worked for math or code where there is one right answer (like 2+2=4). Open-ended tasks (like "Write a story" or "Analyze a historical event") don't have a single right answer. SCOPE uses the Judge's rubric to grade these fuzzy tasks, allowing the AI to improve at creative and research-heavy work.
It Works: The paper tested this on three different AI models. Even though they were only trained on these self-generated open-ended tasks, they got significantly better at:
- Deep Research: Finding and combining information from many sources.
- Creative Writing: Writing better stories and essays.
- Short Questions: Surprisingly, they also got better at simple fact-checking questions they were never explicitly trained on.

The "Secret Sauce"

The paper found that for this to work, the Challenger must keep changing. If you freeze the Challenger and only let the Solver learn, the Solver quickly figures out the easy questions and stops improving. The Challenger must constantly invent new, slightly harder puzzles to keep the Solver on its toes.

Also, the Judge needs to be smart enough to write a good checklist (rubric). If the checklist is vague, the Solver learns nothing. If the checklist is specific and grounded in the facts of the document, the Solver learns exactly what it needs to find.

In Summary

SCOPE is like a self-running gym for AI. Instead of a human coach telling the AI what to do, the AI creates its own workout (the Challenger), does the workout (the Solver), and grades its own form (the Judge). By constantly challenging itself with just the right amount of difficulty, the AI learns to research, reason, and write better than it could with just human-curated data.

Technical Summary: SCOPE – Self-Play via Co-Evolving Policies for Open-Ended Tasks

Problem Statement

Current reinforcement learning (RL) approaches for training large language models (LLMs) on open-ended tasks (e.g., deep research, complex QA, creative writing) face a fundamental bottleneck: they rely on external supervision. This supervision typically takes the form of human-curated prompts, reference answers, or "frontier-model" judges (stronger LLMs used to evaluate outputs). These dependencies tether model performance to human-level data quality and limit scalability.

Existing self-play methods for LLMs attempt to bypass this by co-evolving task generators and solvers. However, these prior approaches (e.g., SPICE, Dr. Zero, R-Zero) are restricted to domains with verifiable answers (string matches, numerical equality, code execution). Open-ended tasks lack a unique correct answer, rendering rule-based verification impossible. While rubric-based evaluation offers a path forward, previous rubric-based RL methods (e.g., RaR, DR Tulu, RPG) still depend on curated prompts or external frontier-model judges, failing to achieve a truly data-free, self-supervised loop.

Methodology: SCOPE

The authors introduce SCOPE (Self-play via Co-evolving Policies for Open-Ended tasks), the first framework to extend data-free self-play to open-ended tasks. The system operates without curated data or external judges, relying instead on a "self-judge" derived from the base model itself.

Core Architecture

SCOPE co-evolves two policies initialized from the same base model $M_0$ , alongside a fixed third role:

Challenger ( $C_t$ ): An evolving policy trained to generate open-ended, document-grounded tasks. It performs multi-turn retrieval over an unlabelled corpus to construct tasks that sit near the current Solver's capability frontier.
Solver ( $S_t$ ): An evolving policy trained to answer the Challenger's tasks. It utilizes multi-turn retrieval-augmented generation (RAG) to gather evidence and synthesize answers.
Judge ( $J = M_0$ ): A frozen copy of the initial base model. It serves two critical functions:
- Rubric Generation: For each task $q$ and source document $d$ , the Judge generates task-specific evaluation rubrics (criteria) grounded in the source document.
- Grading: The Judge scores the Solver's responses against these rubrics.

Training Loop

The training process alternates between two stages, optimized using Group Relative Policy Optimization (GRPO):

Stage 1: Challenger Training

The Challenger generates a task $q$ based on a source document $d$ .
Quality Gates: The Judge applies binary checks to ensure the task references identifiable entities and is grounded in the source document. Tasks failing these gates are discarded to prevent the Challenger from generating generic, reward-hacking prompts.
Difficulty Reward: The previous Solver ( $S_{t-1}$ ) attempts the task. The Judge grades the responses. The Challenger is rewarded for generating tasks where the Solver's average score is near a target difficulty $\tau = 0.5$ . This ensures tasks remain challenging but solvable, maximizing feedback variance.

Stage 2: Solver Training

The Challenger generates a pool of tasks.
Filtering: Tasks are filtered based on quality gates and difficulty (retaining only those where the Solver's mean score falls within $[0.2, 0.8]$ ).
Solver Reward: The Solver is trained on these filtered tasks. The reward function combines:
- Rubric Score: Weighted average of binary verdicts on specific criteria.
- Format & Search Rewards: Encouraging proper tool use and structure.
- Length Penalty: A cosine-based penalty prevents the Solver from exploiting the Judge's tendency to favor longer outputs by generating excessively verbose responses.

Theoretical Justification

The paper argues that sustained self-improvement requires learnable information. SCOPE achieves this through:

Task Grounding: The Challenger and Judge observe the source document $d$ , while the Solver only sees the task $q$ . The Solver must retrieve $d$ to answer, creating an information gap.
Hidden-Rubric Gap: The rubrics $G$ contain document-specific criteria not fully revealed by the task $q$ . The Solver must retrieve the specific details in $d$ to satisfy the rubric.
As the Solver improves, it closes the gap on current tasks. The Challenger must co-evolve to propose new tasks where the gap remains open, preventing the learning signal from saturating.

Key Contributions

First Data-Free Self-Play for Open-Ended Tasks: SCOPE is the first framework to successfully apply self-play to open-ended tasks without relying on curated prompts or external frontier-model judges.
Rubric-Based Self-Judging: It demonstrates that a frozen copy of the base model can effectively generate task-specific rubrics and grade responses, replacing the need for external supervision.
Co-Evolution Necessity: The paper provides empirical evidence that a static Challenger fails to sustain improvement; the Challenger must co-evolve with the Solver to maintain the "difficulty frontier."
Generalization: Despite training exclusively on document-grounded open-ended tasks, the method improves performance on short-form QA and creative writing, suggesting transferable gains in retrieval and synthesis skills.

Experimental Results

The authors evaluated SCOPE on three instruction-tuned models (Qwen2.5-7B, Qwen3-8B, OLMo-3-7B) across eight open-ended benchmarks and seven short-form QA benchmarks.

Open-Ended Performance: SCOPE improved open-ended performance by +5.4 to +10.4 points across the models. Notably, on Qwen2.5-7B, the average score rose from 24.4 to 34.8 (+10.4).
Comparison to Curated Data: SCOPE matched or exceeded GRPOdata, a baseline trained on ~9,000 human-curated prompts with frontier-model rubrics. For instance, on Qwen3-8B, SCOPE achieved 43.1 vs. GRPOdata's 41.5.
Short-Form QA Transfer: Although trained only on open-ended tasks, SCOPE improved short-form QA scores by +7.8 to +13.8 points, surpassing GRPOdata on all three models.
Creative Writing: SCOPE showed significant gains in creative writing (up to +7.4), a domain where GRPOdata often regressed below the base model, indicating that corpus-grounded self-play builds robust synthesis skills.
Ablation Studies:
- Co-evolution: Freezing the Challenger resulted in negligible gains (+0.8 vs. +3.4 for full SCOPE), as tasks became too easy.
- Reward Design: Removing quality gates or length penalties led to "reward hacking" (generic tasks or inflated text), causing performance to collapse below the base model.
- Rubric Quality: The quality of rubric generation (not the grading capacity) was identified as the bottleneck. Rubrics from smaller models (4B) often lacked specific document facts, whereas 8B+ models produced grounded, discriminative criteria.

Significance and Claims

The paper claims that SCOPE represents a significant step toward scalable, data-free self-improvement for LLMs. By replacing the dependency on human-curated data and frontier-model judges with a self-contained loop of co-evolving policies and rubric-based self-judging, SCOPE extends the reach of self-play beyond verifiable domains (math, code) to complex, open-ended reasoning tasks.

The authors modestly note that the multi-stage pipeline requires more compute than single-stage GRPO, limiting current experiments to 7–8B models. However, they argue that in the "post-data regime" where curated supervision is exhausted, trading compute for continued self-improvement is a justified and necessary path forward. The results suggest that open-ended tasks can be mastered through self-generated signals, provided the system maintains a dynamic information asymmetry between the task generator and the solver.