Technical Summary: From RLVR to RLSVR

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has driven significant progress in reasoning-oriented Large Language Models (LLMs) for domains with deterministic outcomes, such as mathematics and coding. In these settings, a verifier $V(x, y)$ can deterministically check if an output $y$ is correct given input $x$, providing unbiased, unlimited, and cost-free supervision.

However, RLVR remains largely inapplicable to open-ended tasks (e.g., creative writing, text summarization, general reasoning) where correctness is subjective and lacks a deterministic ground truth. Current approaches to bridge this gap rely on:

• Learned preference signals (e.g., RLHF, DPO), which require expensive human annotation.
• Model-based proxies (e.g., LLM-as-a-Judge, self-rewarding mechanisms), which introduce evaluation bias, are capped by the judge's capability, and incur significant inference costs.

The core challenge is that these methods attempt to approximate an unverifiable quality objective $Q(x, y)$ directly, rather than constructing a verifiable signal from the data itself.

2. Methodology: RLSVR and SpyRL

The paper proposes Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a paradigm that extends the principles of self-supervised learning (constructing pretext tasks with automatically generated labels) to the domain of RLVR. RLSVR transforms an open-ended task into a proxy environment where rewards are self-verifiable—derived from the environment's internal rules and interaction outcomes rather than external judges.

2.1 The RLSVR Framework

RLSVR operates through a task transformation $\Phi$ that maps a target task $(D, \tau)$ to a proxy environment $E$:

1. Latent-Variable Injection: The environment samples an input $x$ and a latent variable $z$ (e.g., a hidden identity or perturbed state), recording $z$ as the ground truth.
2. Conditioned Task Execution: Agents perform the target task based on observations derived from $(x, z)$.
3. Verifiable Interaction: The environment poses a question about $z$ that must be answered using the task outputs. Crucially, answering correctly depends on the quality of the output.
4. Rule-Based Reward: The reward is computed by checking the interaction outcome against the recorded $z$, requiring no external verifier.

2.2 SpyRL: An Instantiation via Information-Asymmetric Self-Play

The authors instantiate RLSVR with SpyRL (Self-PlaY Reinforcement Learning), a multi-agent framework inspired by the social deduction game Who Is the Spy?.

Game Mechanics:

• Agents: A group of $n$ agents consists of $n-1$ "Civilians" and one "Spy."
• Asymmetric Information:
  • Civilians: Receive the complete task input $x$.
  • Spy: Receives a degraded version $g(x)$ (e.g., a continuous span of text masked or removed), creating an information deficit.
• Performing Stage: All agents generate outputs $y_i$ for the target task (e.g., summarization, creative writing, math reasoning) based on their respective observations.
• Detection Stage: Agents analyze all outputs and vote to identify the spy.
• Reward Signal:
  • Detection Reward ($R_D$): Deterministically verifiable. If an agent votes for the correct spy index $u$, they receive a positive reward; otherwise, zero.
  • Performing Reward ($R_P$): Derived from the voting outcomes. The spy receives a negative reward proportional to the number of votes they receive (suspicion), while civilians receive rewards based on a zero-sum constraint relative to the group average.
  • Alignment: Outputs that fail to hide the information deficit (i.e., lower quality) attract more votes, resulting in lower rewards. Thus, the unverifiable "quality" of the output is converted into a verifiable "identity recognition" problem.

Optimization Strategy:
SpyRL employs alternating optimization between the Performing policy ($\pi^P_\theta$) and the Detection policy ($\pi^D_\phi$).

• Role-Advantage Estimation (RAE): To address the structural imbalance where spies inherently face harder tasks, RAE calibrates rewards by subtracting role-specific baselines, ensuring the gradient signal reflects performance relative to the role's difficulty rather than absolute reward magnitude.
• Alternating Updates: The system switches between optimizing the performers (to generate better, more deceptive outputs) and the detectors (to better identify weaknesses). This prevents policy stagnation and local equilibria common in fixed self-play.

3. Key Contributions

1. RLSVR Paradigm: The proposal of Reinforcement Learning with Self-Verifiable Rewards, which integrates self-supervised learning principles into RLVR to enable scalable training for open-ended tasks without external verifiers.
2. SpyRL Framework: The introduction of SpyRL, an information-asymmetric self-play method that reformulates open-ended generation as a multi-agent social deduction game, producing verifiable rewards through rule-based identity inference.
3. Empirical Validation: Comprehensive evaluation across text summarization, creative writing, and mathematical reasoning, demonstrating that SpyRL outperforms existing self-improvement methods (R-Zero, Absolute Zero) on non-verifiable tasks while maintaining or improving performance on verifiable reasoning tasks.
4. Generalization: Evidence that the approach generalizes to domain-specific summarization (scientific papers) and transfers capabilities across tasks (e.g., summarization-trained models improving creative writing).

4. Experimental Results

The authors evaluated SpyRL on Qwen3-4B and Qwen3-8B models across three domains:

• Non-Verifiable Tasks (Summarization & Creative Writing):

  • Summarization: SpyRL achieved the highest ROUGE-L scores across five benchmarks (GovReport, Multi-News, etc.). For example, on GovReport with Qwen3-4B, SpyRL improved ROUGE-L from 33.2 (Absolute Zero) to 36.7. In pairwise A/B tests against baselines, SpyRL won the majority of comparisons (e.g., 74.6% win rate on GovReport).
  • Creative Writing: SpyRL significantly outperformed baselines in GPT-4o pairwise evaluations across dimensions like Novelty, Emotion, and Coherence. On WritingPrompts, SpyRL achieved an 84.3% win rate against the untrained backbone and 78.9% against R-Zero.
  • Correlation: Analysis showed a strong positive correlation between the number of suspicion votes received and lower output quality (ranked by GPT-4o), validating that the vote-based reward aligns with actual task performance.

• Verifiable Tasks (Mathematical Reasoning):

  • SpyRL consistently outperformed baselines on seven benchmarks (GSM8K, Math500, AIME 24/25, Minerva, MMLU-Pro, GPQA-D).
  • On Qwen3-8B, SpyRL improved Math500 accuracy to 81.2% (vs. 78.4% for R-Zero) and AIME 25 to 23.3% (vs. 14.2% for R-Zero).

• Cost Efficiency:

  • Unlike Rubric-as-Reward (RaR) baselines that incur ~$200–$900 in verifier costs per experiment, SpyRL requires no external verifier, relying solely on the game's internal rules.

• Ablation Studies:

  • Alternating Optimization: Essential for performance; joint training led to degradation (e.g., Math500 accuracy dropped from 68.2% to 53.1%).
  • Group Size: Increasing players from 3 to 5 yielded the largest gains; further scaling showed diminishing returns.
  • RAE: Removing Role-Advantage Estimation caused significant performance drops, confirming the necessity of calibrating for information asymmetry.

5. Significance and Claims

The paper claims that verifiability need not be an intrinsic property of a task but can be engineered through task transformation. By converting open-ended quality assessment into a verifiable identity-recognition problem, SpyRL enables scalable, verifier-free self-improvement for general open-ended capabilities.

The authors argue that their approach addresses the fundamental bottleneck of RLVR in non-deterministic domains:

• It eliminates reliance on biased human preferences or expensive LLM judges.
• It provides a stable, rule-based reward signal derived from the environment itself.
• It demonstrates that self-play is not restricted to inherently verifiable domains (like math/code) but can be extended to creative and generative tasks through appropriate environment design.

The results suggest that task transformation can extend scalable RLVR-based self-improvement beyond its current limits, offering a path toward training open-ended LLM capabilities without human supervision.