💬 NLP

RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments

This paper introduces RLVE, a framework that scales reinforcement learning for language models by utilizing a large suite of 400 procedurally generated, adaptive verifiable environments (RLVE-Gym) to dynamically adjust problem difficulty, thereby significantly improving generalizable reasoning capabilities compared to traditional static data approaches.

Original authors: Zhiyuan Zeng, Hamish Ivison, Yiping Wang, Lifan Yuan, Shuyue Stella Li, Zhuorui Ye, Siting Li, Jacqueline He, Runlong Zhou, Tong Chen, Chenyang Zhao, Yulia Tsvetkov, Simon Shaolei Du, Natasha Jaques

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhiyuan Zeng, Hamish Ivison, Yiping Wang, Lifan Yuan, Shuyue Stella Li, Zhuorui Ye, Siting Li, Jacqueline He, Runlong Zhou, Tong Chen, Chenyang Zhao, Yulia Tsvetkov, Simon Shaolei Du, Natasha Jaques, Hao Peng, Pang Wei Koh, Hannaneh Hajishirzi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to solve puzzles. In the past, researchers gave the robot a giant stack of puzzle cards. Some cards were incredibly easy (like "1+1"), and some were impossibly hard (like "solve the universe").

The problem with this old method is that the robot gets bored with the easy cards and frustrated by the hard ones. It stops learning because it's either not challenged enough or it's failing so much it can't figure out how to improve. This is like a student trying to learn calculus by staring at a kindergarten math book, or by trying to solve a PhD thesis before they know how to add.

The paper introduces a new method called RLVE (Reinforcement Learning with Adaptive Verifiable Environments). Think of RLVE not as a stack of static cards, but as a smart, living video game that adjusts itself in real-time.

Here is how it works, broken down into simple concepts:

1. The "Smart Gym" (Adaptive Verifiable Environments)

Instead of a fixed list of problems, the researchers built a "gym" with 400 different types of exercise machines (like sorting numbers, solving Sudoku, or writing code).

The Magic: This gym is alive. It watches how well the robot is doing.
The Adjustment: If the robot is crushing the "easy" level, the gym automatically upgrades the difficulty. If the robot is struggling, it keeps the difficulty steady or lowers it slightly.
The Goal: The gym always keeps the robot in the "Goldilocks zone"—not too easy, not too hard, but just right to keep learning.

2. The "Infinite Puzzle Generator"

Usually, to train a robot, humans have to write down millions of specific problems and check the answers. That is slow and expensive.

The RLVE Trick: The researchers built "generators." Imagine a machine that can create an infinite number of unique Sudoku puzzles or math problems on the fly.
The Verifier: Crucially, the machine also has a built-in "answer key" that can instantly check if the robot's answer is right. It doesn't need a human to grade the test; the environment does it automatically and instantly.

3. The "Sliding Window" of Difficulty

The paper describes a clever way to manage the difficulty. Imagine a sliding window on a ruler.

The robot starts at the bottom (easy).
As it gets good, the window slides up to harder problems.
But the window has a size limit. It doesn't just jump to the hardest problems immediately; it keeps a mix of problems the robot is just starting to master and problems it has already mastered. This ensures the robot is always practicing the "edge" of its ability.

4. The Results: More Variety, Not Just More Data

The researchers tested this on several language models (AI brains). They found two big things:

Beating the "Boredom" Limit: When they tried to keep training a very smart model on the same old data, the model stopped improving (it hit a wall). But when they used the RLVE "gym" with its 400 adaptive environments, the model kept getting smarter.
Quality over Quantity: They found that having more types of puzzles (scaling up the environments) was more important than just having more copies of the same puzzle. It's like saying a chef gets better by learning to cook 400 different types of dishes, rather than cooking the same dish 10,000 times.

The Bottom Line

The paper claims that by building a system where the "teacher" (the environment) constantly adapts the lesson plan to the student's current skill level, and by using a huge variety of automatically generated puzzles, we can train AI to reason much better than before.

They tested this on a model that was already very good at reasoning. By switching to this new "adaptive gym" method, they improved its performance by 3.37% across various logic and math tests. In comparison, trying to squeeze more training out of the old, static method only improved it by 0.49%, even though they used three times as much computer power.

In short: Don't just give the AI more of the same; give it a smart, changing curriculum that grows with it.

Technical Summary: RLVE – Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments

1. Problem Statement

Scaling Reinforcement Learning (RL) for Large Language Models (LMs) faces two primary bottlenecks when relying on static data distributions:

Data Collection Cost: Acquiring large-scale datasets with ground-truth answers for verifiable reward computation is expensive and finite.
Vanishing Learning Signals: Static problem distributions often lead to training stagnation. If problems are too easy, they provide no meaningful learning signal; if too hard, they yield consistently poor rewards that impede gradient-based updates. As a policy model improves, a static distribution eventually becomes either trivially easy or inappropriately difficult, causing learning efficiency to drop or stall completely.

Existing approaches typically rely on fixed datasets where the problem distribution is predetermined and static, preventing adaptation to the evolving capabilities of the policy model.

2. Methodology: RLVE and RLVE-GYM

The paper introduces RLVE (Reinforcement Learning with Adaptive Verifiable Environments), a framework designed to scale LM RL training by utilizing procedurally generated, verifiable environments that dynamically adapt their difficulty to the policy's current capabilities.

2.1 Verifiable Environments

A verifiable environment is defined as a tuple $E = (I, P, R)$ :

Input Template ( $I$ ): A structured prompt format.
Problem Generator ( $P$ ): A program that procedurally samples parameters to instantiate specific problems from the template.
Verifier ( $R$ ): An algorithmic function that computes a reward by verifying the model's output against the problem constraints, rather than comparing it to a pre-computed solution.

Crucially, these environments exploit solving-verification asymmetry. The environment can execute programs to verify outputs (e.g., checking if a Sudoku grid is valid or if a sorting result is correct) even when the LM cannot execute code. In some cases (e.g., NP-complete problems), verification is computationally easier than solving, allowing for rewards without needing to implement complex solvers.

2.2 Adaptive Difficulty Mechanism

RLVE maintains a dynamic difficulty range $[\ell_\pi, h_\pi]$ for each environment.

Initialization: The range starts at the simplest difficulty level ( $\ell_\pi = h_\pi = 0$ ).
Sampling: During training, a difficulty level $d$ is uniformly sampled from $[\ell_\pi, h_\pi]$ .
Adaptation: The system tracks the model's accuracy on problems generated at the current upper bound $h_\pi$ . If the accuracy exceeds a threshold $\tau_{acc}$ (e.g., 90%) over a minimum sample size $\tau_{num}$ , the upper bound is incremented ( $h_\pi \leftarrow h_\pi + 1$ ).
Sliding Window: To prevent the range from becoming too wide (exposing the model only to trivially easy or impossibly hard problems), a lower bound $\ell_\pi$ is adjusted using a sliding window size $d_\Delta$ . This ensures the model is consistently exposed to problems that are "appropriately challenging."

2.3 RLVE-GYM

To implement this, the authors constructed RLVE-GYM, a suite of 400 verifiable environments developed through manual environment engineering. These environments are designed as pedagogical tools to develop reasoning capabilities (e.g., sorting, integration, logic puzzles) rather than to mimic specific real-world tasks directly. The suite includes diverse sources such as programming competition problems, mathematical operations, optimization problems, classical algorithms, logical puzzles, and NP-complete problems.

2.4 Training Algorithm

The framework is agnostic to the specific RL algorithm used. The authors employ DAPO (a variant of GRPO) with dynamic sampling. They define an "effective prompt ratio" to measure learning efficiency: the percentage of prompts retained after dynamic sampling that yield non-identical rewards. A high ratio indicates that the difficulty is well-matched to the policy.

3. Key Contributions

Adaptive Verifiable Environments: The proposal of a method where problem difficulty distributions shift dynamically based on policy performance, preventing the "vanishing signal" problem inherent in static datasets.
RLVE-GYM: The creation of a large-scale, manually engineered suite of 400 verifiable environments that serve as a scalable source of training data.
Environment Scaling: The demonstration that expanding the collection of training environments (scaling the environment dimension) is more critical for generalizable reasoning than merely increasing the volume of data from a single environment.
Cost-Efficient Scaling: A framework that achieves superior performance with significantly lower computational and financial costs compared to training on massive, static, human-curated datasets.

4. Experimental Results

The authors evaluated RLVE across two scenarios using six reasoning benchmarks (covering mathematics, code generation, and logical reasoning).

4.1 Data-Saturation Scenario

Setup: Starting from ProRL-1.5B-v2, a strong 1.5B reasoning model already saturated on a large RLVR dataset (ProRL) after ~20,000 H100 GPU hours.
Comparison: Continuing RL training on the original ProRL dataset vs. joint training on all 400 RLVE-GYM environments.
Result: RLVE achieved a 3.37% absolute average improvement across the six benchmarks. In contrast, continuing the original RL training yielded only a 0.49% gain, despite using more than 3× the compute (3,600 vs. ~1,100 H100 GPU hours).

4.2 Compute-Constrained Scenario

Setup: Starting from OpenThinker3-1.5B, a strong SFT model with no prior reasoning RL.
Comparison: Training with RLVE (400 environments) vs. training on DeepMath-103K, a high-quality, math-specific RLVR dataset.
Result: RLVE consistently outperformed DeepMath-103K, achieving higher performance on non-mathematical benchmarks (LiveCodeBench, BBEH) and most mathematical benchmarks. Notably, RLVE required no benchmark-specific data, whereas DeepMath-103K was explicitly designed for math.
Cost Efficiency: The authors note that constructing DeepMath-103K required ~$138,000 USD and 127,000 GPU hours, whereas RLVE-GYM offers a more cost-efficient path to stronger generalizable reasoning.

4.3 Component Analysis

Adaptivity vs. Static: Experiments showed that static difficulty distributions either lead to early saturation (if the max difficulty is too low) or inefficient learning (if the range is too wide but unadapted). Adaptive difficulty maintained a high effective prompt ratio and superior performance.
Environment Scaling: Increasing the number of training environments (from 1 to 4, 16, and 256) consistently improved out-of-distribution (OOD) performance, confirming that environment diversity is a key driver of generalizable reasoning.

5. Significance and Claims

The paper claims that environment engineering is becoming as foundational to LM development as feature, data, and prompt engineering. The significance of RLVE lies in its ability to:

Scale RL beyond data saturation: It enables further improvement even when a model has already saturated on a massive, high-quality static dataset.
Provide continuous learning signals: By dynamically adjusting difficulty, it ensures the model is always operating at the edge of its capability, avoiding the inefficiencies of static distributions.
Offer a scalable, cost-effective alternative: It demonstrates that procedurally generated, adaptive environments can outperform expensive, manually curated datasets, offering a path to train models on unbounded, diverse reasoning tasks without the prohibitive cost of collecting ground-truth answers for every problem.

The authors position this work as a step toward a broader effort to scale RL training through adaptive environments, calling for future research into non-verifiable environments and automatic environment engineering.