🤖 machine learning

GRLO: Towards Generalizable Reinforcement Learning in Open-Ended Environments from Zero

The paper introduces GRLO, a highly efficient reinforcement learning approach that trains models from scratch on a small set of open-ended conversational interactions to achieve strong generalization across diverse domains like math and coding, significantly reducing the data and compute requirements compared to traditional in-domain RLVR methods.

Original authors: Shangjian Yin, Yu Fu, Yue Dong, Zhouxing Shi

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shangjian Yin, Yu Fu, Yue Dong, Zhouxing Shi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read student (a large language model) who has already read millions of books during their "pre-training" phase. They know a lot, but they are a bit clumsy at following instructions, solving puzzles, or writing code. To turn them into a top-tier expert, we usually need to give them a "post-training" course.

Traditionally, there are two ways to run this course:

The Specialist Tutor (RLVR): You hire a tutor who only teaches math or coding. They use a strict answer key (a "verifier") to grade every single step. If the student gets a math problem right, they get a gold star. If they get it wrong, they get a red X. This works incredibly well for math, but the student often forgets how to have a normal conversation or write a creative story because they only practiced for the test.
The General Chat Coach (RLHF): You hire a coach who teaches the student how to be polite, helpful, and engaging in general conversation. They use human feedback (like "this answer was nice") rather than a strict answer key.

The Problem:
The "Specialist Tutor" approach is expensive and requires massive computing power. It also makes the student great at math but bad at chatting. The "General Chat Coach" is cheaper but hasn't been proven to make the student better at hard reasoning tasks like math or coding.

The New Idea: GRLO (The "Open-Ended Gym")
The authors of this paper propose a new training method called GRLO. Instead of sending the student to a math-only boot camp or a generic chat class, they send them to a gym with open-ended challenges.

Imagine a gym where the equipment isn't just a treadmill (math) or a punching bag (chat). Instead, the student is given 5,000 diverse, tricky, and open-ended questions like:

"Analyze the history of the Moon's South Pole."
"Explain how a specific philosophy connects to modern politics."
"Design a guide for keeping exotic pets."

These questions don't have a single "right" answer you can check with a computer. Instead, the student gets feedback based on how well-structured, logical, and helpful their long, detailed answers are.

The Big Surprise (The "Zero" Magic)
The paper claims something surprising happened: Just by practicing in this "open-ended gym," the student got significantly better at math, coding, and chatting.

The Results: They took a base model (Qwen3-4B) and trained it for only 22.7 hours on a small amount of data (5,000 prompts).
The Comparison: This tiny amount of training boosted the model's average performance from 24.1 to 63.1.
The Efficiency: This was 46 times less data and 68 times less computing power than the massive "Specialist Tutor" (RLVR) methods usually required to get similar results.
The Transfer: The model didn't just get better at chatting; it implicitly learned how to think better, which helped it solve math problems and write code, even though it never saw a single math problem or coding task during this specific training phase.

What Happens Next?
The authors tried adding a little bit of "Specialist Tutor" training (math-only) after the open-ended gym. It helped a tiny bit, but only on the hardest, competition-level math problems. The main boost came entirely from the open-ended training.

The Takeaway
The paper suggests that you don't need to build a massive, expensive, math-specific training pipeline to get a smart, reasoning AI. If you take a strong base model and train it on a small, diverse set of open-ended conversations where it has to think deeply and organize its thoughts, it naturally becomes better at everything else, including hard reasoning tasks. It's like saying, "If you train an athlete to be a versatile, adaptable all-rounder, they will naturally become better at running, jumping, and throwing, even if you never made them practice those specific sports."

In short: A little bit of smart, open-ended conversation training can unlock a model's hidden potential for reasoning and coding, saving a massive amount of money and time compared to traditional methods.

Technical Summary: GRLO – Towards Generalizable Reinforcement Learning in Open-Ended Environments from Zero

Problem Statement
Recent advancements in post-training large language models (LLMs) have largely bifurcated into two paradigms: Reinforcement Learning from Human Feedback (RLHF), which optimizes for human preferences, and Reinforcement Learning from Verifiable Rewards (RLVR), which optimizes for correctness in domains like mathematics and code. While RLVR has demonstrated superior efficiency and performance on specific reasoning tasks, it suffers from significant limitations: it is computationally expensive, heavily dependent on verifiable ground-truth signals, and exhibits weak transfer to general conversational abilities. Conversely, standard RLHF often requires large-scale, domain-specific data and struggles to induce broad reasoning capabilities without explicit reasoning traces.

The core problem addressed by this work is the "domain-coverage gap": current methods either specialize heavily in verifiable domains (losing general chat ability) or require massive computational budgets to achieve broad capabilities. Furthermore, it remains unclear whether reinforcement learning conducted in open-ended, non-verifiable environments can unlock general conversational capabilities that implicitly transfer to downstream reasoning and code generation tasks without subsequent domain-specific fine-tuning.

Methodology: GRLO
The authors propose GRLO (Generalizable Reinforcement Learning from Open-Ended Environments), a training recipe that shifts the focus from the optimizer to the training environment. Rather than introducing a novel algorithmic optimizer, GRLO applies standard RLHF-style optimization (specifically PPO) on a small, curated pool of open-ended prompts.

Training Environment: The authors construct a pool of approximately 5,000 synthetic prompts spanning diverse, cognitively demanding, and largely non-verifiable domains, including scientific analysis, argumentative synthesis, conceptual explanation, and long-form reasoning. A topic audit confirms the pool is not math-centric but covers policy, history, biomedicine, technology, and humanities.
Optimization: The policy is initialized from a pretrained base model (e.g., Qwen3-4B-Base) and optimized using PPO with a learned reward model (Skywork-Reward-V2) and a KL-regularized objective. Crucially, the training distribution ( $D_{open}$ ) consists of open-ended prompts where exact verification is not possible, contrasting with the verifiable distributions ( $D_{ver}$ ) used in RLVR.
Hypothesis: The central hypothesis is that optimizing a model on a small set of open-ended interactions can reshape its output distribution to better exploit pre-trained capabilities, thereby improving general conversational ability which implicitly transfers to downstream tasks like mathematical reasoning and code generation.

Key Contributions

Re-evaluation of In-Domain RL: The paper challenges the necessity of in-domain RL training as the primary source of downstream transfer, evaluating whether open-ended RL can serve as a practical, lightweight alternative.
Lightweight Generalization: The authors demonstrate that a single stage of open-ended RL (GRLO) can jointly improve reasoning, code generation, and general chat performance. This approach achieves aggregate performance comparable to strong, resource-intensive in-domain RLVR baselines (like General-Reasoner) while using significantly less data (46× less) and compute (68× less).
Systematic Analysis: The work provides a detailed analysis of scaling behaviors, cross-family transfer (across Qwen and Llama architectures), and response lengths, clarifying that the observed gains are not merely artifacts of verbosity but stem from improved decision quality and response organization.

Experimental Results
The authors evaluated GRLO on multiple backbones, including Qwen3-4B, Qwen3-8B, Qwen2.5-3B, and Llama3.2-3B.

Qwen3-4B Performance: On the Qwen3-4B backbone, GRLO increased the average performance across reasoning (Math500, GPQA), code generation (HumanEval, MBPP), and general chat (AlpacaEval 2 LC) from a baseline of 24.1 to 63.1. This result nearly matches the official Qwen3-4B (Non-Thinking) post-trained model and outperforms the General-Reasoner-4B baseline on aggregate metrics, despite using a fraction of the training resources.
Specific Gains: Relative to the base model, GRLO improved Math500 from 73.6 to 79.2, HumanEval from 6.1 to 84.8, and AlpacaEval 2 LC from 1.6 to 45.8.
Cross-Model Consistency: Similar trends were observed on Qwen3-8B (average score rising from 33.7 to 67.3) and Llama3.2-3B (rising from 21.5 to 39.3), indicating the method's robustness across different model families.
Ablation Studies: Experiments showed that using in-domain math prompts for RL (instead of open-ended) improved math scores slightly but severely degraded chat transfer. Similarly, swapping PPO for GRPO in the open-ended setting yielded suboptimal results, confirming that the combination of the open-ended environment and PPO is critical.
Supplementary RLVR: A subsequent in-domain RLVR stage provided only marginal gains (e.g., increasing the average from 63.1 to 63.2 on Qwen3-4B), with improvements primarily visible on harder competition-style math benchmarks (AIME). This suggests that the broad transfer is driven by the initial open-ended stage, while RLVR serves as a specialized refinement for difficult verifiable tasks.
Efficiency: The training required only 5,000 prompts and 22.7 GPU hours. Analysis of output lengths revealed that GRLO models produced shorter, higher-quality responses compared to baselines that relied on verbose reasoning traces.

Significance and Claims
The paper claims that GRLO offers a "simple and efficient recipe" for building broadly capable post-trained models. Its primary significance lies in demonstrating that open-ended RL can recover a large fraction of the broad gains typically attributed to much heavier, verifier-backed training pipelines.

The authors argue that the environment in which RL is applied is a more critical factor than the specific optimizer or the scale of the data. By moving RL into a cognitively demanding open-ended environment, models can implicitly acquire reasoning and coding skills without explicit domain-specific supervision. This challenges the prevailing notion that high-performance reasoning requires massive, domain-specific RLVR budgets, suggesting instead that a lightweight, open-ended RL stage from "Zero" is sufficient to unlock generalizable capabilities. The work concludes that while a subsequent in-domain RLVR stage can further specialize performance on hard benchmarks, the foundational broad transfer is achieved through the open-ended interaction itself.

Technical Summary: GRLO – Towards Generalizable Reinforcement Learning in Open-Ended Environments from Zero

More like this