💬 NLP

The Invisible Leash: Why RLVR May or May Not Escape Its Origin

This paper empirically demonstrates that while Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM precision, it ultimately fails to expand reasoning boundaries because its support-constrained optimization narrows the model's solution space, often causing it to overlook correct answers that were previously accessible to the base model.

Original authors: Fang Wu, Weihao Xuan, Ximing Lu, Mingjie Liu, Yi Dong, Zaid Harchaoui, Yejin Choi

Published 2026-02-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fang Wu, Weihao Xuan, Ximing Lu, Mingjie Liu, Yi Dong, Zaid Harchaoui, Yejin Choi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Invisible Leash"

Imagine you have a very talented student (the Base Model) who has read a library of books and knows a lot of facts. They can solve math problems, but sometimes they get stuck or give a slightly messy answer.

To make them better, you introduce a strict coach (the RLVR system). This coach doesn't teach the student new subjects from scratch. Instead, the coach watches the student solve problems, and every time the student gets the exact right answer, the coach gives a high-five (a reward). If the answer is wrong, the coach gives a gentle "try again."

The paper asks a crucial question: Does this coaching actually teach the student new ways to think, or does it just train them to repeat the specific answers they already knew, but with more confidence?

The authors call this an "Invisible Leash." They found that while the student gets much better at getting the first answer right, they are actually tied to the same limited set of ideas they started with. They can't easily jump over the fence to discover brand-new solutions they didn't already have a hint of.

Key Findings Explained

1. The "Support" Trap: Keeping the Old Map

Think of the student's knowledge as a map of a city. The "Base Model" has a map that shows many possible routes to a destination, including some obscure, winding paths.

What RLVR does: It looks at the map and says, "Hey, this one route works perfectly! Let's pave it with gold and make it the only road we drive on."
The Result: The student becomes incredibly fast and accurate on that one paved road. However, they start to forget the other valid, winding roads that were on the original map.
The Paper's Finding: In almost every test (math, logic, coding), the RLVR-trained models kept the "good" answers they already knew (about 93–99% of them) but lost access to other correct answers that the original model could have found. They didn't add new roads to the map; they just narrowed the traffic down to a single lane.

2. The "Precision vs. Diversity" Trade-off

Imagine you are fishing in a lake.

The Base Model casts a wide net. It catches a few big fish, but it also catches some smaller, different kinds of fish. It's a bit messy, but it finds everything that's there.
The RLVR Model uses a spear. It is incredibly precise. If it sees a fish, it hits it every time. But because it's so focused on hitting the target, it stops casting the net wide.

The Catch: If you only need one fish (one correct answer), the spear (RLVR) is better. But if you need to catch any fish in a large bucket (trying many different times to find a solution), the wide net (Base Model) is actually better because it covers more ground. The paper found that while RLVR is great at getting the first try right, the original model often wins if you give it many chances to try.

3. The "Confusing Noise" Illusion

Sometimes, the RLVR-trained student looks like they are thinking harder. They might pause longer, say more words, or seem more "uncertain" while they are talking (this is called Token-Level Entropy).

The Analogy: Imagine a chef who is chopping vegetables very loudly and chaotically. It looks like they are experimenting with new techniques.
The Reality: Despite the noise, they are still only making the exact same three dishes they always made. They aren't inventing a new recipe; they are just making the old ones with more dramatic flair.
The Paper's Finding: Even though the models sound more "uncertain" step-by-step, they actually converge on a much smaller set of final answers. They are less diverse in the end.

4. When Does It Actually Learn Something New?

The paper did find a few rare exceptions where the RLVR model did find a new solution. This only happened in two specific scenarios:

Reassembling Puzzle Pieces: The student already knew the individual puzzle pieces (sub-skills) but couldn't put them together correctly. The coach helped them snap the pieces together.
Fixing the Format: The student knew the answer but didn't know how to write it down in the specific format the coach wanted. The coach taught them the formatting rules, unlocking the answer that was already there.

In these cases, the model wasn't discovering a new universe of thought; it was just polishing what was already hidden in the shadows of its original knowledge.

The Conclusion: Breaking the Leash

The paper concludes that current RLVR methods are like a precision tool, not a creativity engine. They are excellent at refining and perfecting what a model already knows, but they are "leashed" to the initial distribution of the base model. They cannot easily discover solutions that the base model had zero probability of finding.

To truly expand a model's reasoning abilities—to let it discover things it has never seen before—we need to add something new to the mix: explicit exploration. We need to deliberately push probability mass into the "dark corners" of the solution space, rather than just sharpening the focus on the bright spots we already know.

In short: RLVR makes the model a better executor of known ideas, but it doesn't necessarily make it a better thinker of new ones.

Technical Summary: The Invisible Leash? Why RLVR May or May Not Escape Its Origin

Problem Statement

Recent advances in Large Language Models (LLMs), particularly in mathematical and logical reasoning, have been driven by Reinforcement Learning with Verifiable Rewards (RLVR). While RLVR has demonstrated significant improvements in single-shot accuracy (pass@1), a fundamental debate persists: does RLVR genuinely expand a model's reasoning boundaries to discover novel solutions, or does it merely amplify high-reward outputs that the base model already possesses, thereby improving precision at the cost of diversity?

Current literature offers conflicting views. Some studies suggest RLVR performs conservative optimization within existing capabilities, while others argue it expands reasoning in specific domains. Furthermore, standard evaluation metrics like pass@k primarily capture solution retrieval rather than the capacity for novel reasoning. There is a lack of systematic empirical investigation into how RLVR reshapes the model's effective reasoning support throughout training, specifically whether it can discover solutions that were previously inaccessible to the base model.

Methodology

The authors introduce a framework centered on Empirical Support, defined as the set of correct solutions a model can realistically discover under finite sampling. They formalize the dynamics of solution accessibility by categorizing correct completions into four regions based on whether the base model ( $q$ ) and the RLVR-trained model ( $\pi_\theta$ ) assign non-negligible probability mass (above a threshold $\epsilon$ ) to them:

Support Preservation: Correct solutions accessible to both models.
Support Expansion: Correct solutions newly accessible to RLVR (negligible in base, non-negligible in RLVR).
Support Shrinkage: Correct solutions accessible to the base model but lost in RLVR.
Out of Support: Correct solutions missed by both.

To quantify these dynamics, the paper proposes four metrics:

Support Retention Rate (SRR): Measures preservation of base model solutions.
Net Discovery Rate (NDR): Measures the fraction of new, genuine discoveries.
Support Dynamic Score (SDS): A harmonic mean balancing retention and discovery.
Net Support Change Rate (NSCR): Captures the net expansion or shrinkage of the solution space.

The study employs extensive empirical experiments across diverse model scales (1.5B to 32B parameters) and domains (mathematics, logical reasoning, factual QA, code generation, and visual reasoning). Models evaluated include ProRL, Nemotron, Skywork, AceReason, Phi4, and DAPO variants. The authors also analyze entropy dynamics, distinguishing between token-level entropy (local uncertainty during generation) and answer-level entropy (global diversity of final outputs). Additionally, they provide theoretical proofs regarding the limits of on-policy RL updates.

Key Contributions and Results

1. Predominant Preservation with Limited Expansion

The empirical results indicate that RLVR acts primarily as a support-constrained optimization mechanism. Across all tested models and domains:

High Retention, Low Discovery: Support Retention Rates (SRR) are consistently high (0.90–0.99), indicating RLVR reliably preserves the base model's accessible solutions. However, Net Discovery Rates (NDR) are extremely low (often $\le 0.04$ ), meaning genuine discovery of new reasoning paths is rare.
Net Shrinkage: In almost all cases, the number of correct solutions lost (Shrinkage) exceeds the number of new solutions found (Expansion). The Net Support Change Rate (NSCR) is consistently negative, suggesting RLVR systematically narrows the accessible solution set.
Paradoxical Performance: While RLVR models often outperform base models at low sampling budgets (e.g., pass@1), base models frequently dominate at high sampling budgets (e.g., pass@8192) due to their broader solution coverage.

2. The Precision-Diversity Trade-off

The study identifies a fundamental trade-off where RLVR enhances precision by concentrating probability mass on known high-reward trajectories but reduces the diversity of global solutions.

Entropy Dynamics: While some RLVR models exhibit increased token-level entropy (appearing more uncertain during generation), they consistently show decreased answer-level entropy. This indicates that despite local stochasticity, the models converge onto a smaller set of distinct final answers.
Perplexity Analysis: RLVR models show higher perplexity when evaluated against diverse external reasoning traces (e.g., from DeepSeek-R1 or Claude Sonnet), confirming that they reduce the model's ability to assign probability to diverse solution styles.

3. Mechanisms of Rare Expansion

When expansion does occur (e.g., in tasks like Dice, Arc 1D, or Boxnet), the paper identifies two specific mechanisms:

Recomposition of Subskills: The base model possesses the necessary sub-components (fragments) but fails to assemble them globally. RLVR amplifies these weakly represented components.
Prompt-Format Alignment: RLVR corrects misalignments in response formatting, unlocking capabilities that were present but suppressed due to instruction-following failures.
Crucially, these expansions do not represent the discovery of solutions truly absent from the base distribution; they are reweightings of existing long-tail capabilities.

4. Theoretical Limits

The paper provides theoretical proofs (Theorem C.1 and Corollary C.2) demonstrating that standard on-policy RLVR updates cannot expand the support of the policy beyond the initial support of the base model. If the base model assigns zero probability to a correct solution, RLVR cannot discover it. Furthermore, the asymptotic pass@k performance of an RLVR model is bounded by that of the base model.

Significance and Claims

The paper claims to reveal an "invisible leash" constraining current RLVR practices. The primary significance of this work lies in:

Reframing RLVR's Role: It positions RLVR not as a capability expander that unlocks new reasoning horizons, but as a precision enhancer that sharpens the distribution around known high-reward modes.
Limitations of Current Paradigms: It challenges the assumption that RLVR inherently leads to deeper reasoning or novel problem-solving. Instead, it suggests that without explicit exploration strategies, RLVR is fundamentally limited by the initialization of the base model.
Implications for Evaluation: The findings suggest that relying solely on pass@1 or low-budget metrics may overstate RLVR's capabilities, as these metrics favor the concentrated distributions RLVR produces, while ignoring the loss of diversity that harms performance in high-budget or multi-solution scenarios.
Future Directions: The authors conclude that breaking this "invisible leash" requires future innovations that explicitly seed probability mass into underrepresented solution regions, moving beyond standard on-policy optimization to hybrid strategies or explicit exploration mechanisms.

The work does not claim to solve these limitations but aims to provide a rigorous empirical and theoretical foundation for understanding the inherent trade-offs between precision and diversity in verifiable reward-based reinforcement learning.