💻 computer science

Why Do Reasoning Models Lose Coverage? The Role of Data and Forks in the Road

This paper investigates the "reasoning shrinkage" phenomenon in fine-tuned reasoning models, identifying the prevalence of ambiguous decision-point scenarios in training data as the primary cause and demonstrating that targeted data synthesis and diversity-aware decoding can effectively mitigate this coverage loss.

Original authors: Ngoc-Hieu Nguyen, Parshin Shojaee, Phuc Minh Nguyen, Nan Zhang, Chandan K Reddy, Khoa D Doan, Rui Zhang

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ngoc-Hieu Nguyen, Parshin Shojaee, Phuc Minh Nguyen, Nan Zhang, Chandan K Reddy, Khoa D Doan, Rui Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a student how to solve a maze. You want them to be able to find the exit no matter which path they take, but you also want them to be fast and accurate.

This paper investigates a strange problem that happens when we train advanced AI models (like the ones that do complex math or logic) to be "reasoning" experts. We call this problem "Coverage Shrinkage."

Here is the simple breakdown of what the authors found, using everyday analogies:

The Problem: The "Shrinking" Map

When these AI models are trained, they get really good at finding the one best path to the answer. If you ask them a question once, they are very likely to get it right (this is called pass@1).

However, if you ask them the same question 10 or 100 times to see all the different ways they could solve it, they stop giving you variety. They all start giving the exact same answer, or worse, they all get stuck on the same wrong path. Their "map" of possible solutions has shrunk. They have lost the ability to explore different routes.

The Culprit: "Forks in the Road"

The authors argue that this isn't because the AI's learning algorithm is broken. Instead, the problem lies in the training data.

Imagine a storybook where a character reaches a fork in the road.

The Real World: The character might go left or right. Both paths look valid, but only one leads to the treasure.
The Training Data: The storybook only shows the character taking the right path. It never explains why they chose right over left, or even mentions that the left path was an option.

The authors call these moments "Forks in the Road." These are decision points where the model has to choose between multiple valid options, but the data only shows one.

Because the model only ever sees one path taken, it learns a bad habit: "I must pick the first thing that looks right, and I must be 100% confident I'm right, even if I'm not." It stops exploring. It stops considering the "left" path because the storybook never showed it.

The Experiments: Proving the Theory

The researchers tested this with two clever setups:

The Star Graph (The Maze):
They created a simple math puzzle that looked like a star-shaped maze. At the start, there were two paths. Only one led to the answer.
- Group A was trained on data showing the model walking forward through the maze, making a choice at the fork.
- Group B was trained on data showing the model walking backward from the answer to the start (no choices needed).
- Result: Group A (the ones who had to make choices) lost their ability to find alternative paths. Group B kept their full map. This proved that the act of making a choice without enough information causes the shrinkage.
The Style Switch (Code vs. Words):
They looked at math problems that could be solved by writing code OR by writing in plain English.
- Scenario 1: They gave the model a mix of problems, but each problem only had one solution style (some were code, some were words).
- Scenario 2: They gave the model the same mix, but for every single problem, they showed both the code solution AND the word solution.
- Result: In Scenario 1, the model got confused and started picking a style based on tiny, irrelevant clues (like the order of words), shrinking its options. In Scenario 2, where the model saw both options for the same problem, it stayed flexible and kept its "map" wide.

The "Spurious Cues" (The Bad Habits)

Because the model is forced to choose at these "forks" without knowing the right answer, it starts grabbing onto spurious cues (fake signals).

Example: If the problem starts with the word "Let," the model might decide, "Okay, I'll use the 'Backtracking' thinking style." If it starts with "To," it might decide, "Okay, I'll use the 'Linear' style."
The model isn't actually thinking about the math; it's just reacting to the first word. This makes it brittle. If you change the first word, the model might switch to a completely different (and worse) way of thinking.

The Solution: Wiggling the First Step

The paper offers two ways to fix this without retraining the whole model:

Better Data Design: When creating training data, don't just mix different types of problems. For each specific problem, show the model multiple valid ways to solve it. This teaches the model that "at this fork, both paths are possible."
The "First Word" Trick: Since the model gets stuck on the very first word it generates, the researchers found they could "jiggle" the start. By forcing the model to try starting with different common words (like "Okay," "Let's," or "To"), they could unlock the different thinking styles that were locked away.
- Analogy: It's like a door that is stuck. The model thinks the door is locked, but if you push the handle in a slightly different direction (change the first word), the door swings open, and the model remembers all the paths it used to know.

The Bottom Line

The paper concludes that AI models don't "forget" how to be diverse. They just get locked into a single path because their training data forced them to make blind choices at "forks in the road" without showing them the alternatives. By changing how we present these choices in the data, or by nudging the model at the very start of its answer, we can get that lost diversity back.

Technical Summary: Why Do Reasoning Models Lose Coverage? The Role of Data and Forks in the Road

Problem Statement

Recent advances in reasoning models, achieved through specialized post-training procedures like Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR), have significantly improved pass@1 accuracy (the probability of the first generated solution being correct). However, a counterintuitive phenomenon known as coverage shrinkage has been observed: while pass@1 improves, pass@k (the probability that at least one correct solution appears in $k$ independent samples) degrades relative to the base model.

Prior literature has largely attributed this shrinkage to the optimization dynamics of RLVR, suggesting that reward-driven training overemphasizes narrow, high-reward paths leading to mode collapse. However, recent evidence indicates that shrinkage also occurs under SFT, implying that the root cause may not be the algorithm itself but a more fundamental driver. This paper investigates the hypothesis that data structure, specifically the presence of "forks in the road" (decision points where multiple valid reasoning paths exist but the rationale for selection is obscured), is the primary driver of coverage shrinkage.

Methodology

The authors propose a data-centric view, arguing that reasoning problems often involve decision points where a model must choose between multiple valid strategies without knowing which will succeed. If training data only exposes the successful path (survivorship bias) without explaining the choice, the model learns to rely on spurious cues to resolve ambiguity, eventually suppressing alternative trajectories.

To test this, the authors designed controlled case studies isolating these decision points:

Graph Navigation (Synthetic Task):
- Setup: A star-graph navigation task where a model must traverse from a start node to a target. The graph contains "indecipherable nodes" (branching points) where multiple branches are valid continuations, but only one leads to the target.
- Variants:
  - Forward: The model must navigate from source to target, encountering branching points (decision points).
  - Reverse (w/o DP): The model navigates from target to source, presenting the solution in reverse order, effectively removing the need for path selection.
- Models: Evaluated on Qwen2.5-0.5B and EvoLM-1B.
Reasoning Mode Selection (Real-world Tasks):
- Natural Language (NL) vs. Code: The authors analyzed mathematical reasoning datasets (OpenMathInstruct-1 and 2) to compare two data diversity structures:
  - Data-level diversity: The dataset is globally balanced (50% NL, 50% Code), but each problem is solved with only one mode.
  - Problem-level diversity: Each problem is presented with both NL and Code solutions in the training data.
- Linear vs. Backtracking: Using distilled reasoning models (e.g., DeepSeek-R1 variants), the authors examined how initial prefix tokens (e.g., "Okay," "Let," "To") act as decision points that commit the model to either linear or backtracking reasoning structures.
Mitigation Strategies:
- Data Design: Comparing the effects of problem-level vs. data-level diversity.
- Inference-Time Intervention: Testing a "Top-k prefix" decoding mechanism where the model is forced to start from different high-probability initial tokens to recover diversity.

Key Results

1. Decision Points Drive Shrinkage

In the Graph Navigation task, models trained on the Forward setting (with decision points) showed a classic shrinkage pattern: pass@1 improved, but pass@k degraded significantly as training progressed. Conversely, models trained on the Reverse setting (without decision points) maintained high pass@k performance across all $k$ values.

Confidence Analysis: Models trained with decision points became overconfident at branching points, assigning high probability to both correct and incorrect branches. This suggests they learned to commit to a path based on spurious cues rather than calibrated uncertainty.

2. Structure of Diversity Matters

In the Reasoning Mode Selection task, the Data-level diversity setup (global balance, single mode per problem) resulted in severe coverage shrinkage. The model became increasingly confident in selecting a single mode (NL or Code) per problem, leading to a bimodal distribution and degraded pass@k.

In contrast, the Problem-level diversity setup (multiple modes per problem) maintained stable pass@k performance. This indicates that coverage is not determined solely by the amount of diversity in the dataset, but by how that diversity is structured within individual problems.

3. Brittleness at Decision Points

Analysis of Linear vs. Backtracking reasoning revealed that models are highly sensitive to initial prefix tokens. Small changes in the first token (e.g., "Okay" vs. "Let") could drastically shift the model's reasoning mode, response length, and accuracy.

Over-thinking/Under-thinking: Models often applied the wrong reasoning mode by default (e.g., backtracking on simple factual questions or linear thinking on complex counterfactual arithmetic). However, manipulating the prefix could recover the appropriate mode and significantly boost performance (up to 60% on specific tasks).

4. Mitigation via Inference-Time Intervention

The authors demonstrated that coverage shrinkage is not a permanent loss of capability but a suppression of alternative paths. By applying a Top-k prefix sampling strategy (forcing the model to start from diverse initial tokens) during inference, they could recover the lost coverage.

Result: While standard decoding showed declining pass@k in later training stages, the Top-8 prefix setting maintained high pass@k, effectively reactivating the suppressed reasoning trajectories without additional retraining.

Key Contributions

Data-Centric Hypothesis: The paper identifies "forks in the road" (decision points with obscured rationale) in fine-tuning data as a key driver of coverage shrinkage, challenging the view that shrinkage is solely an artifact of optimization algorithms like RLVR.
Controlled Empirical Evidence: Through synthetic graph tasks and real-world reasoning mode analysis, the authors provide evidence that the prevalence and structure of decision points correlate strongly with the severity of coverage shrinkage.
Diversity Structure Insight: The study reveals that how diversity is distributed (per-problem vs. across-problems) is critical; per-problem diversity is far more effective at preserving coverage.
Practical Mitigation: The paper introduces two strategies to control shrinkage:
- Designing synthetic data to expose multiple reasoning modes within single problems.
- Using inference-time prefix perturbation to recover suppressed diversity, demonstrating that the model's reasoning capacity is latent rather than lost.

Significance

The paper argues that coverage shrinkage is a data-driven artifact where models learn to lock alternative reasoning paths behind uncalibrated decision points. By shifting the focus from algorithmic optimization to data curation and inference-time intervention, the authors suggest that the lost coverage in reasoning models is not permanently forgotten but merely suppressed. These findings provide a lens for improving post-training data design and test-time scaling, highlighting that effective reasoning requires not just finding the correct path, but maintaining the ability to explore multiple valid paths when the "right" choice is ambiguous.