💬 NLP

Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs

This paper demonstrates that Reinforcement Learning with Verifiable Rewards (RLVR) improves LLM reasoning through highly sparse, targeted token-level distributional shifts, where a small fraction of critical token decisions are responsible for the performance gains and can be isolated via cross-sampling interventions.

Original authors: Haoming Meng, Kexin Huang, Shaohang Wei, Chiyu Ma, Shuo Yang, Xue Wang, Guoyin Wang, Bolin Ding, Jingren Zhou

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haoming Meng, Kexin Huang, Shaohang Wei, Chiyu Ma, Shuo Yang, Xue Wang, Guoyin Wang, Bolin Ding, Jingren Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart student (the Base Model) who is good at math but sometimes gets stuck or takes a wrong turn when solving a really hard problem. You decide to give them a tutor using Reinforcement Learning with Verifiable Rewards (RLVR). The tutor doesn't rewrite the student's entire personality; instead, they whisper specific advice at just the right moments to help them get the right answer.

This paper is a "forensic investigation" into exactly how that tutoring works. The researchers wanted to know: Does the tutor change everything the student says, or do they just tweak a few critical words?

Here is the breakdown of their findings using simple analogies:

1. The "Scalpel vs. Hammer" Discovery

Most people thought the tutor (RL) was like a hammer, smashing the student's old habits and rebuilding them from scratch.

The Reality: The tutor is actually a scalpel.
The Finding: When comparing the student before and after tutoring, the researchers found that 98% of the words the student says remain exactly the same. The changes are incredibly sparse. The tutor only changes the student's mind on a tiny fraction of words (less than 2% in some cases).

2. The "Cross-Sampling" Experiment (The Magic Switch)

To prove these few words were the only thing that mattered, the researchers did a clever experiment called Cross-Sampling. Imagine two versions of the student:

Student A (The Tutor's Star): Always gets the right answer.
Student B (The Original): Often gets stuck.

Experiment 1: The "Magic Injection"
They took Student B's essay and secretly swapped in just 4% of the words from Student A at the specific moments where Student A made a different choice.

Result: Student B suddenly started getting the right answers! By changing only a tiny handful of words, they unlocked the full intelligence of the tutor.

Experiment 2: The "Sabotage"
They took Student A's perfect essay and swapped in just 5% of the words from Student B (the original, less smart version) at those same critical moments.

Result: Student A's perfect essay collapsed. The answer became wrong.
The Lesson: The entire success of the "smart" model hinges on a few specific, high-stakes decisions. If you mess up just those few words, the whole reasoning chain falls apart.

3. What Kind of Words Are Changed?

The researchers looked closely at which words were swapped.

They aren't inventing new words: The tutor isn't teaching the student to use a vocabulary they didn't know.
They are re-ranking choices: Imagine the student has a list of 10 possible next steps. The base model thinks, "Option 3 looks okay, but Option 1 is the best." The tutor says, "Actually, look at Option 3 again; that's the one that leads to the solution."
The Analogy: It's like a GPS. The car (the model) knows all the roads. The GPS (RL) doesn't build new roads; it just says, "Don't take the highway (Option 1); take this small side street (Option 3) instead." The side street was always there, but the car needed a nudge to choose it.

4. Where Do These Changes Happen?

The changes aren't random. They happen in two main places:

The Beginning: When the student first decides how to approach the problem (the "branching" decision).
The End: When the student is formatting the final answer or deciding to stop.
The Middle: Surprisingly, the middle of the reasoning process stays mostly the same. Once the student is on the right path (thanks to the early nudge), they can usually finish the job on their own.

5. The "Divergence" Metric

The researchers used a tool called Jensen-Shannon Divergence to measure how different the two students' minds were at every step.

The Graph: If you plot this on a graph, it looks like a flat line near zero with a few sharp spikes.
Meaning: For almost every word, the two students agree perfectly. The "spikes" are the critical moments where the tutor stepped in to change the direction.

6. The "Weighted Advantage" Idea (The Future)

Because they found that only a few words matter, the researchers tried a new training trick. Instead of treating every word equally during training, they told the AI: "Pay extra attention to the words where you are changing your mind the most."

Result: This made the training slightly more efficient, proving that focusing on these "critical few" is the key to better reasoning.

Summary: The "Targeted Refinement"

The big takeaway is that RLVR (Reinforcement Learning) doesn't rewrite the brain; it refines the path.

Think of the Base Model as a hiker who knows the forest well but sometimes gets lost at a fork in the road. The RLVR process doesn't give the hiker a new map or new legs. It just places a small sign at the one specific fork where the hiker usually turns left, telling them to turn right instead. Once they turn right, they can walk the rest of the way to the treasure on their own.

The paper proves that the "magic" of AI reasoning improvements isn't a global overhaul; it's a sparse, targeted intervention at the most critical decision points.

1. Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR), such as Group Relative Policy Optimization (GRPO) and DAPO, has significantly improved the reasoning capabilities of Large Language Models (LLMs) on benchmarks like AIME and GPQA. However, the mechanisms underlying these improvements remain opaque.

The Gap: Most evaluations rely on aggregate metrics (accuracy, reward), offering only a high-level view. It is unclear how RLVR reshapes the token-level predictive distributions of a base model.
Key Questions:
1. How broadly are distributional shifts distributed across token positions?
2. Are these shifts responsible for downstream reasoning gains?
3. What is the fine-grained nature of these changes (e.g., new tokens vs. re-ranking existing ones)?

2. Methodology

The authors conducted a systematic empirical study using Qwen2.5-32B (and other variants like Qwen2.5-Math-7B, Qwen3-8B, Mistral-Small-24B) trained with RLVR methods (DAPO, SimpleRL) on mathematical reasoning datasets (AIME 2024/2025, GPQA).

The study is organized around three core analyses:

A. Token-Level Distributional Characterization

Metric: Used Jensen-Shannon (JS) Divergence to measure the difference between the base model's distribution ( $\pi_{base}$ ) and the RL-finetuned model's distribution ( $\pi_{RL}$ ) at every token position.
Approach: Evaluated conditional distributions on identical sequence contexts generated by the RL policy.
Comparisons: Contrasted RLVR with Supervised Fine-Tuning (SFT) and analyzed the impact of hyperparameters (e.g., clip-higher settings in DAPO).

B. Cross-Sampling Interventions (Functional Importance)

To determine if sparse shifts drive performance, the authors performed forward and reverse cross-sampling:

Forward Cross-Sampling: Generate a sequence primarily with $\pi_{base}$ , but inject tokens sampled from $\pi_{RL}$ at positions where divergence exceeds a threshold.
Reverse Cross-Sampling: Generate a sequence primarily with $\pi_{RL}$ , but replace tokens with $\pi_{base}$ samples at high-divergence positions.
Budget Control: Varied the number of interventions to observe how performance scales with the quantity of swapped tokens.

C. Fine-Grained Mechanics

Analyzed the structural nature of changes at high-divergence positions:

Top-k Overlap: Measured the intersection of top- $k$ candidate tokens between base and RL models.
Rank Reordering: Tracked how token rankings shifted.
Low-Probability Behavior: Checked if RLVR promoted tokens that were highly unlikely under the base model.
Training Evolution: Tracked divergence changes across training checkpoints.

D. Diagnostic Intervention

Proposed and tested Divergence-Weighted Advantages, where the RL advantage signal is reweighted based on token-level divergence (High-KL boost vs. Low-KL boost) to see if focusing learning signals on specific tokens improves training.

3. Key Findings & Results

A. Sparsity of Distributional Shifts

Highly Sparse: RLVR induces changes in only a tiny fraction of tokens.
- SimpleRL: >98% of token positions show near-zero JS divergence.
- DAPO: >83% of positions show near-zero divergence.
Contrast with SFT: Supervised Fine-Tuning produces dense, global distributional shifts. RLVR is a "scalpel," while SFT is a "hammer."
Positional Concentration: Shifts are not random; they concentrate at the beginning of responses (high-level branching decisions) and the end (formatting/termination), with sporadic high-divergence points throughout.

B. Functional Criticality (Cross-Sampling Results)

Forward Intervention: Injecting a small fraction of RL tokens into base generations recovers RL-level performance.
- Example: On AIME 2024, injecting <4% RL tokens (approx. 38 tokens) into base generations raised accuracy from ~8% to >25% (matching RL).
- Surprise: In some cases, the mixed policy outperformed the standalone RL policy.
Reverse Intervention: Reverting a small fraction of RL tokens to base choices collapses performance.
- Example: Replacing ~5% of RL tokens with base tokens dropped accuracy from ~25% back to ~8%.
Conclusion: RL and base policies are behaviorally identical on most tokens but differ critically at a sparse set of high-impact decisions that steer the reasoning trajectory.

C. Mechanics of the Shifts

Re-ranking, Not Invention: At high-divergence positions, RLVR rarely introduces "new" tokens. Instead, it reorders and amplifies tokens that were already plausible (high probability) in the base model's top-k set.
- 80% of RL top-1 tokens were already in the base model's top-3.
Low-Probability Promotion: While rare, some methods (like DAPO with clip-higher) do promote low-probability tokens, which may be crucial for difficult reasoning steps.
Entropy Interaction: Low-divergence tokens are generally low-entropy (confident). High-divergence tokens span a wide entropy range, showing RLVR can override even confident base predictions.

D. Divergence-Weighted Advantages

Modifying the RL objective to weight advantages by token-level divergence yielded improvements over baselines (e.g., DAPO baseline 33.6% $\to$ 45.0% with High-KL boost on AIME 2024). This suggests that explicitly targeting sparse, high-divergence tokens during training is beneficial.

4. Significance and Contributions

Mechanistic Insight: The paper provides the first fine-grained, token-level lens showing that RLVR does not globally rewrite model behavior but performs targeted probability reallocation on a sparse set of critical decision points.
Validation of Sparsity: It empirically proves that reasoning gains are not due to broad distributional changes but hinge on a few "critical tokens" that steer the generation trajectory.
Distinction from SFT: It clarifies that RLVR and SFT operate via fundamentally different mechanisms (sparse refinement vs. dense imitation), explaining why RL often generalizes better for reasoning tasks.
Practical Implications:
- Efficiency: Suggests that future RL objectives could focus learning signals on these sparse, high-divergence regions (as demonstrated by the divergence-weighted advantage experiments).
- Interpretability: Provides a framework for diagnosing where and why an RL model fails or succeeds by analyzing specific token distributions.
- Safety/Control: Highlights that small, local edits to token choices can drastically alter reasoning outcomes, which is vital for understanding model robustness and potential vulnerabilities.

In summary, the paper reframes RLVR fine-tuning not as a global policy shift, but as a sparse, structured refinement process that unlocks effective reasoning trajectories already accessible (but unchosen) by the base model.