🤖 AI

RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs

This paper introduces a novel framework analyzing reasoning paths at trajectory and step levels to reveal that Reinforcement Learning with Verifiable Rewards (RLVR) compresses incorrect trajectories and concentrates reasoning into fewer steps, while Supervised Fine-Tuning (SFT) expands correct trajectories and distributes reasoning across many steps, thereby explaining the complementary success of the current two-stage training paradigm.

Original authors: Kohsei Matsutani, Shota Takashiro, Gouki Minegishi, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo

Published 2026-05-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kohsei Matsutani, Shota Takashiro, Gouki Minegishi, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Two Ways to Teach a Robot to Think

Imagine you have a robot that is good at talking but not great at solving complex math or coding puzzles. To make it smarter, researchers use two main training methods:

SFT (Supervised Fine-Tuning): This is like imitation. You show the robot a perfect example of how a human expert solves a problem, step-by-step, and say, "Do exactly this."
RL (Reinforcement Learning): This is like trial and error with a scorecard. You let the robot try to solve the problem on its own. If it gets the right answer, it gets a "gold star" (reward). If it gets it wrong, it gets nothing. Over time, it learns to avoid the paths that lead to failure.

The paper asks: What actually happens inside the robot's brain when we use these two different methods? Do they change how the robot thinks, or just what it answers?

The authors discovered that these two methods do opposite things to the robot's "thought process."

Analogy 1: The "Reasoning Trail" (Trajectory Level)

Imagine the robot's thinking process as a hiker walking through a massive, foggy forest to find a hidden treasure (the correct answer). There are many paths: some lead to the treasure, and many lead to dead ends or cliffs.

SFT (The Imitator) "Expands" the Good Paths:
When you show the robot an expert's map (SFT), it learns to walk on the correct path. But here's the catch: it also starts walking on many different variations of that correct path. It becomes very creative and diverse in how it finds the treasure.
- The Catch: It doesn't stop walking on the bad paths. If the robot was already wandering into a swamp (making mistakes), SFT teaches it new ways to walk, but it might still wander into the swamp occasionally. It "expands" the number of correct routes but doesn't necessarily clear the bad ones.
RL (The Scorer) "Squeezes" the Bad Paths:
When you let the robot play the game with rewards (RL), it quickly realizes that the swamp paths lead to zero points. So, it stops walking there. It "squeezes" all the incorrect, confusing, or dead-end paths out of existence.
- The Catch: It also tends to "squeeze" the good paths. It might stop exploring creative variations of the correct answer and just stick to the one path it knows works best. It becomes very focused, but less diverse.
The Winning Combo (SFT + RL):
The paper explains why the current best practice is to do SFT first, then RL.
1. First, use SFT to teach the robot new ways to find the treasure (expanding the correct paths).
2. Then, use RL to punish the robot for wandering into the swamp (squeezing the incorrect paths).
- Result: You get a robot that knows many ways to solve the problem but is very disciplined about avoiding mistakes.

Analogy 2: The "City of Thoughts" (Step Level)

Now, let's look at the robot's thinking not as a single path, but as a city map. Every "step" in the reasoning (like "calculate the area" or "check the formula") is a building in this city. The connections between steps are the roads.

RL Creates a "Hub-and-Spoke" City:
After RL training, the city changes. The robot starts relying heavily on a few specific, super-important buildings (hubs). It visits these key steps over and over again.
- The Effect: The city becomes very efficient but crowded around these few spots. The robot concentrates its "thinking power" into a small number of critical steps. It's like a commuter who only uses three specific bridges to cross the river, ignoring all the others.
SFT Creates a "Distributed" City:
After SFT training, the city looks different. The robot spreads its thinking out. It visits many different buildings, and no single building is visited as overwhelmingly as in the RL city.
- The Effect: The thinking is "homogenized" or spread out evenly. It's like a city where everyone uses a wide variety of streets, and no single street is a traffic jam.

The Key Finding:

RL makes the robot's thinking intense and focused on a few key steps (Squeezing).
SFT makes the robot's thinking broad and distributed across many steps (Expanding).

The Shape of the City (Topology)

The researchers also looked at the "shape" of these thinking cities using geometry.

The Base Model (Before training): The city is broken into isolated neighborhoods (communities). It's hard to get from one neighborhood to another. The robot gets stuck in local loops.
RL: It breaks down the walls between neighborhoods. It creates a city dominated by a few massive "hubs" that connect everything together. This makes the robot very fast at finding the answer if it hits the right hub, but it relies on those specific hubs.
SFT: It also breaks down the walls, but instead of creating hubs, it builds a web where everything is connected to everything else. This makes the city very robust and easy to navigate from any point to any other point.

Summary of the Paper's Claims

RL is a "Squeezer": It crushes incorrect thinking paths and concentrates the robot's reasoning into a few highly efficient, high-traffic steps. It makes the robot very good at getting the right answer on the first try (Pass@1) by eliminating bad options.
SFT is an "Expander": It teaches the robot new, correct ways to think and spreads the reasoning load across many steps. However, it doesn't automatically get rid of the bad paths the robot might have been using before.
Why SFT + RL Works: The best results come from using SFT to teach the robot how to think correctly (expanding the good paths) and then using RL to force it to stop making mistakes (squeezing the bad paths).
Structure Matters: RL creates a "hub-heavy" reasoning structure, while SFT creates a "distributed" structure. Both are different from the "fragmented" structure of an untrained model.

The paper concludes that understanding these mechanical differences helps explain why the current "two-stage" training recipe (SFT then RL) is so successful for creating smart reasoning AI.

Technical Summary: RL SQUEEZES, SFT EXPANDS: A COMPARATIVE STUDY OF REASONING LLMS

Problem Statement

While Reinforcement Learning with Verifiable Rewards (RLVR) and Supervised Fine-Tuning (SFT) are the dominant paradigms for post-training Large Language Models (LLMs) to enhance reasoning capabilities, the specific mechanisms by which these methods shape the underlying reasoning processes remain elusive. Previous studies have largely focused on outcome-based metrics (e.g., Pass@k), often concluding that RL merely elicits pre-existing base capabilities or that SFT leads to memorization while RL promotes generalization. However, these studies lack a granular understanding of how RL and SFT fundamentally alter the structure of reasoning paths and the topology of the reasoning space. This paper addresses the question: How do RL and SFT shape the reasoning process beyond accuracy measurements?

Methodology

The authors introduce a novel analysis framework that quantifies reasoning paths and captures their qualitative changes under different training regimes. The study employs models of varying sizes (1.5B, 7B, and 14B parameters) across mathematical (AIME24, AIME25, AMC23) and code (HumanEval) domains. The analysis is conducted at two levels of granularity:

Trajectory-Level Analysis:
- Method: Multiple outputs (M=256) are sampled from Base, RL, SFT, and SFT+RL models. Unique reasoning trajectories are identified by applying hierarchical clustering (UPGMA) to the outputs based on character-level n-gram similarity (chrF).
- Goal: To count and compare the number of unique correct and incorrect trajectories before and after training.
Step-Level Analysis (Reasoning Graphs):
- Construction: Model outputs are segmented into sentences, embedded into a vector space (using BGE-large-en-v1.5), and clustered (K-means) to define nodes. Transitions between sentences form directed edges, creating a "reasoning graph" for each problem.
- Metrics: The study analyzes global and local topological properties, including:
  - Exponential Decay Rates: Fitting rank plots of node visitation frequency, degree, and betweenness centrality to an exponential law ( $X(R) \propto e^{-\lambda R}$ ) to determine the decay rate $\beta$ .
  - Global Topology: Edge density, clustering coefficient, assortativity, modularity, Freeman centralization, average path length, global efficiency, and algebraic connectivity.
  - Local Structure: Graphlet analysis (counting 4-node subgraphs) to detect cyclic vs. acyclic structures.

Key Contributions

The paper establishes a "reasoning path perspective" that explains the complementary roles of SFT and RL:

Trajectory-Level Dynamics:
- RL Squeezes: RL training significantly compresses the number of unique incorrect trajectories, effectively pruning the search space of errors. However, it also reduces the number of unique correct trajectories, concentrating probability mass on fewer paths.
- SFT Expands: SFT training increases the number of unique correct trajectories, introducing new solution strategies absent in the base model. Crucially, SFT alone tends to preserve incorrect trajectories, failing to guarantee Pass@1 improvements.
- Synergy: The success of the two-stage SFT+RL approach is attributed to SFT expanding the space of correct solutions, followed by RL compressing the space of incorrect ones.
Step-Level Structural Changes:
- Decay Rates: RL steepens the decay rates ( $\beta$ ) of visitation frequency, degree, and betweenness centrality distributions. This indicates that RL consolidates reasoning functionality (e.g., hubs, central nodes) into a small subset of steps. Conversely, SFT flattens these decay rates, homogenizing functionality across many diverse steps.
- Global Topology:
  - Base Models: Exhibit high modularity and low global efficiency, indicating reasoning is trapped in distinct, weakly connected communities.
  - RL Models: Transform graphs into hub-centralized structures with high edge density and low modularity, enabling efficient traversal through a few central nodes.
  - SFT Models: Weaken community boundaries to form globally connected graphs with high robustness and reachability efficiency, though they may retain some incorrect paths.
- Local Structure: Both RL and SFT increase the proportion of cyclic subgraphs (reflecting backtracking and verification), suggesting that local cyclic structures alone cannot explain performance differences; global topology is the differentiator.

Results

Mathematical and Code Domains: The findings hold consistently across mathematical datasets (AIME, AMC) and code generation (HumanEval).
Pass@k vs. Pass@1: The reduction of unique correct trajectories by RL explains why Base models can sometimes outperform RL models in Pass@k (at large $k$ ) due to the broader coverage of the base model's distribution, while RL excels in Pass@1 by concentrating mass on the most probable correct path.
Topological Correlations: Global efficiency and algebraic connectivity are positively correlated with Pass@1/Pass@k ratios, while modularity is negatively correlated. This suggests that reasoning graphs facilitating efficient traversal across the entire solution space are critical for high performance.
Ablation Studies: The results regarding decay rates and topological shifts remain robust across different cluster counts ( $K$ ), distance metrics (Euclidean vs. Cosine), and sentence encoders.

Significance and Claims

The paper claims to provide empirical support for the "RL squeezes, SFT expands" hypothesis, offering a mechanistic explanation for why the current best practice of two-stage training (SFT followed by RL) is successful.

Interpretation of Training: It posits that SFT acts as an "expander" of the reasoning space (learning new strategies), while RL acts as a "compressor" (optimizing the distribution to suppress errors and focus on high-probability correct paths).
Data Curation: The findings suggest that data construction for reasoning should consider graph metrics (e.g., ensuring low modularity and high global efficiency) rather than just accuracy.
Future Directions: The authors suggest that applying RL only to functional steps (hubs) could improve efficiency and that exploring whether exploration bonuses in RL prevent "excessive squeezing" is a valuable direction.

The work does not claim to invent new training algorithms but rather offers a diagnostic framework to understand existing ones, interpreting why current post-training recipes work and suggesting directions for more efficient learning approaches and data curation.