💬 NLP

Understanding Performance Gap Between Parallel and Sequential Sampling in Large Reasoning Models

This paper investigates why parallel sampling outperforms sequential sampling in Large Reasoning Models despite the latter's theoretical advantages, concluding through empirical analysis that the performance gap is primarily caused by sequential sampling's reduced exploration due to conditioning on previous answers, rather than aggregation limitations or context length constraints.

Original authors: Xiangming Gu, Soham De, Larisa Markeeva, Petar Veličković, Razvan Pascanu

Published 2026-04-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiangming Gu, Soham De, Larisa Markeeva, Petar Veličković, Razvan Pascanu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a very difficult puzzle, like a complex math problem or writing a piece of software that needs to work perfectly. You have a super-smart AI assistant (a "Large Reasoning Model") to help you.

The paper asks a simple but crucial question: What is the best way to use this AI to get the right answer?

There are two main ways to play the game:

1. The "Team of Explorers" (Parallel Sampling)

Imagine you hire 8 different detectives. You give them all the same case file (the question) and say, "Go solve this!"

They all work independently in separate rooms.
They don't talk to each other while they work.
Once they are done, you collect all 8 solutions.
You look at them, pick the best one, or take a vote to decide the final answer.

2. The "One Detective with a Notebook" (Sequential Sampling)

Imagine you hire one detective.

You give them the case file. They write a solution.
You read their solution, say, "Hmm, try again," and give them their own previous answer back to them.
They read their old answer, try to fix it, and write a new one.
You repeat this 8 times. The detective is constantly looking at their own previous attempts, trying to refine them.

The Big Surprise:
Intuitively, you might think the "One Detective" (Sequential) should be better. Why? Because they get to learn from their mistakes in real-time. They have more "context" and can build on their previous work. It's like having a conversation where you can correct yourself.

But the paper found the opposite. The "Team of Explorers" (Parallel) almost always wins. The single detective keeps getting stuck in a loop, making tiny tweaks to a bad idea, while the team of 8 comes up with 8 completely different, often better, ideas.

Why does the "One Detective" fail? (The Three Suspects)

The researchers acted like detectives to figure out why the single detective was losing. They had three theories:

Suspect #1: The "Voting System" (Aggregation)

Theory: Maybe the Team wins just because they have a "voting system" at the end to pick the best answer, while the Single Detective just has to stick with their last guess.
The Verdict: Not the main culprit. Even when the researchers gave the Single Detective a way to pick the best of their 8 attempts, they still lost. The voting system helps, but it's not the magic ingredient.

Suspect #2: The "Heavy Backpack" (Context Length)

Theory: Maybe the Single Detective is losing because they have to carry a heavy backpack of all their previous answers. The more they try, the longer the list of past mistakes gets, and the AI gets confused or tired by the sheer amount of text.
The Verdict: Not the main culprit. The researchers tested this by giving the Team of Explorers a heavy backpack of fake text (irrelevant code) to read. The Team didn't care; they still performed well. The length of the text wasn't the problem.

Suspect #3: The "Tunnel Vision" (Lack of Exploration)

Theory: This is the real villain. When the Single Detective looks at their previous answer, they get lazy. They start copying their own mistakes. Instead of thinking of a new way to solve the problem, they just tweak the old one slightly. They get stuck in a "Tunnel Vision."
The Verdict: This is the winner. The researchers looked inside the AI's "brain" (using something called attention maps) and saw that the AI was literally copying patterns from its previous answers. It wasn't exploring new paths; it was just echoing itself.

The "Lazy Student" Analogy

Think of the Single Detective as a student taking a test who is allowed to look at their own previous answers.

Round 1: They guess an answer. It's wrong.
Round 2: They look at Round 1. They think, "Okay, I'll just change one number." They don't realize the whole method was wrong.
Round 3: They look at Round 2. They change another number.
Result: They are just polishing a broken car. They never stop to realize they need a different engine entirely.

Meanwhile, the Team of Explorers are 8 different students. One might try a completely different engine. Another might try a different fuel. One of them just happens to get it right.

The Solution?

The paper suggests that if you really want the Single Detective to win, you have to force them to stop being lazy. You need to give them high-quality feedback that proves they are wrong (like showing them a specific error message that they can't ignore). If you do that, they can sometimes catch up to the team. But without that harsh, specific feedback, they will just keep copying their own mistakes.

The Takeaway

When using AI to solve hard problems, diversity is better than iteration.
It's better to ask the AI to generate 10 totally different ideas at once and pick the best one, rather than asking it to try to fix its own idea 10 times in a row. The AI gets "lazy" and stuck in its own patterns when it has to look at its own past work.

1. Problem Statement

Large Reasoning Models (LRMs) have demonstrated significant improvements in complex tasks like mathematics and coding by scaling "thinking traces" (Chain-of-Thought). However, simply extending the thinking budget has diminishing returns and can lead to "overthinking" or inverse scaling. To further scale performance, researchers typically employ two strategies:

Parallel Sampling: Generating $N$ independent solutions from the same prompt and aggregating them (e.g., via majority voting or Best-of-N).
Sequential Sampling: Generating solutions in a chain where each subsequent solution is conditioned on the previous one(s), often with self-refinement or error feedback.

The Core Question: Empirical observations suggest that Parallel Sampling consistently outperforms Sequential Sampling, even though Sequential Sampling theoretically possesses greater representation power (as it can refine and correct errors iteratively). This paper aims to rigorously investigate the underlying causes of this performance gap.

2. Methodology

The authors conducted extensive empirical studies across multiple model families (Qwen3, DeepSeek-R1 distilled models, Gemini 2.5) and domains (Math competitions like AIME2025 and Code generation via LiveCodeBench).

They formulated and tested three primary hypotheses to explain the performance gap:

The Aggregation Hypothesis: Parallel sampling wins because it uses an aggregation operator (e.g., voting) to select the best answer, whereas sequential sampling simply takes the final output.
The Context Length Hypothesis: Sequential sampling suffers because the input context grows longer with each step (accumulating previous thoughts and answers), potentially overwhelming the model or diluting attention.
The Exploration Hypothesis: Sequential sampling leads to "limited exploration" because the model conditions on previous answers, causing it to get stuck in local optima or repeat previous mistakes (laziness).

Experimental Design:

Math Tasks: Used majority voting for aggregation. Compared parallel sampling against Markov sequential (using only the last answer) and Auto-regressive sequential (using full history).
Code Tasks: Used Best-of-N aggregation based on test case execution. Compared sampling strategies with and without "running error" feedback.
Mechanistic Analysis: Visualized attention maps to identify "induction heads" (mechanisms that copy patterns from previous tokens) in Transformer models.
Control Experiments:
- Applied aggregation to sequential sampling to test Hypothesis 1.
- Injected irrelevant context (random code) to test Hypothesis 2.
- Measured solution diversity via embedding similarity and analyzed "laziness" (verbatim repetition).

3. Key Contributions & Findings

A. Disproving the Aggregation Hypothesis

The authors applied aggregation operators (Majority Voting and ideal Best-of-N with a perfect verifier) to sequential sampling.

Result: While aggregation smoothed the scaling curve for sequential sampling, it did not close the performance gap. In some cases, the gap even widened.
Conclusion: The superiority of parallel sampling is not merely due to the aggregation step.

B. Disproving the Context Length Hypothesis

The authors compared input context lengths and total token usage.

Result: Even when auto-regressive sequential sampling used significantly longer contexts (up to 20k tokens), the performance gap persisted. Furthermore, experiments adding irrelevant context (up to 32k tokens) to parallel sampling showed no significant degradation in performance.
Conclusion: Long context length is not the primary driver of the performance gap.

C. Validating the Exploration Hypothesis (The Main Culprit)

The study found that Sequential Sampling suffers from a lack of exploration, characterized by model "laziness."

Observation: In sequential chains, models tend to generate solutions that are semantically very similar or even verbatim copies of previous attempts, even when errors are present.
Evidence:
- Embedding Similarity: Solutions in sequential chains had significantly higher cosine similarity (0.96–0.99) compared to parallel samples (0.95–0.97), indicating less diversity.
- Mechanistic Analysis: Attention map visualizations revealed strong induction heads in sequential sampling. These heads actively attend to and copy patterns from previous solutions ( $y_1, y_2...$ ) when generating the current solution ( $y_n$ ), effectively locking the model into a suboptimal reasoning trajectory.
- Laziness: Models often failed to explore alternatives when stuck on a specific error, repeating the same code or logic despite feedback.

D. The Role of High-Quality Feedback

When the authors provided high-quality feedback (specifically, running errors from both public and private test cases for coding tasks), sequential sampling performance improved significantly, narrowing the gap with parallel sampling.

Implication: The performance gap is largely due to the model's inability to break out of local optima without strong, diverse guidance. Parallel sampling naturally provides this diversity by starting fresh from the prompt.

4. Significance

Theoretical Insight: The paper shifts the understanding of LLM scaling from purely "more tokens" to "diversity of thought." It identifies that conditioning on previous outputs in a sequential chain induces a "Tunnel Vision" effect (via induction heads), preventing the model from exploring the solution space effectively.
Practical Guidance: For deploying LRMs in high-stakes scenarios (math, coding), Parallel Sampling is currently the more reliable strategy for scaling performance.
Future Directions: To make Sequential Sampling viable, future work must focus on mechanisms that force exploration (e.g., stronger error signals, diversity-promoting penalties, or hybrid approaches) to break the pattern-copying behavior of induction heads.

Summary

The paper concludes that Parallel Sampling outperforms Sequential Sampling primarily because Sequential Sampling restricts solution exploration. The model's tendency to copy previous patterns (induction heads) and its "laziness" in refining answers lead to a lack of diversity, which aggregation and context length cannot compensate for. Parallel sampling succeeds because it maintains independent, diverse reasoning trajectories.