🤖 machine learning

When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling

This paper argues that test-time scaling in reasoning systems faces fundamental "modal" and "correlation ceilings" where excessive sampling fails to improve answer selection and can even degrade performance, suggesting that the true bottleneck lies in recognizing correct answers rather than generating them.

Original authors: Yong Yi Bay, Kathleen A. Yearick

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yong Yi Bay, Kathleen A. Yearick

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a very difficult riddle. You have a smart assistant (the AI) who can try to answer it. The paper argues that simply asking this assistant to try more and more times is often a waste of time and money, and can even make the final result worse.

Here is the breakdown of why "more sampling" hits a wall, explained through three simple analogies.

1. The "Crowded Room" vs. The "Best Answer" (The Selection Ceiling)

Imagine you ask a room full of people to solve a riddle.

Coverage (The "Someone Knows It" Metric): If you ask 1,000 people, the odds are high that at least one person in the room knows the right answer. As you add more people, the chance that "someone knows it" keeps getting closer to 100%. This is what researchers call Coverage. It looks like progress.
Selection (The "Who Do We Trust?" Metric): But in the real world, you can't just say, "Okay, someone in the room knows it, but I don't know who." You have to pick one person to give the final answer. Usually, you pick the person whose answer is the most popular (the "majority vote").

The Problem:
If the riddle is tricky, the room might be full of people giving the same wrong answer because they all got confused in the same way.

If you ask 10 people, maybe 6 say "Blue" (wrong) and 4 say "Red" (right). You pick "Blue."
If you ask 1,000 people, maybe 600 say "Blue" and 400 say "Red." You still pick "Blue."
The Ceiling: No matter how many more people you add, the "Blue" crowd just gets louder. The correct answer ("Red") might be in the room, but it's not the most common answer. The paper calls this the Modal Ceiling. Once the crowd agrees on a wrong answer, adding more people just makes the model more confident in its mistake.

2. The "Echo Chamber" (The Correlation Ceiling)

Now, imagine you are trying to guess the average height of people in a city by asking a few people.

The Ideal: You ask 100 strangers from different neighborhoods. You get a very accurate average.
The Reality: You ask 100 people, but they are all from the same family. They share the same genes and diet. They are all roughly the same height.

In the paper's terms, when an AI tries to solve a problem 100 times, those 100 attempts aren't 100 independent guesses. They are like 100 members of the same family. They are correlated. They tend to make the same mistakes or get stuck in the same "thinking trap."

The Ceiling:
Because they are so similar, asking for 1,000 attempts from the same model is like asking 1,000 copies of the same person. You aren't getting 1,000 new pieces of information; you are just getting the same information repeated.

The paper says there is a Correlation Ceiling. If the attempts are 10% similar (correlated), asking for 1,000 attempts is only as useful as asking for about 10 truly independent attempts.
After a certain point, every extra attempt you pay for is just "noise." It costs money but adds zero new value.

3. The "Hidden Gap" (The Identifiability Gap)

This is the most important part of the paper. There is a gap between:

What the model can do: It can generate the right answer (it's in the room).
What the model will give you: It won't pick the right answer because the wrong answer is more popular.

The paper calls this the Identifiability Gap.

The Trap: We see the "Coverage" line going up (the model is finding the right answer more often) and think, "Great! The model is getting smarter!"
The Reality: The "Selection" line (what we actually get) has hit a wall. The model is getting better at finding the right answer, but it's not getting better at recognizing it as the right one.

The Practical Takeaway: "Stop Sooner"

The paper concludes that we are overthinking (or over-sampling).

For checking if an answer exists: Keep sampling if you have a perfect "verifier" (like a math checker) that can spot the right answer among the wrong ones.
For picking a final answer: Stop very early. The paper suggests that for most tasks, you only need a few dozen attempts. After that, you are just paying to make the model more confident in the wrong answer.
For measuring performance: If you want to know how good a model is on a test, you don't need 1,000 tries per question. Because the tries are so similar (correlated), 1,000 tries might only count as 2 or 3 real tries. You are better off testing the model on 1,000 different questions instead of 1,000 tries on one question.

In short: The bottleneck isn't generating the right answer anymore; it's recognizing it. Throwing more compute at the problem doesn't fix that; it just makes the model louder about its mistakes.

Technical Summary: When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling

Problem Statement

Current reasoning systems often employ test-time scaling, specifically repeated sampling (generating $n$ responses to a single prompt), to improve performance. The prevailing metric for success is coverage (pass@k), defined as the fraction of problems where at least one of the $n$ samples is correct. Coverage curves typically rise steadily as $n$ increases, suggesting that simply increasing the sampling budget yields continuous capability gains.

However, deployed systems must return a single answer without access to a perfect verifier to certify correctness. This necessitates selection (e.g., via majority vote, self-consistency, or reward models). The paper identifies a critical disconnect: while coverage continues to climb with more samples, selection accuracy often stalls or even declines. The core problem is that the community conflates the ability to generate a correct answer with the ability to recognize and select it. The paper argues that beyond a certain point, extra sampling adds computational cost without improving, and potentially harms, the final selected answer.

Methodology

The paper reframes test-time sampling through the lens of cluster sampling from survey statistics, rather than independent draws.

Cluster Sampling Framework: A single prompt and its $n$ repeated attempts are treated as a cluster. The attempts are modeled as exchangeable binary variables (correct/incorrect) rather than independent and identically distributed (i.i.d.) variables.
Intraclass Correlation ( $\rho$ ): The dependence between attempts on the same problem is quantified by the intraclass correlation $\rho$ $ρ$ .
- Between-problem correlation ( $\rho_b$ ): Arises from the heterogeneity of problem difficulty.
- Within-problem correlation ( $\rho_w$ ): Arises from the model's internal state or decoding configuration.
Effective Sample Size ( $n_{eff}$ ): Using the design effect ( $deff = 1 + (n-1)\rho$ ), the paper derives the effective number of independent samples:
$n_{eff} = \frac{n}{1 + (n-1)\rho}$
This quantity saturates at a correlation ceiling of $1/\rho$ as $n \to \infty$ .
Separation of Metrics: The analysis explicitly decouples three distinct objectives:
- Coverage: The probability of generating at least one correct answer.
- Selection: The probability that the most common answer (mode) is correct.
- Estimation: The precision of estimating the benchmark mean accuracy.

Key Contributions

1. The Correlation Ceiling (Estimation Limit)

The paper establishes that for estimating benchmark accuracy, the effective number of samples cannot exceed $1/\rho$ .

Mechanism: As $n$ increases, the marginal value of each additional sample decays quadratically ( $O(1/n^2)$ ).
Result: Even with massive budgets (e.g., $10^4$ samples), if $\rho \approx 0.5$ , the effective information is equivalent to only $\approx 2$ independent samples. This explains why benchmark accuracy estimates plateau despite massive sampling efforts.

2. The Modal Ceiling (Selection Limit)

The paper identifies a distinct ceiling for selection methods (self-consistency, plurality voting) that is independent of the correlation ceiling.

Mechanism: Selection converges to the mode of the answer distribution ( $a^*_q$ ). If the most common answer is incorrect ( $a^*_q \neq c_q$ ), increasing $n$ does not fix the error; it merely makes the model more confident in the wrong answer.
Result: Selection accuracy is capped at the modal-hit rate ( $\pi_{mode}$ ), the probability that the model's most frequent answer is correct.
Anti-Scaling: On problems where the mode is wrong but the correct answer is reachable, increasing $n$ causes selection accuracy to drop toward 0 while coverage rises toward 1. This phenomenon is termed anti-scaling.

3. The Identifiability Gap

The paper defines the identifiability gap as the set of problems where the correct answer is present in the sample pool (high coverage) but is not the most frequent answer (low selection).

This gap represents the "solvable but unselectable" problems.
The paper demonstrates that this gap is significant in real-world logs, where coverage can reach 1.00 while self-consistency plateaus at 0.87 (GSM8K) or 0.45 (MATH).

4. Empirical Validation on Real Logs

The authors validate their theoretical framework using released sampling logs:

Brown et al. [2] (Independent Draws): Analyzed GSM8K and MATH datasets. Found high between-problem correlation ( $\hat{\rho}_b \approx 0.4\text{--}0.6$ ), confirming the correlation ceiling for estimation. Observed the divergence between coverage (rising to 1.0) and self-consistency (plateauing).
Beeching et al. [20] (Dependent Draws): Analyzed MATH-500 with 256 samples per problem. Found that within-problem correlation ( $\hat{\rho}_w$ ) is near zero, but the answer distribution is highly concentrated (median ~13 distinct values). This confirmed that the selection ceiling is driven by answer mode collapse, not run-to-run drift.

Results

Coverage vs. Selection: Coverage follows a power-law rise (due to difficulty heterogeneity) and has no hard ceiling other than model capability. Selection, however, hits a hard wall at $\pi_{mode}$ .
Diminishing Returns: The marginal value of the $n$ -th sample decays rapidly. For estimation, the budget should be capped at $n \approx 1/\rho$ . For selection, the budget should be capped at the effective number of distinct answers.
Inefficiency of Large $n$ : Sampling $10^4$ times per problem yields the statistical precision of only ~2 independent samples for benchmark estimation.
Anti-Scaling: In scenarios where the model's most common answer is wrong, increasing $n$ actively degrades the selected output's accuracy.

Significance and Claims

The paper claims to provide a citable derivation rather than a new algorithm. Its primary significance lies in:

Reframing the Bottleneck: It shifts the focus of test-time scaling from "generating a right answer" to "recognizing one." The bottleneck is not the sampling budget, but the identifiability of the correct answer within the distribution.
Correcting Misinterpretations: It explains why "more sampling" often fails to scale linearly in practice, attributing this to the correlation ceiling (for estimation) and the modal ceiling (for selection).
Actionable Metrics: It proposes reporting the effective number of samples ( $n_{eff}$ ) alongside the nominal count $n$ to reveal the true information content of a sampling run.
Budget Allocation: It argues that compute budgets should be goal-dependent:
- For estimation: Stop at $n \approx 1/\rho_b$ and spend remaining compute on more problems.
- For selection: Stop once the plurality stabilizes (often within a few dozen samples) and focus on decorrelation (e.g., temperature, prompt diversity) rather than increasing $n$ .
- For coverage (with a verifier): Sampling can continue indefinitely as there is no within-problem ceiling.

The paper concludes that the "modal ceiling" and "correlation ceiling" are fundamental limits that render indiscriminate test-time scaling inefficient. The real challenge is not generating more samples, but designing systems that can better distinguish the correct answer from the modal error.