Technical Summary: Hard or Just Unreached? Diagnosing the Sampling Blind Spot in Math-Reasoning Difficulty Estimation

1. Problem Statement

Current pipelines for mathematical reasoning—encompassing Reinforcement Learning (RL) with verifiable rewards, data curation, synthetic curriculum generation, and difficulty annotation—rely heavily on pass@k as the canonical signal for per-example difficulty. In this framework, an example is labeled "hard" if no sampled chain among $k$ attempts reaches the gold answer. Consequently, examples with $pass@k=0$ are often filtered out, downweighted, or treated as intrinsically unsolvable.

The paper argues that this reliance on stochastic sampling creates a persistent blind spot on the hardest stratum of problems. The core hypothesis is that examples flagged as "hard" by sampling-only estimators are not necessarily intrinsically difficult; rather, they may be structurally unreachable on the stochastic axis but reachable via deterministic decoding at matched computational budgets. The authors posit that the "hard" label conflates "intrinsically hard" with "unreached by stochastic sampling."

2. Methodology

Experimental Setup

The study evaluates four open-weight instruction-tuned models (Qwen-2.5-3B, Llama-3.2-3B, Llama-3.1-8B, Mistral-Nemo-12B) across three reasoning benchmarks (GSM8K, MATH, MMLU-Pro).

• Sampling Regime: Six independent chains ($k=6$) generated with temperature $T=0.7$ and top-p $p=0.9$.
• Deterministic Regime: A "six-chain deterministic regime" constructed by running greedy decoding ($T=0$) with activation grafting.
• Activation Grafting: This is a diagnostic intervention, not a decoding method. At layer $\ell=26$ (the last prompt token), the model's hidden state ($h_{last\_token}$) is replaced with one of several cheap, parameter-free synthetic vectors before continuing the forward pass. The graft vectors include:
  • $g_B$: Greedy baseline (no graft).
  • $g_Z$: Zero vector.
  • $g_R$: Norm-matched Gaussian.
  • $g_{RUNIT}$: Random unit direction.
  • $g_{SHUF}$: Hidden-dimension permutation.
  • $g_{BOS}$: BOS-token activation.
  • $g_{AVG}$: Mean over prompt-token activations.
  • $g_{PREV}$: State from the previous layer at the same position.
• Matched Compute: The comparison is strictly controlled for computational cost. The "six-chain deterministic regime" performs the same number of forward passes (prefill + decode) as the six-chain sampling regime.

Diagnostic Logic

The authors isolate the $pass@6=0$ stratum (examples where no sampling seed succeeds). They then measure how many of these specific examples are solved by the deterministic regime ($R_k$). If a significant fraction of these "hard" examples are solved by deterministic grafts, the "intrinsic hardness" assumption is falsified.

3. Key Contributions and Findings

(1) Greedy is Competitive at $k=1$

Contrary to the view that greedy decoding is a weak baseline compared to sampling, the authors find that at $k=1$, greedy decoding is competitive with or superior to the mean single-sample accuracy across most cells. The gap reaches up to +2.8% on Llama-3.2-3B/MATH. This suggests that the default single-pass baseline for self-consistency is already weaker than a deterministic alternative on the same compute.

(2) The Sampling Blind Spot is Structural

On the $pass@6=0$ stratum (which comprises 5.1%–43.5% of prompts depending on the model/benchmark), a deterministic regime at matched compute recovers a non-trivial fraction of examples:

• Recovery Rate: The six-chain deterministic regime ($R_6$) solves 10.3%–22.9% of the examples that six sampling seeds collectively miss across the eight free-form math cells (GSM8K and MATH). Across all twelve cells, this ranges from 10% to 29%.
• Scaling: Recovery scales with the deterministic budget. $R_1$ (greedy alone) recovers only 0–9%, but adding diverse grafts increases recovery significantly, indicating that the "hard" slice is accessible via multiple distinct deterministic trajectories.
• Mechanistic Distinctness: The recovery is not driven by a single "privileged" vector. Cross-kind fix-set Jaccard similarity is low ($\le 0.47$) across all cells, proving that different graft types recover disjoint subsets of the stratum.

(3) Mechanism: Residual-Stream Perturbation, Not Attention Rerouting

The paper investigates why grafts succeed. Instrumentation of the Qwen-2.5-3B model reveals:

• Hidden-State Divergence: Grafting at layer 26 causes the hidden state to diverge significantly from the baseline (L2 distance grows monotonically through remaining layers; cosine similarity remains low).
• Attention Stability: The graft has a negligible direct effect on attention weights ($\sim 10^{-5}$ difference).
• Conclusion: The graft acts as a different initial condition for the remaining transformer blocks. The model's forward pass does not project these perturbations back onto the baseline trajectory; instead, it carries them forward, locking in distinct downstream trajectories. This confirms that the "hard" slice is structurally identifiable in the residual stream.

(4) Practical Utility

The authors propose two deployable strategies based on these findings:

1. Matched-Cost Substitution: Replacing one sampling pass with a single deterministic graft (specifically the mean-prompt-activation graft, $AVG$) in a curation pipeline covers more of the dataset than using all passes for sampling in the majority of configurations.
2. Label-Free Identification: A simple probe using chain disagreement among three chains (greedy + two sampling seeds) can rank recoverable items (those in the $pass@6=0$ stratum but solvable deterministically) with a 3–5× lift on free-form math tasks, without requiring gold labels.

4. Significance and Claims

The paper claims that sampling-derived difficulty estimates systematically mislabel a structurally identifiable fraction of the hardest stratum.

• Redefining "Hard": The "hard" label in current pipelines often signifies "unreached on the stochastic axis" rather than "intrinsically hard."
• Impact on Pipelines:
  • Benchmark Construction: "Hardest" buckets in benchmarks like GSM8K and MATH are partly artifacts of the decoding regime, not just the items.
  • Data Curation: Rejection-sampling pipelines may discard solvable problems as "unsolvable," wasting potential training data.
  • RL and Verifier Training: Preference datasets built from sampled chains inherit this blind spot, contributing only negative signals for items that a deterministic chain could solve.
• Modesty of Claims: The authors explicitly state they do not propose activation grafting as a new inference method for deployment. Instead, it is a diagnostic tool to expose a slice of the problem space that is currently invisible to standard sampling-based metrics. The unmodified model does not reach these items under ordinary inference; the slice is only reachable via specific internal interventions.

5. Limitations

• Incomplete Recovery: Even with eight deterministic chains, 66–88% of the $pass@6=0$ stratum remains unrecovered. These may be genuinely hard or reachable via diversity axes not tested.
• Scope: The study is limited to four models and three benchmarks.
• Label-Free Precision: While chain disagreement can rank recoverable items, the precision guarantee for label-free identification is not fully quantified and degrades on multiple-choice benchmarks (MMLU-Pro) due to chance agreement.
• Noise: On the smallest strata (e.g., ~50 examples), absolute recovery counts are small, though rates remain consistent.

In summary, the paper demonstrates that the "hardest" problems in math reasoning benchmarks are often not intrinsically unsolvable but are simply missed by stochastic sampling, a fact that can be revealed through deterministic interventions on the residual stream.