Technical Summary: Calibrated e-CUSUM Decoding for Quantized Reasoning Models

Problem Statement

Low-bit quantization (e.g., INT4) enables the deployment of small reasoning models on commodity hardware but significantly degrades their chain-of-thought capabilities. These models suffer from early cascading errors, inflated reasoning lengths, and a tendency to "overthink" or fail to terminate. A common intuitive remedy is to monitor the decoder during generation and intervene when the trajectory appears to fail.

The paper identifies a critical flaw in the most natural candidate for such a monitor: a martingale constructed from token log-probabilities ($X_t = \log p(w_t | w_{<t})$).

• The Flaw: Under the model's own sampling law, the increment $D_t = \log p(w_t) + H_t$ (where $H_t$ is Shannon entropy) is a mean-zero martingale by construction.
• The Consequence: This observable measures the self-consistency of sampling rather than the health of the trajectory. In a "confident repetition loop" (a common failure mode where the model repeats tokens with high probability), $p(w_t) \approx 1$ and $H_t \approx 0$, resulting in $D_t \approx 0$. Consequently, the martingale remains silent precisely when the model is confidently wrong. Furthermore, because it reacts to high-variance (high-entropy) stretches, it triggers alarms on uncertain but potentially correct reasoning paths, leading to a high false-alarm rate (93–95% in uncalibrated tests).

Methodology

The authors propose a replacement controller built on two components: a degeneration-aware alarm score and a calibrated sequential detector.

1. Alarm Score ($a_t$)

Instead of relying solely on entropy, the controller computes a bounded score $a_t \in [0, 1]$ that fuses two signals:
$$a_t = \min(1, w_{rep} r_t + w_{ent} u_t)$$

• $u_t$ (Uncertainty): Flags spikes in Shannon entropy relative to recent history.
• $r_t$ (Repetition): Flags verbatim repetition by measuring the coverage of recurring n-grams in a sliding window. Crucially, $r_t$ approaches 1 during a confident loop even when entropy is near zero, addressing the blind spot of log-probability martingales.
• Weights: Empirically set to $w_{rep} = 0.7$ and $w_{ent} = 0.3$.

2. Calibrated e-CUSUM Detector

The paper utilizes a betting construction (Shafer, 2021) to monitor $a_t$.

• Raw e-Process: A non-negative product process $E^{raw}_n = \prod (1 + \lambda(a_t - \mu_0))$. Under the conditional null hypothesis that the generation is healthy ($E[a_t | \mathcal{F}_{t-1}] \le \mu_0$), this process satisfies Ville's inequality, offering a time-uniform false-alarm guarantee.
• Implementation (CUSUM-floored): The deployed statistic is a log-CUSUM with a floor: $S_n = \max(0, S_{n-1} + \log(1 + \lambda(a_n - \mu_0)))$. An alarm is raised when $S_n \ge \tau$.
• Calibration: Because $a_t$ is strongly history-dependent and autocorrelated, the theoretical conditional null is difficult to certify. The authors perform offline calibration by setting the baseline $\mu_0$ to the 90th percentile of $a_t$ observed on healthy (FP16, correct) reference traces. This transforms the detector from an indiscriminate alarm into a selective one.

3. Remediation

When an alarm is triggered:

1. Backtracking: The last $k$ tokens are truncated, and the key-value cache is rebuilt.
2. Resampling: The model resumes with adjusted parameters:
   • Min-p sampling is widened or tightened (rather than inflating temperature, which increases randomness).
   • On repeated alarms, a correction marker ("Wait, let me re-check...") is injected.
3. Budget: A hard cap limits the number of interventions to prevent infinite loops.

Key Results

Experiments were conducted on the GSM8K dataset using DeepSeek-R1-Distill-Qwen-1.5B in both FP16 and INT4 quantization.

• Failure Mode Analysis: Contrary to the intuition that confident looping is the primary failure, the study found that non-termination is the dominant failure mode for INT4 models on GSM8K (up to 49% of incorrect traces exhaust the token budget). Verbatim loops were rare ($\le 4\%$ of tokens even on failing traces).
• Selectivity via Calibration:
  • An uncalibrated monitor ($\mu_0 = 0.15$) fired on 93–95% of generations, rendering it useless.
  • A calibrated monitor ($\mu_0 = 0.41$) achieved a precision of $\approx 0.6$ and recall of $\approx 0.46$ (F1 $\approx 0.3$) against a base rate of 0.38 bad traces.
• Degeneration Metrics: The controller reduced consecutive repetition rates in INT4 from 0.010 to 0.003 and eliminated severe loops (from 1% to 0%). It also slightly reduced truncation rates (22% to 19%).
• Accuracy:
  • FP16: Accuracy improved from 68% to 72% (McNemar $p=0.48$).
  • INT4: Accuracy improved from 63% to 69% (McNemar $p=0.18$).
  • Cost: The intervention increased token usage by 28% and wall-clock time by 30%.
  • Conclusion: The accuracy trend is positive but statistically inconclusive at $n=100$. The authors explicitly state they do not claim an accuracy improvement, noting that the gain could be attributed to the increased token budget rather than the controller itself.

Significance and Claims

The paper positions itself as a preliminary, methodological study with the following specific contributions:

1. Negative Result: A clear, analytical, and empirical demonstration that token log-probability martingales are fundamentally inadequate for decoding monitors because they are blind to confident degeneration.
2. Methodological Replacement: A training-free, calibrated e-CUSUM controller that successfully detects degeneration by combining entropy with explicit repetition signals.
3. Honest Evaluation: The authors emphasize that the deployed statistic is an empirical change detector rather than a strictly valid e-process under the stated $\delta$, due to the history-dependence of the alarm score and the marginal nature of the calibration.
4. Insight on Failure Modes: The finding that non-termination, not looping, is the primary failure mode for quantized models on GSM8K challenges the initial motivation for the method and suggests that remediation strategies must address "rambling" as much as repetition.

The authors release all code, calibration tools, and traces to facilitate larger-scale studies required to definitively answer the accuracy question. They caution that the method adds computational cost and that any future claims of accuracy improvement must be made against compute-matched baselines (e.g., self-consistency with the same token budget).