Technical Summary: The Readout Shortcut in Small Language Models

Problem Statement

Chain-of-thought (CoT) prompting is widely recognized as necessary for arithmetic reasoning in small language models (SLMs), yet empirical observations show that shuffling the logical order of CoT steps often preserves performance. This creates a tension: if the logical sequencing of steps is not the primary driver of accuracy, what mechanism allows the model to generate the correct answer? Prior work has established that CoT can be unfaithful (e.g., biased demonstrations alter answers despite correct reasoning), but the specific signal the readout mechanism uses, its location within the network, and whether it actively suppresses available computation remain unexplained.

Methodology

The authors investigate this phenomenon using three instruction-tuned 1–3B parameter models: Qwen2.5-1.5B-Instruct, Llama-3.2-1B-Instruct, and Gemma-2-2B-it. The study focuses on the GSM8K dataset (first 500 problems) and extends to GSM-Symbolic and BIG-Bench-Hard (BBH).

The core experimental protocol involves prefix completion (teacher-forced injection). A CoT prefix (potentially modified) is injected into the assistant turn, followed by the answer delimiter ####. The model then generates the final answer token greedily. This isolates the "readout" stage from the generation-time confounds of reasoning.

Key experimental interventions include:

1. Corruption Decomposition: Systematically corrupting numbers in the CoT prefix to isolate the contribution of the gold answer's presence versus intermediate computation.
2. Causal Ladder: A series of conditions (e.g., replacing the trailing number with a distractor, truncating the CoT before the answer, removing the trailing number entirely) to determine if the model relies on a "copy" channel or retained-context completion.
3. Positional and Content Controls: Injecting wrong numbers (distractors) at the trailing position, varying the framing context (answer-relevant vs. bare numbers), and shuffling steps at different granularities (token, word, step).
4. Mechanistic Ablation: Zero-ablation and mean-ablation of attention heads, and activation patching, to identify architecture-specific circuits responsible for the readout.
5. Scaling and Generalization: Testing larger models (7–8B) and non-arithmetic tasks to determine the boundaries of the phenomenon.

Key Contributions and Results

1. The Positional Readout Shortcut

The primary finding is that in 1–3B models, the CoT readout is dominated by a positional shortcut: the model copies the number occupying the trailing position immediately before the answer delimiter, provided it appears in an answer-relevant context.

• Gold-Answer Dependence: The presence of the gold answer in the trailing position accounts for 54–92 percentage points (pp) of accuracy (89–92% of the model's teacher-forcing ceiling).
• Masking of Computation: When a trailing number is present, the model ignores correct intermediate steps. Replacing the trailing gold number with a wrong value (distractor) collapses accuracy to near-zero, even if all intermediate reasoning is correct.
• Override Mechanism: The "copy channel" takes precedence over retained-context completion. Removing the trailing distractor (via truncation) recovers 5–32 pp of accuracy, demonstrating that the model can compute from retained context but is actively suppressed by the positional copy signal.

2. Content Gating and Architecture Differences

While the positional mechanism is universal across the tested 1–3B models, the degree of content selectivity (the ability to reject a wrong number) varies:

• Qwen & Llama: Exhibit novelty-permissive copying. They copy wrong numbers placed at the end 87–95% of the time, regardless of whether the number is semantically plausible.
• Gemma: Shows strong content gating. It rejects novel distractors 81–88% of the time (copying them only 12–19% of the time), indicating a more selective readout mechanism.
• Framing Sensitivity: The shortcut is "answer-context-gated." Bare numbers at the end do not trigger copying; the number must be framed within answer-relevant text (e.g., "the answer is X").

3. Mechanistic Localization

Head-level ablation reveals architecture-specific profiles for the copy circuit:

• Llama: Localized sensitivity (specific heads in layers 10–15).
• Qwen: Distributed sensitivity (requiring ablation of ~17 heads to see a phase transition).
• Gemma: Concentrated but compensable (dominated by a single head, L14H4, in zero-ablation).
• Activation Patching: Patching the top-20 heads in Qwen recovers 61% of the accuracy gap caused by shuffling, confirming a causal link between these heads and the readout.

4. Scaling and Generalization Boundaries

• 7–8B Models: As models scale to 7–8B, the gold-presence dependence remains, but content selectivity emerges. The rate of copying wrong distractors drops significantly (below 32%), and a substantial "off-copy" pathway (relying on intermediate computation) reappears.
• Non-Arithmetic Tasks: The shuffle tolerance collapses on non-arithmetic tasks (e.g., logical deduction in BBH) where no copyable numeric answer occupies the trailing position. Retention drops to chance levels, confirming the shortcut is specific to the arithmetic/numeric readout pattern.
• Base Models: The mechanism exists in base model weights, suggesting it is not solely an artifact of instruction tuning, though tuning amplifies the effect.

Significance and Claims

The paper claims to resolve the "tension" between CoT necessity and shuffle tolerance by identifying a specific failure mode in small models: the readout stage is largely decoupled from the reasoning stage.

• Implications for Oversight: The findings undermine the premise of step-level CoT monitors (e.g., process-reward models) for 1–3B models. Since the readout prioritizes the trailing token over intermediate computation, a model can generate a logically correct chain of thought but output a wrong answer based on a trailing distractor, or vice versa. Step-level rewards may operate on a signal weakly coupled to the actual answer-producing computation.
• Diagnostic Utility: The authors propose that monitoring whether the answer matches the last CoT number serves as a simple diagnostic for copy-dominance.
• Safety and Vulnerability: The paper highlights a vulnerability where adversaries could construct CoTs with correct intermediate steps but a trailing distractor to evade step-level oversight, routing the final answer through the positional shortcut.

The authors remain modest regarding the scope, noting that the phenomenon is primarily observed in 1–3B arithmetic tasks and that content selectivity recovers at larger scales (7–8B) and in reasoning-trained models (e.g., DeepSeek-R1-Distill), though the positional mechanism persists even there.