Technical Summary: Causal Registers in Scratchpad Reasoning

Problem Statement

Process supervision aims to align models by rewarding intermediate reasoning steps (e.g., scratchpads, chain-of-thought) rather than just final answers. However, a critical gap exists between legibility and causality: a model may write plausible intermediate states that correlate with the correct answer without actually using those states as inputs for subsequent computations. If the written text is merely a post-hoc justification or a correlated signal, it cannot be effectively audited or intervened upon.

The central question addressed is: Does a model actually compute from the intermediate states it writes? Specifically, if an internal representation of a written state is altered while the visible text remains fixed, does the model's next step change in accordance with the known update rule of that state?

Methodology

1. Controlled State-Tracking Task

The authors utilize a finite transition system with known update rules to create an unambiguous causal test.

• Task Structure: The model is given a starting state and a sequence of moves. It must track the state after each move.
• State Representation: The state is defined as $v | p$, where $v$ is a visible coordinate (one of four grid cells) and $p$ is a "phase bit" ($0$ or $1$).
• Order Sensitivity: The phase bit $p$ is not determined solely by the visible coordinate $v$; it depends on the history of moves (the order of operations). This creates matched states (e.g., $11|0$ and $11|1$) that share the same visible text but differ in the internal phase.
• Models: The study uses Qwen2.5-Coder-7B as the primary model and Mistral-7B-v0.3 for replication.
• Training Conditions:
  1. Pretrained Base: No fine-tuning.
  2. Final-Answer Only: Fine-tuned to output only the final state.
  3. Running-State: Fine-tuned to write the full running state after every move before the final answer.

2. Intervention: Counterfactual State Editing

The core experimental intervention involves activation patching (specifically modifying the KV cache) at the token position corresponding to the current state.

• The Edit: The internal residual stream representation of the phase bit at the current state token is edited from $p$ to $1-p$.
• The Constraint: The visible token ID and the printed scratchpad text remain unchanged. The prefix supplied to the model is identical in both the clean and patched runs.
• The Test: The model is asked to predict the next state given the same upcoming move $m$.
  • Success Condition: The model predicts the phase bit resulting from applying move $m$ to the edited state $\tilde{s} = (v, 1-p)$.
  • Failure Condition: The model ignores the internal edit and predicts the result based on the original state $s = (v, p)$, or follows a generic bias.

3. Controls and Selectivity

To rule out alternative explanations (e.g., generic next-token steering or copying continuations), the authors employ two specific controls:

• Move-Swap Control: The edited state is held fixed, but the upcoming move is changed. A valid state-update mechanism should change its prediction based on the new move; a fixed bias would not.
• Conflicting-Continuation Control: A counterfactual state is injected from a different context where the future trajectory disagrees with the current move. If the model follows the current move, it is recomputing from the edited state; if it follows the injected context's future, it is merely copying a continuation.

Key Results

1. Causal Use of Scratchpad States

In the Running-State models (fine-tuned to write intermediate states), the internal edit successfully redirects the next-state computation:

• Qwen2.5-Coder-7B: The model predicts the next phase bit implied by the edited state on 80% of held-out examples (Q8 task) and 91% (D8 task).
• Controls: Pretrained and Final-Answer-only models remain near baseline (chance) levels, indicating they do not use the written state as a computational input.
• Random/Orthogonal Edits: Random patches or patches in orthogonal subspaces yield agreement rates near 0.02, confirming the effect is specific to the phase-bit subspace.

2. Selectivity and Recomputation

The Running-State models demonstrate high selectivity, proving they are not simply biasing tokens or copying:

• Move-Specificity: The prediction changes significantly when the move changes (Selectivity gaps: +0.57 for Q8, +0.68 for D8).
• Recomputation vs. Copying: When faced with a conflicting continuation, the model follows the current move (and thus the edited state) rather than the injected source future (Selectivity gaps: +0.59 for Q8, +0.81 for D8).

3. Mechanistic Analysis

• Compact Representation: The phase-bit information is captured by a low-rank subspace. In Q8, a rank-1 edit is sufficient; in D8, rank-2 is sufficient, far below the 3584-dimensional residual stream.
• Routing:
  • Q8: The update flows primarily through a single attention connection (from the current phase token to the next prediction layer). Ablating this connection removes 94% of the effect.
  • D8: The update uses a distributed, four-edge route. Ablating this specific route removes ~69% of the effect, showing that compact state variables can be processed via distributed mechanisms.
• Replication: These causal patterns replicate in Mistral-7B-v0.3, with Running-State models achieving 0.93–0.94 edited-state agreement.

4. Generated Prefixes

When the model generates the prefix itself (rather than being supplied with it), the effect persists but is slightly less stable in Q8 (0.66 one-step use) compared to D8 (0.90 one-step use), though it remains significantly above baseline.

Significance and Claims

The paper claims that running-state supervision (fine-tuning to output intermediate states) transforms a scratchpad from a mere transcript into a causal register within the model's computation.

• Operational Semantics: A useful intermediate for alignment must have operational semantics inside the model. The results show that for the trained models, changing the internal value associated with a written state causally changes the next step in a way consistent with the transition rule.
• Auditability: This establishes a "sharper goal for scratchpad oversight." Oversight is not just about making reasoning legible to humans, but about training models to use written states as actual inputs for their own computation. This allows for "evidence-distortion audits" where an overseer can theoretically change a state and verify if the model's subsequent behavior aligns with that change.
• Distinction of Concepts: The work successfully separates three often-conflated ideas:
  1. Whether a state is written.
  2. Whether a state is internally represented.
  3. Whether the model actually computes from that state.
     The results show that supervision is required to bridge the gap between representation and computation.

The authors conclude that while this structure was demonstrated in a controlled, synthetic environment, the goal for future process supervision is to build tasks and training methods where models expose similarly usable, causally active state variables in richer domains (e.g., program states, theorem proving, tool use).