💻 computer science

When Does a Readout Reflect Computation? Dose-Response Interventions in Continuous Thought Models

This paper demonstrates that in continuous thought models, readout-based interpretability often fails to reflect underlying computation because the latent states required to change the model's answer are significantly larger than those needed to alter the projected readout, necessitating dose-response curves rather than single-magnitude interventions for accurate causal analysis.

Original authors: ChaeWoo Son

Published 2026-08-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: ChaeWoo Son

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a super-smart robot thinks. Usually, when we talk to these robots, they speak in words, so we can read their "thoughts" as they happen. But there's a new kind of robot that thinks in a secret, invisible language—a continuous, swirling cloud of numbers called "latent space." It does all its math and reasoning inside this cloud without ever saying a word until it's ready to give the final answer. This is like a chef cooking a complex meal in a sealed kitchen; you can't see the chopping or the stirring, only the final dish.

To peek inside this secret kitchen, scientists use a special tool called a "readout." Think of it as a magical window that projects the robot's invisible number-cloud onto a wall of vocabulary words. If the robot is thinking about "Paris," the window might flash the word "Paris" brightly. For a long time, researchers assumed that if they saw the window flash a word, the robot was definitely thinking that word. They believed the window was a perfect mirror of the robot's brain. But what if the window is a bit of a trickster? What if it flashes a word just because you tapped the glass lightly, even though the robot hasn't actually changed its mind? This is the big question this paper asks: Does the window really show us what the robot is computing, or is it just reacting to the tap?

The researchers, led by ChaeWoo Son, decided to test this by playing a game of "tug-of-war" with the robot's brain. They used a specific type of robot called "Coconut" (Chain of Continuous Thought), which reasons in that secret number-cloud. To make the test fair, they first trained the robot so that its "window" (a tool called a J-lens) would reliably show a specific word, like a bridge between two facts. Then, they started poking the robot's brain with tiny, controlled nudges. They didn't just poke randomly; they measured exactly how hard they pushed (the "dose") and watched two things: did the window change its word, and did the robot's final answer change?

The results were a huge surprise. They found that the window and the robot's actual brain have very different "sensitivity thresholds." Imagine the window is like a very sensitive motion sensor that beeps if you walk past it, while the robot's brain is like a heavy safe that only opens if you push the button with a lot of force. The researchers discovered that when they nudged the robot just enough to make the window flip to a new word (a tiny nudge of about 0.13 to 0.16 units of force), the robot's brain didn't budge at all. The robot kept its original answer. In fact, on one type of task, the robot's brain needed a push more than twice as hard (2.03 times stronger) to actually change its mind.

Even stranger, they found a "secret tunnel" in the robot's brain. They found a direction to push that the window couldn't see at all. Even though the window stayed frozen on the old word, the robot's brain completely swapped its answer to a new one in 96–97% of the cases! It was like the robot was whispering a new secret to itself while the window kept shouting the old one. When they tried pushing in a random direction with the same strength, nothing happened, proving that this wasn't just about how hard they pushed, but where they pushed.

The paper also admits a funny mistake they made. At first, they tried to test if the robot was using its "bridge" thoughts by nudging just enough to make the window change. They stopped pushing the moment the window flipped, thinking they had done enough. But because the window is so sensitive, they stopped way too early—before the robot's brain had even started to change. They got a result that said "nothing happened," but it was a false alarm because they didn't push hard enough.

The main takeaway is a warning for anyone trying to read a robot's mind: You can't just look at the window and assume you know what the robot is thinking. The window might change its mind long before the robot does, or it might not change at all even when the robot is doing something huge. To really understand the robot, you have to measure how hard you are pushing and see how the robot reacts at every step, not just at one single moment. The window is a useful tool, but it's not a perfect mirror of the computation happening inside.

Technical Summary: When Does a Readout Reflect Computation? Dose–Response Interventions in Continuous Thought Models

1. Problem Statement

Chain of Continuous Thought (Coconut) models perform reasoning in a continuous latent space by feeding the last hidden state back as the next input embedding, rather than decoding intermediate steps into natural language tokens. While this offers expressivity, it creates a "monitorability problem": the intermediate computation cannot be observed directly.

Current interpretability practices rely on readouts—projections of hidden states into vocabulary space (e.g., via a Logit Lens or Tuned Lens). The standard inference is: if the readout changes, the underlying computation has changed. However, this inference has historically been tested at a single intervention magnitude. The paper posits that if the readout and the computation possess different sensitivities to the same intervention direction, a single-magnitude test cannot reveal the discrepancy, potentially leading to false conclusions about the model's internal reasoning.

2. Methodology

To measure the sensitivity of the readout versus the computation as a function of intervention magnitude, the author constructed a controlled testbed and a dose–response protocol.

2.1 Controlled Testbed Construction

J-Lens Supervision: The author trained a 2B parameter Coconut model on two-hop factual recall tasks. They added a J-lens term to the loss function to force the continuous thought (latent state) to align with a specific "bridge entity" token, while leaving the answer loss unmodified.
Validation: This supervision achieved >0.95 probability mass on the bridge token in the readout while maintaining 100% answer accuracy. This ensured the readout was "controlled" (pointing to a known token) without degrading the model's computational capability.
Self-Labeling: The author demonstrated that the model could generate its own bridge labels via the J-lens readout, removing the need for additional human annotation for the supervision signal.

2.2 Dose–Response Intervention Protocol

The core innovation is a protocol that decouples intervention magnitude from direction:

Fixed Relative Magnitude ( $\alpha$ ): Interventions are applied with a fixed relative magnitude $\alpha = \|\delta\|/\|h\|$ , recomputed at every latent step to account for state divergence.
Orthogonal Decomposition: The causal delta ( $d = h_{donor} - h_{recipient}$ $d = h_{d o n or} - h_{r ec i p i e n t}$ ) is decomposed into two orthogonal components:
- $d_{\parallel}$ : The component parallel to the lens gradient (the direction the lens detects).
- $d_{\perp}$ : The component orthogonal to the lens gradient (the direction the lens is "blind" to).
Matched Random Control: A random direction is generated that is orthogonal to the lens gradient and matched in norm to $d_{\perp}$ . This controls for magnitude and lens-invisibility, isolating the effect of direction.
Dose Sweep: The author swept $\alpha$ across a grid (0.10 to 0.60) to generate dose–response curves for both the readout (token change) and the computation (answer change).

3. Key Results

3.1 Dissociation of Thresholds (Lens-Visible Direction)

Along the direction the lens detects ( $d_{\parallel}$ ), the readout and computation have significantly different activation thresholds:

Readout Sensitivity: The readout flips to the donor token at relative magnitudes of 0.125–0.158.
Computation Sensitivity: The answer does not change at these magnitudes.
- On the lmk (landmark $\to$ country $\to$ capital) relation, the answer threshold is 2.03 $\times$ higher (0.253) than the readout threshold.
- On the typeA (capital $\to$ country $\to$ continent) relation, the answer never reaches a majority flip before the model breaks (outputs an unrelated token) at high magnitudes.
Implication: A minimum-norm intervention that stops as soon as the readout flips (the standard procedure) stops at roughly half the causal threshold (or below it entirely), failing to detect the causal link.

3.2 The "Invisible" Channel

Along the direction the lens does not detect ( $d_{\perp}$ ):

Computation: Interventions successfully transplant the answer in 96–97% of trials (thresholds $\approx$ 0.45–0.48).
Readout: The readout never changes (0/900 trials) and remains pinned to the recipient token.
Control: The matched random control produced no changes in either readout or answer, confirming the effect is due to the specific direction of $d_{\perp}$ , not just magnitude.

3.3 The "False Null" Trap

The author reports a pre-registered initial experiment that fell into this trap:

They intervened until the readout flipped (stopping magnitude $\approx$ 0.119).
Result: 0/90 answer flips. They initially concluded the bridge was not used in computation.
Correction: The dose–response curve revealed the answer threshold was 0.253. The initial test failed because it stopped too early, measuring below the causal threshold.

3.4 Robustness

The threshold gap persists (1.74 $\times$ ) even without J-lens supervision, indicating the dissociation is not an artifact of the training signal but a structural property of the model.
The gap is not explained by the model "breaking" at high magnitudes; the dissociation occurs in the stable operating window.

4. Contributions

Dose–Response Protocol: A method to separate readout sensitivity from computational sensitivity using fixed-magnitude, orthogonal decomposition, and matched random controls.
Quantified Dissociation: Evidence that the readout and computation have different thresholds along the same direction, and that computation can be transplanted along directions the readout does not register.
Diagnosis of Minimum-Norm Interventions: A demonstration that standard readout-targeted interventions systematically produce false negatives (false nulls) because they stop before the computation changes.
Controlled Testbed: A method to align readouts to specific tokens without accuracy loss, enabling precise causal measurement.

5. Significance and Claims

The paper argues that readout sensitivity is a property of the readout, not the computation. The fact that a readout changes does not imply the computation has changed, nor does a lack of readout change imply a lack of computational change.

Methodological Warning: Interpretability claims based on single-magnitude interventions are insufficient. A null result (no answer change) cannot distinguish between "the direction is not causal" and "we did not push hard enough."
Prescription: Readout-based causal claims must be reported as dose–response curves that include:
1. The magnitude at which the readout responds.
2. The magnitude at which the output responds.
3. A random control matched on magnitude and invariants.
Scope: The author is modest about generalization, noting measurements were made on a single model, a single layer, and synthetic relations. They do not claim to have mapped the entire latent space, but rather to have exposed a structural limitation in how readouts are currently interpreted.

The paper concludes that work asking "what can be read from a readout" must also ask "at what magnitude what is read corresponds to computation."