💬 NLP

Reading Between the Dots: Decoding Hidden Computation across Filler Tokens

This paper demonstrates that frontier LLMs perform structured, legible multi-step reasoning over content-free filler tokens within their hidden states, revealing that hidden computation remains monitorable through unsupervised decoding of the residual stream even when surface tokens provide no visible chain-of-thought.

Original authors: Kaley Brauer, Claudio Mayrink Verdun, Samuel Marks

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kaley Brauer, Claudio Mayrink Verdun, Samuel Marks

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a magician perform a trick. Usually, to understand how they did it, you watch their hands and listen to what they say. In the world of Artificial Intelligence (AI), this is like looking at the "Chain of Thought" (CoT)—the step-by-step explanation the AI writes out before giving an answer.

For a long time, safety experts worried that if an AI stopped writing out its steps and just gave a final answer, we would have no way of knowing if it was thinking correctly or just guessing. This paper, titled "Reading Between the Dots," investigates a specific, weird scenario where the AI is given a bunch of meaningless "filler" tokens (like a string of dots ...... or counting numbers 1 2 3 4 5) between the question and the answer.

Here is what the researchers found, broken down into simple concepts:

1. The Magic Trick: The AI Uses the "Empty Space"

The researchers asked two very smart AI models (DeepSeek V3 and Kimi K2) difficult math and logic questions.

Without the dots: The AI often got the answer wrong.
With the dots: The AI got the answer right much more often.

It turns out the AI isn't just ignoring the dots. It's using that "empty space" as a scratchpad. Even though the dots look like nothing to us, the AI is secretly doing its math and reasoning inside the digital space where those dots sit. It's like a chef who stops talking to the customer and starts whispering the recipe to themselves while chopping vegetables; the customer hears nothing, but the cooking is happening.

2. The "X-Ray Vision": Seeing the Hidden Thoughts

The big question was: If the AI isn't writing the steps down, can we still see what it's thinking?

The authors say yes. They developed a special "X-ray" tool (called a Logit Lens) that looks at the AI's internal electrical signals (its "residual stream") instead of just reading the words it types.

The Analogy: Imagine the AI's brain is a busy highway. The words it types are just the cars driving on the surface. But underneath the road, there are tunnels and pipes carrying the actual water and electricity that make the cars move.
What they found: By looking at these "tunnels" under the dots, they could see the AI retrieving facts (like "Mozart died at 35") and doing math (like "35 + 93 = 128") before it ever typed the final answer. The dots weren't empty; they were filled with the AI's internal calculations.

3. The "Brain Swap" Experiment

To prove that the dots were actually causing the AI to think correctly (and not just a coincidence), they did a "brain transplant" experiment.

They took the internal "thoughts" (the hidden data) from a successful AI run and swapped them into a different AI run that was about to fail.
The Result: The failing AI suddenly started giving the correct answer, as if it had "caught" the correct thoughts from the other one. This proved that the information in the dots was real, useful, and essential for the answer.

4. The "Decoder Ring": Reading the Secret Code

The researchers built a simple, unsupervised pipeline (a set of instructions that doesn't need to be taught) to decode these hidden thoughts.

They took the hidden signals from the dots, cleaned up the noise, and fed the top candidates into a second, very smart AI (a "Judge").
The Success Rate: This decoder could correctly guess what the AI was thinking about 80% to 95% of the time.
What it found: It could recover specific numbers, names of cities, or elements, even though the AI never wrote them down. If the AI got the answer wrong, the decoder could even tell why (e.g., "It found the right facts but forgot to add them together").

The Big Takeaway

The paper argues that just because an AI doesn't say its reasoning out loud (or in a visible chain of thought), it doesn't mean the reasoning is hidden from us forever.

The Problem: If we only look at the words the AI writes, we might miss its real thinking process, especially when it uses "filler" tokens.
The Solution: We have tools to look "under the hood" at the AI's internal signals. Even when the AI tries to hide its work in "empty" spaces, we can still read the "residual stream" (the internal electrical activity) to see what it's actually doing.

In short: The AI is like a student taking a test who writes the answer on a piece of paper but does all the math in their head. This paper shows that even if we can't see the math on the paper, we can still use special tools to read the student's mind and see exactly how they solved the problem. This suggests that "hidden" computation isn't necessarily "unauditable."

Technical Summary: Reading Between the Dots

Problem Statement

Chain-of-thought (CoT) monitorability relies on the assumption that a model's surface-level output tokens faithfully reflect its underlying reasoning process. However, recent literature suggests this assumption is fragile; models may engage in "hidden computation" where reasoning occurs in ways not visible in the generated text. A specific failure mode of CoT monitoring is filler-token computation, where Large Language Models (LLMs) perform multi-step reasoning over content-free tokens (e.g., sequences of dots, counting numbers, or alphabet sequences) inserted between a question and the answer slot. In these scenarios, the surface tokens carry no semantic information about the reasoning, rendering traditional behavioral auditing ineffective. The central question addressed by this paper is whether such hidden computation is truly unobservable or if it remains decodable from the model's internal states.

Methodology

The authors investigate this phenomenon using two frontier-scale, open-weights models: DeepSeek V3 (671B parameters, ~37B activated) and Kimi K2 (1T parameters, ~32B activated). They evaluate performance and internal mechanics across four task families requiring intermediate values:

1-fact addition: Retrieving a fact and adding a random number.
2-fact addition: Retrieving two facts and summing them.
2-hop letter position: Retrieving a capital city/element and finding a specific letter.
System of equations: Solving linear relations defined in context.

The experimental setup involves appending content-free filler tokens (dots, counting sequences, alphabet sequences) between the prompt and the answer slot. The study proceeds in three phases:

Behavioral Analysis: Measuring accuracy improvements with and without filler tokens.
Mechanistic Analysis: Using attention maps, logit-lens readouts, and KV-cache transplants to determine how information flows and is encoded within the filler region.
Unsupervised Decoding: Developing a pipeline to recover intermediate values solely from hidden states without ground-truth labels or training.

Key Contributions and Results

1. Filler Tokens Enable Hidden Computation

The paper establishes that filler tokens are not merely inert padding but facilitate a "processing relay."

Behavioral Uplift: Both models show significant accuracy gains when filler tokens are present. For example, DeepSeek V3's accuracy on 1-fact addition rises from 54% to 72%, and on systems of equations from 31% to 61%. Scrambled filler sequences underperform, indicating the benefit comes from non-disruptive token sequences rather than specific content.
Attention Routing: Attention analysis reveals that in the presence of filler, the model re-routes its computation. The direct attention path from the answer position to the question is replaced by a relay: Question → Filler → Answer. The filler region acts as a buffer where the question is read, and intermediate values are processed before being passed to the answer position.

2. Mechanistic Evidence of Encoded Intermediates

The authors provide three lines of evidence that filler positions causally carry task-relevant information:

Logit-Lens Readouts: Projecting residual stream states through the unembedding matrix reveals that intermediate values (e.g., retrieved facts $A_1, A_2$ and their sum) are explicitly encoded in the residual stream at filler positions. In 2-fact addition, $A_1$ appears early in the filler sequence, $A_2$ later, and the sum crystallizes in the final layers before the answer.
KV-Cache Transplants (Causal Verification): By transplanting the Key-Value (KV) cache of the filler region from a "donor" example to a "target" example, the authors demonstrate causal dependence. Swapping the filler cache causes the target model to output the donor's answer, with rank improvements of tens to hundreds of places. This confirms that the content within the filler region is not just correlated with but causally necessary for the output.
Positional Specificity: In 2-fact addition, the positional structure of the encoding is causal. Transplanting only the specific positions where $A_1$ is decoded recovers the majority of the effect, while complementary positions have negligible impact.

3. Unsupervised Decoding Pipeline

The paper introduces a pipeline that recovers hidden reasoning steps from residual states without supervision:

Process:
1. Extract residual stream activations at all layers and filler positions.
2. Apply the logit lens to project states to token probabilities.
3. Subtract cross-example means to remove formatting artifacts and isolate example-specific signals.
4. Aggregate residual scores across positions and select the top-50 tokens.
5. Pass these tokens to an LLM judge (e.g., Claude Haiku/Sonnet) to interpret the reasoning.
Performance: The pipeline achieves 80–95% accuracy in recovering intermediate values across all four tasks and both models.
- For 1-fact addition, the target fact is recovered in the top-2 tokens in ~85% of cases without a judge; with a judge, this rises to ~94%.
- For complex tasks like 2-fact addition, the judge significantly improves recovery (from ~35% to ~82% on DeepSeek V3) by utilizing contextual signals from the full top-50 list.
- The pipeline successfully decodes both retrieved facts and values computed in-context (e.g., variables in systems of equations).

Significance and Claims

The paper argues that hidden computation is not synonymous with un-auditable computation. While filler-token reasoning defeats behavioral CoT monitoring (because the output text is empty of reasoning), the computation remains directly readable from the model's residual stream.

Monitorability as a Property of the Trace: The authors posit that monitorability is a property of the model's full computational trace, not just its surface tokens.
Partial Auditability: The study demonstrates that for tasks with discrete, nameable intermediates, "cheap" interpretability tools (logit lens + normalization + LLM judge) can close the gap left by opaque surface tokens.
Limitations: The authors modestly claim that this approach works for tasks with clean intermediates. They acknowledge limitations regarding computation that is optimized to be orthogonal to the vocabulary projection, involves non-tokenizable concepts (like strategies or intents), or relies on attention patterns invisible to logit-lens projections.

In conclusion, the paper suggests that while CoT monitoring is fragile, the existence of hidden computation in filler tokens does not render models entirely opaque. Instead, it highlights the potential for hybrid auditing strategies that combine behavioral observation with residual-stream interpretation.