🤖 machine learning

When Chain-of-Thought Fails, the Solution Hides in the Hidden States

By using activation patching to transfer hidden states from reasoning traces to direct-answer runs, the authors demonstrate that individual Chain-of-Thought tokens encode sufficient task-relevant information to recover correct answers even when the original reasoning trace is flawed.

Original authors: Houman Mehrafarin, Amit Parekh, Ioannis Konstas

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Houman Mehrafarin, Amit Parekh, Ioannis Konstas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a student take a difficult math exam. They are "thinking out loud," writing down every single step on their scratchpad. Suddenly, they reach the final answer and write: "The answer is 12."

You look at their work, do the math yourself, and realize they made a mistake. The real answer is 18.

Most people would assume that because the student wrote "12," their brain must have been totally lost. But what if I told you that if you could peek into their brain at the exact moment they were writing a specific word—like the word "subtract"—you would see that they actually knew the right answer, but they just stumbled at the very end?

That is exactly what this research paper is about.

The Core Idea: The "Hidden Genius" in the Mistake

The researchers wanted to know: When an AI (like ChatGPT) gives a wrong answer after explaining its reasoning (Chain-of-Thought), is the AI actually "stupid" during the whole process, or is the correct information just "hiding" inside its internal thoughts?

To find out, they used a technique called "Activation Patching."

The Analogy: The Radio Signal
Think of the AI's reasoning like a radio broadcast.

The "CoT" (Chain-of-Thought) is the actual song playing out loud (the text you read).
The "Hidden States" are the invisible radio waves traveling through the air.

Sometimes, the "song" (the text) is a distorted, messy version of the music. But the researchers discovered that even when the song sounds terrible and wrong, the "radio waves" (the internal math) are often still broadcasting a perfectly clear, correct signal.

By "patching"—which is like grabbing a clean signal from one moment and plugging it into another—they were able to "fix" the AI's broken reasoning and force it to output the correct answer, even when the AI's own written explanation was wrong.

The Three Big Discoveries

1. The "Verb" vs. "Number" Secret

The researchers found that not all words are created equal.

The "Logic" Words (Verbs and Entities): Words like "subtract," "total," or "Janet" are like the GPS coordinates of the problem. If you "patch" these into the AI, it suddenly finds its way to the right answer. These words carry the strategy.
The "Math" Words (Numbers and Operators): Words like "16" or "+" are like the passengers in the car. They are just numbers floating around. If you patch just a number, the AI often just guesses a random wrong number. It doesn't have the "map" to get home.

2. The "Early Bird" Advantage

The researchers found that the most useful "correct" information usually appears early in the reasoning process.
The Analogy: It’s like a chef preparing a meal. If they pick the wrong ingredients at the very beginning, the meal is ruined. But if they have the right ingredients (the correct logic) in their hands early on, they can still cook a perfect meal even if they get a bit messy during the actual cooking.

3. You Don't Need a Long Story

Usually, we think AI needs to write a long, beautiful essay to solve a hard problem. But this paper shows that the AI's "internal brain" is much more efficient than its "mouth."
The researchers found they could "patch" a single internal thought into the AI, and it would spit out the correct answer in just a few words. It didn't need the long, rambling explanation to get it right; it just needed that one "spark" of correct internal logic.

Why does this matter?

This research tells us that AI "reasoning" is deeper than the words it types.

When an AI fails, it isn't always because it doesn't know the answer; it's often because it fails to translate its internal knowledge into the written words we see. This opens the door to building smarter AIs that don't need to write long, rambling paragraphs to be accurate—they just need to keep their "internal radio signal" clear.

Technical Summary: When Chain-of-Thought Fails, the Solution Hides in the Hidden States

1. Problem Statement

Chain-of-Thought (CoT) prompting is a widely used technique to improve the performance of Large Language Models (LLMs) on multi-step reasoning tasks by eliciting intermediate reasoning steps. However, a fundamental question remains: Is the generated CoT text computationally useful for the model's final answer, or is it merely "explanatory" (post-hoc justification)?

Furthermore, it is unclear whether the internal representations (hidden states) of the model contain the correct reasoning logic even when the model's generated text is incorrect. This paper seeks to determine if task-relevant information is recoverable from the hidden states of CoT tokens, regardless of whether the surface-level text is accurate.

2. Methodology

The authors employ a mechanistic causal analysis using a framework called activation patching on the GSM8K arithmetic reasoning benchmark. The study uses two state-of-the-art models: LLaMA 3.1 8B and Qwen 2.5 7B.

The Patching Framework:

Source Run (CoT): The model is prompted to "think step-by-step." The researchers cache the residual hidden states of every token at every layer during this generation.
Target Run (Direct Answer): The model is prompted to provide a direct answer without showing steps (a prompt that typically fails on GSM8K).
Intervention: At a specific layer $l$ , the hidden state of the final token in the target run is replaced (patched) with the cached hidden state of a specific token from the source CoT run.
Measurement: The researchers measure the change in the model's ability to produce the correct final answer.

Analytical Dimensions:

Token Roles: Tokens are categorized into semantic roles (e.g., verbs, entities, operands, operators, quantities).
Layer Depth: Analysis of whether information is concentrated in early, mid, or late layers.
Token Position: Analysis of whether information is located at the beginning (Q1) or end (Q4) of the reasoning trace.
Generation Behavior: Classification of post-patch outputs (e.g., Full CoT, Equation-Only, Final-Only, Noise).

3. Key Contributions

Discovery of Recoverable Information: The paper proves that CoT hidden states encode sufficient information to recover the correct answer even when the original CoT text is incorrect.
Mechanistic Localization: It identifies exactly where (layers) and when (token positions) this information resides.
Semantic Role Analysis: It distinguishes between "language tokens" (which carry reasoning logic) and "mathematical tokens" (which carry answer-proximal content).
Efficiency Insights: It demonstrates that a full, long-form CoT trace is not strictly necessary for correct reasoning if the right internal states are accessed.

4. Key Results

High Patch Accuracy: Patch accuracy (the ability to recover the correct answer via at least one intervention) exceeds 99% for both models, significantly outperforming both the original CoT and direct-answer baselines.
Correct vs. Incorrect Traces: While incorrect CoT runs still contain recoverable information, they have a significantly lower patch success density than correct runs. In incorrect runs, the "wrong" information often dominates the hidden states, making the correct signal harder to find.
Layer and Position Trends:
- Layers: Task-relevant information is concentrated in mid-to-late layers.
- Position: Information is most effective in earlier tokens (Q1), with success rates declining as the reasoning trace progresses.
The Role of Language vs. Math:
- Verbs and Entities are highly effective at eliciting "Full CoT" or "Equation-Only" outputs that lead to correct answers.
- Mathematical tokens (operands, operators) tend to trigger "Final Only" outputs that are frequently incorrect, suggesting they encode the result but not the logic.
Token Budget Efficiency: Patching can achieve >94% accuracy with as few as 50 tokens, whereas the original CoT traces are much longer. This suggests the "reasoning" is more compressed in the hidden states than in the text.

5. Significance

This research shifts the understanding of CoT from a purely linguistic phenomenon to a mechanistic one. It suggests that reasoning is a latent process that is often more robust than the text the model produces.

Implications include:

Model Debugging: Understanding that reasoning failures might be due to "incorrect signals dominating correct ones" rather than a total absence of logic.
Efficiency: The finding that short, patched outputs can match long CoT accuracy suggests new avenues for distilling reasoning into more compact, efficient formats.
Interpretability: It provides a roadmap for identifying the specific "circuits" or tokens that are responsible for successful multi-step computation.