💬 NLP

Mechanistic Interpretability of Chain-of-Thought Reasoning via Sequential Activation Patching

This paper introduces a sequential activation patching framework to identify distributed attention-head sub-circuits that causally support Chain-of-Thought reasoning in Large Language Models by tracing temporally spread effects across token positions and validating their functional importance through targeted ablation.

Original authors: Murat Dura, Serkan Öztürk, Selma Tekir

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Murat Dura, Serkan Öztürk, Selma Tekir

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Large language models are the engines behind many modern artificial intelligence tools, capable of generating human-like text, solving problems, and answering questions. When these models face complex tasks, such as a difficult math word problem, they often perform much better if they are first asked to write out their thinking step-by-step before giving a final answer. This technique, known as "chain-of-thought" prompting, encourages the model to break a problem down into smaller, manageable pieces, much like a human might scratch out calculations on a piece of paper. However, a lingering mystery has been whether these written steps are actually helping the model think, or if they are just a polite performance. It has been unclear if the model is truly using the intermediate text to guide its internal logic, or if it is simply generating plausible-sounding words while secretly relying on a different, hidden process to reach the answer. Understanding this distinction is crucial because if the model is not truly reasoning, its explanations could be misleading, and its reliability in critical situations would be questionable.

To solve this puzzle, researchers at the İzmir Institute of Technology decided to look inside the model's "brain" while it worked. They focused on a specific type of artificial intelligence called a transformer, which processes information through a vast network of interconnected pathways. They wanted to know exactly where and how the chain-of-thought process changes the model's internal activity. Instead of just watching the final output, they used a technique called activation patching. Imagine the model's internal state as a stream of data flowing through different rooms in a building. In a standard run, the model solves a problem without writing out its steps. In a "clean" run, it solves the same problem while writing out its steps. The researchers took the specific data flowing through certain pathways during the "clean" run and swapped them into the "standard" run. By doing this, they could see if injecting the "thinking" data into the "non-thinking" run would cause the model to suddenly start solving the problem correctly, as if it had been given the missing steps.

The team discovered that the model's ability to reason is not stored in a single, isolated spot, but is spread out across a distributed network of pathways that activate in a specific sequence. When they replaced the internal signals of a model that was not thinking with the signals from a model that was, the non-thinking model began to behave like the thinking one. It moved closer to the correct answer and started to follow the same logical path. This suggests that the chain-of-thought prompts do more than just add text to the screen; they actively reorganize the model's internal computation, steering it toward a reasoning mode. The researchers found that this effect was strongest when they swapped signals from the same specific problem, but it still worked, though less strongly, when they swapped signals from a completely different problem. This indicates that the model has learned general patterns for how to think through a problem, not just how to solve one specific equation.

To confirm that these pathways were truly essential, the researchers performed a second experiment where they simply turned off the identified pathways entirely. When they did this, the model's performance collapsed. It didn't just get the final number wrong; it lost the ability to hold its place in the story, often mixing up the details of the example problem with the new question, or failing to even state the final answer in the expected format. The model would sometimes repeat the same phrase over and over or generate numbers that made no sense. These failures showed that the pathways identified by the researchers are not just for writing the explanation; they are the very circuits that keep the model's reasoning on track, help it separate the example from the real task, and ensure it generates the correct numbers.

The study also ruled out the idea that these improvements were just random noise or a side effect of changing any part of the model. When the researchers swapped in random, meaningless data instead of the specific "thinking" signals, the model did not improve. This confirmed that the specific signals carried by these pathways are what matter. The researchers found that these critical pathways are concentrated in the middle and later sections of the model's layers, rather than at the very beginning or end. They act as a support system that helps the model maintain its train of thought, anchor the final answer to the right number, and keep the example problem from contaminating the new one.

Ultimately, the work suggests that chain-of-thought prompting works because it recruits a specific, distributed set of internal circuits that are designed to support step-by-step reasoning. These circuits are not a separate "reasoning module" that sits apart from the rest of the model; rather, they are overlapping components that the model dynamically uses when it is asked to think aloud. When the model is not prompted to think, these circuits are not fully engaged, and the model struggles with complex tasks. When it is prompted, these circuits activate, guiding the model through the necessary steps to reach a solution. The findings provide a clearer picture of how these powerful tools actually work, showing that their ability to explain their reasoning is not just a linguistic phenomenon, but a reflection of a genuine, internal shift in how they process information.

Technical Summary: Mechanistic Interpretability of Chain-of-Thought Reasoning via Sequential Activation Patching

Problem Statement

Large Language Models (LLMs) demonstrate significant performance improvements on complex reasoning tasks, particularly mathematical word problems, when guided by Chain-of-Thought (CoT) prompting. However, the internal mechanisms driving these improvements remain poorly understood. A central question in the field is whether CoT merely alters the surface form of generated text or if it causally reorganizes the model's internal computation to support reasoning. This relates to the broader issue of "CoT faithfulness": whether generated rationales accurately reflect the model's internal reasoning process or merely rationalize answers influenced by biased features.

Existing mechanistic interpretability tools, such as activation patching, are typically applied at single, static token positions. This approach is insufficient for CoT, which unfolds as a temporally distributed process across multiple generated tokens. Consequently, there is a gap between the sequential nature of CoT reasoning and the static structure of current mechanistic analyses. This paper addresses the need to identify where and how CoT-related causal effects emerge across the reasoning trajectory and which specific attention heads carry signals contributing to final-answer computation.

Methodology

The authors investigate these questions using the Qwen2.5-0.5B model on the SVAMP dataset of mathematical word problems. The study employs a controlled setup comparing CoT prompting (clean run) against No-CoT prompting (corrupted run).

Sequential Activation Patching Framework

To overcome the limitations of static patching, the authors introduce a Sequential Activation Patching framework:

Intervention Strategy: The outputs of attention heads from the clean CoT run are injected into the corrupted No-CoT run. Specifically, for each source CoT token $t$ , layer $l$ , and head $h$ , the head's output is replaced at the No-CoT position immediately following the anchor phrase "The answer is."
POS-Guided Aggregation: To mitigate noise inherent in token-level interventions, results are aggregated based on Part-of-Speech (POS) tags (e.g., nouns, verbs, numbers). This allows for the identification of stable functional patterns across the reasoning trajectory rather than isolated token effects.
Sequential Multi-Head Patching: Recognizing that CoT information relies on distributed representations, the authors implement a selection and merging strategy. They identify the top- $k$ most influential heads for each POS category and aggregate them into a global intervention set. A deduplication mechanism ensures that heads appearing in multiple categories are assigned to their most effective functional role. This set is then jointly patched into the No-CoT run.

Control Conditions

To validate the findings, the study includes:

Sequential Multi-Head Cross Patching: Injecting activations from a different problem's CoT trace into the target problem's No-CoT run to test for problem-specific vs. generalizable reasoning signals.
Random-Activation Sequential Multi-Head Patching: Injecting randomly generated activation tensors to rule out effects caused by arbitrary perturbations.

Evaluation Metrics

Two metrics quantify the causal shift:

Jensen–Shannon Divergence (JSD): Measures the distributional similarity between the patched run and the clean CoT run, capturing global alignment.
Margin Recovery Ratio (MRR): A logit-based metric measuring the recovery of the decision margin between the correct answer and the highest-scoring alternative, capturing local confidence shifts.

Zero-Ablation Experiments

To establish necessity, the authors perform targeted zero-ablation of the identified head sets across three scenarios: Direct-Equation Prompting, No-CoT Prompting, and CoT Prompting.

Key Contributions

Sequential Activation Patching Framework: The paper proposes a many-to-one patching strategy that traces CoT-conditioned activations across token positions and aggregates them via POS-guided analysis, addressing the temporal limitations of standard patching.
Evidence of Causal Contribution: The authors demonstrate that injecting CoT-conditioned activations into No-CoT executions causally shifts the model's behavior toward CoT-like distributions and decision margins. This suggests that CoT-related activations are not merely epiphenomenal but functionally contribute to final-answer computation.
Identification of Distributed Sub-Circuits: Through Sequential Multi-Head Patching, the study identifies that CoT reasoning relies on coordinated activity across multiple attention heads rather than isolated components. Cross-question patching results indicate these heads encode partially reusable reasoning-support patterns, not just problem-specific content.
Functional Importance via Ablation: Targeted zero-ablation reveals that the identified heads are critical for successful reasoning. Their removal disrupts not only final-answer accuracy but also auxiliary mechanisms, including reasoning-trajectory maintenance, answer anchoring, exemplar–target separation, and numerical generation.

Experimental Results

Patching Efficacy: Sequential Multi-Head Patching produced the strongest improvement, significantly reducing JSD and increasing MRR, moving No-CoT executions toward CoT behavior. Cross-patching yielded weaker but meaningful recovery, suggesting generalizable signals. Random patching failed to reproduce these effects.
Head Selection Stability: The targeted selection methods identified a concentrated subset of heads (primarily in middle-to-late layers, with Layer 14 being most frequent) that showed high overlap across different metrics and selection families. In contrast, random selection methods identified a much more diffuse set of heads.
Ablation Impact: In CoT prompting, ablating the heads selected via Sequential Multi-Head Patching using JSD reduced accuracy from a baseline of 65.63% to 0.00%, while MRR-based selection reduced it to 6.25%. In No-CoT and Equation-only settings, ablation also caused significant accuracy drops compared to random baselines, indicating these heads support fundamental arithmetic and abstraction circuits even outside explicit CoT contexts.
Failure Modes: Qualitative analysis of ablation failures showed that removing these heads led to structural breakdowns such as the loss of the "The answer is" anchor, contamination from few-shot exemplars, and corruption of the arithmetic trajectory.

Significance and Claims

The paper claims to provide evidence for distributed reasoning-support sub-circuits associated with CoT-conditioned computation. The authors argue that CoT does not merely add explanatory text but helps steer the model's internal computation toward a reasoning-compatible trajectory.

However, the authors maintain a modest stance regarding the nature of these circuits:

They interpret the identified structure as a distributed reasoning-support sub-circuit dynamically recruited by CoT, rather than a pure, isolated "reasoning-only" circuit.
The ablation results suggest these heads support overlapping functions, including target-question grounding, exemplar separation, answer-slot binding, and numerical generation.
The paper does not claim that every generated rationale token is a fully faithful explanation of internal reasoning; rather, CoT acts as a useful internal control signal whose textual faithfulness may remain partial.

The study concludes that the discovered heads are functionally important for CoT performance and are less effectively recruited in No-CoT settings, highlighting a mechanistic basis for the performance gains observed with Chain-of-Thought prompting.