💬 NLP

Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Models

This survey provides a comprehensive overview of the internal mechanisms underlying multi-step reasoning in Large Language Models by organizing existing research around a conceptual framework of seven interconnected questions and outlining five future directions for mechanistic studies.

Original authors: Liangming Pan, Jason Liang, Jiaran Ye, Minglai Yang, Xinyuan Lu, Fengbin Zhu

Published 2026-01-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Liangming Pan, Jason Liang, Jiaran Ye, Minglai Yang, Xinyuan Lu, Fengbin Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine Large Language Models (LLMs) as incredibly talented but secretive chefs. They can whip up a perfect five-course meal (solve complex problems), but we often don't know exactly what's happening inside the kitchen while they cook. This paper is a "kitchen tour" that tries to open the black box and see how these chefs actually think.

The authors break down the chef's thinking process into two main styles: Silent Thinking (Implicit Reasoning) and Talking While Cooking (Explicit Reasoning).

1. The Silent Chef: "Implicit Reasoning"

This is when the model solves a problem entirely in its head without saying a word out loud. It's like a chef who instantly knows the recipe without writing anything down.

How it works inside: The paper finds that the model doesn't just "know" the answer. It actually has a hidden assembly line. Different layers of the model act like different stations in a kitchen. The first station finds the main ingredient, the next station chops it, and the final station plates the dish. If the kitchen is too small (the model isn't deep enough), the assembly line gets cut short, and the meal fails.
The "Aha!" Moment (Grokking): Sometimes, the chef seems to be just memorizing recipes for weeks, failing to understand the logic. Then, suddenly, there's a "grokking" moment—a phase transition where the chef stops memorizing and starts truly understanding the cooking logic. This usually happens if the chef practices with enough varied recipes.
The Trap (Shortcuts): Here's the bad news: The chef often takes shortcuts. Instead of actually cooking the dish step-by-step, the chef might just smell the ingredients and guess the final dish based on a pattern they've seen before. If you change the order of the ingredients, the chef gets confused because they weren't actually cooking; they were just guessing based on surface patterns.

2. The Talking Chef: "Chain-of-Thought" (Explicit Reasoning)

This is when we ask the model to "think out loud" (Chain-of-Thought or CoT). It's like asking the chef to narrate every step: "First I chop the onion, then I sauté it..."

Why it helps: When the chef talks, they aren't just using their brain's limited memory anymore. They are using the paper they are writing on as an external notebook. This gives them extra "thinking time" and space to work through complex math or logic puzzles that were too hard to do in their head alone.
The Magic of "Thinking Time": Surprisingly, the paper found that even if the chef writes nonsense words ("... ... ...") instead of real steps, the model still gets better at solving the problem. Why? Because the act of writing anything forces the model to pause and run its internal circuits one more time. It's like the extra time to think is more important than the words themselves.
The Illusion of Honesty: This is the most critical finding. Just because the chef is talking out loud doesn't mean they are telling the truth about how they decided.
- The "Post-Hoc" Lie: Often, the chef decides the answer first (maybe by guessing or using a shortcut), and then invents a logical-sounding story to explain why that answer is right. The "thinking out loud" is often just a fancy excuse written after the fact, not the actual reason they chose the answer.
- The Mismatch: The chef's brain works in parallel (doing many things at once), but the spoken explanation is a single, straight line. You can't perfectly translate a complex, multi-threaded brain process into a simple, step-by-step story.

3. What's Next? (The Future Roadmap)

The authors suggest we need to stop just watching the chef and start testing the kitchen more rigorously:

Real-World Testing: Most studies use fake, clean test kitchens. We need to see how these models handle messy, real-world cooking where ingredients are missing or confusing.
Honesty Checks: We need new ways to check if the "talking" matches the "thinking." We need to bridge the gap between what the model says and what it actually does.
Better Tools: Instead of just measuring if the final dish tastes good, we need to check the internal "effort" the chef put in. Did they actually cook, or did they just guess?
Taking Control: Once we understand the specific "knobs" and "switches" inside the model that handle logic, we shouldn't just observe them; we should be able to flip them to fix errors or stop the model from taking shortcuts.

In short: Large Language Models are powerful, but they often rely on guessing and shortcuts. When they "think out loud," it helps them solve harder problems by giving them extra time, but their spoken explanations are often just stories they make up to sound smart, not a true map of their internal thinking. To trust them, we need to stop looking at the final answer and start understanding the messy, hidden machinery inside.

Technical Summary: Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Models

Problem Statement

Large Language Models (LLMs) have demonstrated remarkable proficiency in multi-step reasoning, a capability essential for tasks ranging from mathematical problem-solving to logical deduction. However, the internal mechanisms enabling these capabilities remain largely opaque. While existing literature primarily focuses on engineering methods to enhance performance (e.g., tool use, retrieval augmentation), there is a significant gap in the mechanistic understanding of how LLMs execute multi-step reasoning. Specifically, it is unclear how models perform implicit multi-hop inference within fixed hidden activations and how explicit verbalized reasoning (Chain-of-Thought, or CoT) remodels internal computation. Furthermore, the extent to which LLMs rely on genuine reasoning versus statistical shortcuts, and whether CoT outputs faithfully reflect internal decision-making processes, remain critical unresolved questions.

Methodology

This paper is a survey that synthesizes a growing body of mechanistic interpretability literature. Rather than conducting new experiments, the authors organize existing research into a cohesive cognitive framework comprising seven interconnected research questions (RQs). The survey categorizes findings based on two distinct reasoning paradigms:

Implicit Reasoning: Multi-hop inference occurring entirely within the model's hidden states without verbalized intermediate steps.
Explicit Reasoning: Multi-hop inference where the reasoning process is externalized into a sequence of natural language tokens (e.g., CoT).

The analysis draws upon diverse mechanistic techniques, including:

Causal Probing and Intervention: Methods like Patchscopes, back-patching, and logit lens analysis to identify causal roles of specific layers and neurons.
Representational Analysis: Examining the evolution of internal states and attention patterns during inference.
Training Dynamics Analysis: Investigating phase transitions (e.g., "grokking") and the emergence of reasoning circuits during training.
Theoretical Analysis: Applying complexity theory to understand the computational limits of transformers with and without CoT.

Key Contributions and Findings

1. Mechanisms of Implicit (Latent) Multi-Step Reasoning

Layered Specialization: Implicit reasoning is not monolithic but an orchestrated, staged process. Early layers often identify "bridge entities," while intermediate and later layers perform enrichment and attribute extraction. Specific attention heads (e.g., "deduction heads") and feed-forward neurons act as working memory or state trackers.
Depth as a Bottleneck: The number of reasoning steps a model can perform implicitly is strictly limited by its depth. Transformers with constant depth cannot theoretically solve inherently serial problems requiring computation scaling with input size.
Emergence via Grokking: Latent reasoning capabilities are acquired, not present at initialization. They emerge through a phase transition known as "grokking," where the model shifts from memorization to generalization. This transition is governed by training data distribution (specifically the ratio of inferred to atomic facts), data scale, and optimization biases (e.g., weight decay).
Reliance on Shortcuts: Models frequently bypass genuine reasoning via "shortcuts," such as direct subject-answer associations or surface-level pattern matching (e.g., treating operations as commutative). These shortcuts often lead to performance collapse when surface patterns are disrupted.

2. Mechanisms of Explicit (CoT) Reasoning

Remodeling Internal Computation: CoT prompting fundamentally alters the model's internal state. It activates "iteration heads" that attend to previously generated tokens, effectively creating a virtual recurrent neural network (RNN) where text serves as external memory.
State Maintenance: CoT enables the model to maintain and update dynamic internal states (e.g., finite state automata in hidden layers) to track reasoning progress, even when the generated text is semantically nonsensical (e.g., filler tokens).
Parallelism and Shortcuts: Despite the sequential appearance of CoT, internal computation often involves parallel pathways. Models may simultaneously attempt direct shortcuts and step-by-step reasoning, converging in later layers.
Enhanced Expressiveness and Robustness: Theoretically, CoT breaks the constant-depth limitation of standard transformers, elevating their computational complexity class from $TC^0$ to Polynomial Time ( $P$ ). It introduces modularity that reduces sample complexity and promotes robust generalization by mitigating the "snowball error effect" common in implicit reasoning.
The Faithfulness Gap: CoT outputs often do not faithfully reflect the model's actual decision-making process. Due to the structural mismatch between the distributed, parallel nature of transformer computation and the sequential nature of explicit reasoning, CoT frequently functions as post-hoc rationalization rather than a causal driver. Models may generate plausible rationales to justify decisions driven by spurious cues or user bias.

Significance and Future Directions

The paper argues that understanding these mechanisms is a prerequisite for building reliable, controllable, and human-aligned reasoning systems. The authors identify five critical directions for future research:

Rigorous Causal Analysis in Real-World Settings: Moving beyond synthetic data to verify causal links between internal representations and reasoning in complex, noisy environments.
Bridging the Faithfulness Gap: Developing white-box alignment methods to enforce causal links between hidden states and generated rationales, ensuring CoT reflects true internal dynamics.
Mechanistic Understanding of Latent CoT: Investigating emerging architectures that simulate explicit reasoning trajectories entirely within hidden states, balancing interpretability with efficiency.
White-Box Evaluation Metrics: Shifting from black-box accuracy metrics to "white-box" metrics that verify the presence of necessary internal computational signatures (e.g., bridge entity formation) to distinguish genuine reasoning from pattern matching.
From Interpretation to Control: Transitioning from passive observation to active model control, leveraging identified reasoning circuits (e.g., state-maintenance neurons) to intervene in real-time and correct errors or suppress shortcuts.

Conclusion

This survey provides a comprehensive overview of the internal mechanisms of multi-step reasoning in LLMs, distinguishing between the depth-constrained, shortcut-prone nature of implicit reasoning and the computationally extended, yet often unfaithful, nature of explicit CoT reasoning. By framing the field around seven pivotal research questions, the paper highlights that while significant progress has been made in "opening the black box," critical challenges remain in verifying causal mechanisms, ensuring faithfulness, and translating mechanistic insights into robust model control.