🤖 AI

Revealing Algorithmic Deductive Circuits for Logical Reasoning

This paper investigates how Large Language Models perform logical reasoning by using causal mediation analysis to identify specific attention heads that retrieve factual rules and higher-layer heads that integrate information to execute global algorithmic strategies like graph traversal.

Original authors: Phuong Minh Nguyen, Tien Huu Dang, Naoya Inoue

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Phuong Minh Nguyen, Tien Huu Dang, Naoya Inoue

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a brilliant, super-fast detective who has been trained to solve logic puzzles. You give the detective a few examples of how to solve a puzzle, and then you ask them to solve a new one. They usually do a great job, writing out their thought process step-by-step (a method called "Chain-of-Thought").

But here's the mystery: How does the detective actually "think"? Do they truly understand the logic, or are they just guessing based on patterns they've seen before?

This paper, "Revealing Algorithmic Deductive Circuits for Logical Reasoning," acts like a mechanic opening up the detective's brain to see exactly which gears are turning when they solve a logic problem.

The Detective's Toolkit: "Uncertain" Moments

The researchers noticed that when the detective is solving a puzzle, most of the words they write are easy. They know them with 100% confidence. For example, writing "Therefore..." or "The answer is..." is easy.

However, there are specific moments where the detective hesitates. These are the critical decision points:

Choosing the Fact: "Okay, which piece of evidence should I look at first?"
Stopping the Search: "Have I looked at enough evidence, or should I keep looking?"
Picking the Rule: "Which rule applies to this specific situation?"

The paper found that these are the moments where the model is least confident. It's like a human detective pausing to think, "Hmm, does this clue fit?" The researchers realized that if they can understand how the model makes these specific, difficult choices, they understand the whole reasoning process.

The "Brain Scan": Finding the Specialized Gears

To figure out what's happening inside the model, the researchers used a technique called Causal Mediation Analysis. Think of this as a "brain scan" for the AI.

They created a scenario where they took a "clean" puzzle and a "corrupted" puzzle (where they secretly changed a fact or a rule). Then, they played a game of "switcheroo" with the model's internal parts (called attention heads).

They would take the "thinking" part of the model from the clean puzzle and paste it into the corrupted puzzle.
If the model suddenly started solving the corrupted puzzle correctly, they knew that specific part was responsible for the logic.

What They Found: A Specialized Assembly Line

The study revealed that the model doesn't use its whole brain for every task. Instead, it uses a very specific, small team of specialized "gears" (about 3% of its total internal parts) to handle logic.

They found a clear hierarchy, like an assembly line in a factory:

The Early Layers (The Librarians): The lower parts of the model act like librarians. Their only job is to find the right facts and rules from the text and pull them off the shelf. They don't make decisions; they just gather information.
The Middle Layers (The Matchmakers): These gears take the facts and the rules and check if they fit together. "Does Fact A match Rule B?"
The Higher Layers (The Managers): The top layers act like the project managers. They don't look at individual facts; they look at the big picture. They decide the overall strategy, like "Let's use a Breadth-First Search" (checking all options at once) or "Let's go deep down one path first." They coordinate all the lower gears to solve the whole puzzle.

The "Knockout" Test

To prove these gears were actually necessary, the researchers did a "knockout" experiment. They temporarily disabled the specific 3% of gears responsible for logic and asked the model to solve puzzles again.

Result: The model's ability to solve logic puzzles collapsed. It went from being a brilliant detective to someone who couldn't solve a riddle, often just guessing randomly.
The Twist: When they asked the model general knowledge questions (like "Who was the first president?"), it barely noticed the missing gears. It could still answer those fine.

This proves that these specific gears are dedicated solely to logical reasoning. They aren't just general "smart" parts; they are specialized tools for deduction.

The Big Picture

The paper concludes that when an LLM learns to reason from just a few examples, it isn't just memorizing the answers. It is actually "learning" to activate a specific, sparse network of internal circuits that mimic an algorithm (a step-by-step computer program).

Low layers gather the data.
Middle layers check the rules.
High layers run the strategy.

It's as if the model has a hidden, tiny team of logic experts inside it that only wake up when you ask it to solve a puzzle, working together like a well-oiled machine to find the truth.

Technical Summary: Revealing Algorithmic Deductive Circuits for Logical Reasoning

Problem Statement

Recent studies indicate that Large Language Models (LLMs) can achieve strong reasoning performance by incorporating functional symbolic representations that abstractly describe graph traversal algorithms within few-shot Chain-of-Thought (CoT) settings. However, it remains unclear how LLMs genuinely understand the abstract meaning of individual reasoning steps and the overall algorithm from a limited number of demonstrations. While previous works have focused on developing sophisticated reasoning frameworks or improving performance through prompt engineering and fine-tuning, this study addresses a fundamental research question: What mechanisms do LLMs internally employ to solve logical reasoning tasks?

Specifically, the paper investigates deductive reasoning, which requires multi-hop reasoning to determine if a conclusion logically follows from a set of premises. The authors hypothesize that LLMs "learn" the meaning of inference steps by activating specific subsets of attention heads that act as mediators for executing inference steps and adhering to graph-traversal algorithms (e.g., Breadth-First Search or Depth-First Search).

Methodology

The authors employ Causal Mediation Analysis (CMA) within the field of Mechanistic Interpretability to localize and characterize the internal components (circuits) responsible for logical reasoning.

1. Symbolic-Aided CoT and Dataset Construction

The study utilizes a Symbolic-Aided CoT prompting format where the reasoning process is modeled as an inference graph constructed over predefined facts and rules.

Synthesized Dataset ( $D_{syn}$ ): A dataset was constructed with simplified premises (uppercase letters A–Z), random rules (8–18 per sample), and reasoning chains generated via the Breadth-First Search (BFS) algorithm.
Uncertain Token Identification: The authors first identified "uncertain tokens"—positions in the reasoning chain with low prediction probabilities (confidence < 0.8). Analysis revealed that three specific reasoning components account for the majority of this uncertainty:
1. Premise Selection: Choosing a fact from the Knowledge Base (KB) that satisfies a rule.
2. Premise Selection Termination: Deciding when to stop selecting premises for the current step.
3. Rule Selection: Determining which rule to apply given the selected premises.

2. Circuit Discovery via Activation and Path Patching

To identify the specific attention heads responsible for these components, the authors constructed pairs of clean and corrupted prompts. Corrupted prompts systematically modified causal elements (e.g., fact values, rule definitions, or traversal algorithms in demonstrations) to induce different token choices at the reasoning component positions.

Activation Patching: Used to score the contribution of individual attention heads by measuring the Average Indirect Effect (AIE). This involves patching clean activations into corrupted forward passes to see if the model recovers its original behavior.
Path Patching: Used to quantify the information flow between pairs of heads by systematically corrupting and restoring activations between them. This reveals the causal influence of one head on another.

3. Ablation Studies

To validate the necessity of the identified circuits, the authors performed ablation experiments where the top- $k$ identified reasoning heads were "knocked out" (set to zero activation). This was tested across:

The synthesized dataset ( $D_{syn}$ ).
Established logical reasoning benchmarks: ProntoQA and ProofWriter.
A general knowledge benchmark: MMLU.

Key Findings and Results

1. Localization of Reasoning Heads

Hierarchical Structure: The analysis reveals a clear temporal and hierarchical structure in the processing of reasoning tasks.
- Early-to-Middle Layers: Contain "Reading Heads" (e.g., Read Fact, Read Rule Condition) that extract relevant information from the problem description and propagate it through the residual stream.
- Middle-to-Higher Layers: Contain "Decision Heads" (e.g., Select Premise, Select Rule) that perform logical operations and integrate information.
Sparsity: Reasoning-critical heads are highly sparse. For instance, in the Llama-3.1-8B-Instruct model, a single head can account for over 30% of the effect for rule selection. Across all models, the identified reasoning heads constitute approximately 3% of the total attention heads.
Polysemanticity: Many attention heads perform multiple sub-tasks (e.g., reading rule conditions and facts), acting as centers for information integration.

2. Circuit Network Characteristics

Specialization: Specific circuits handle distinct reasoning components (e.g., a "Rule Condition Matching Circuit" transfers rule condition information from reading heads to decision heads).
Information Flow: Information flows from reading heads in lower layers to decision heads in deeper layers. Higher layers predominantly facilitate information integration and the emergence of global reasoning strategies (like graph traversal algorithms) that coordinate multiple intermediate steps.
Model Differences: While Llama-3.1-8B-Instruct shares heads primarily for causal reading roles, Qwen models tend to share heads across decision-making roles.

3. Impact of Ablation

Deductive Reasoning: Removing the identified reasoning heads (the /3Roles setting, ablating ~3% of heads) caused the reasoning ability of all LLMs to collapse to nearly zero on synthesized data and significantly degraded performance on ProntoQA and ProofWriter.
General Knowledge: On the MMLU dataset, ablating individual reasoning components had a limited impact comparable to random ablation. However, ablating all reasoning heads simultaneously caused a substantially larger degradation than random ablation, suggesting these heads are recruited even for broad knowledge retrieval, though their collective contribution is most critical for structured deduction.
Generalization: The circuits identified on the synthesized dataset generalized well to standard benchmarks (ProntoQA, ProofWriter), indicating they are not biased toward specific data features.

Significance and Claims

The paper claims to provide mechanistic insights into how LLMs internalize abstract reasoning algorithms from limited demonstrations. By revealing a sparse and modular circuit architecture that supports complex multi-hop inference, the work demonstrates that LLMs do not merely memorize patterns but activate specific, localized neural pathways to execute algorithmic reasoning steps.

The authors position this work as a step toward improving reasoning robustness, interpretability, and targeted interventions for future language models. They emphasize that their findings highlight the importance of specific attention heads in steering the reasoning process, particularly at "uncertain" token positions where the model must satisfy strict constraints imposed by the reasoning algorithm and few-shot demonstrations.

The study acknowledges limitations, noting that the analysis is primarily conducted on synthesized datasets with explicit logical structures and may not fully generalize to free-form text or implicit chain-of-thought processes in naturalistic contexts. Additionally, the focus remains on attention mechanisms, excluding potential contributions from MLP layers.