Technical Summary: Local Causal Attribution of Chain-of-Thought Reasoning

Problem Statement

The success of Large Reasoning Models (LRMs) relies heavily on Chain-of-Thought (CoT) reasoning, where models generate long sequences of intermediate steps. However, the faithfulness of these verbalized CoTs to the model's underlying latent reasoning process remains a critical concern. Research indicates that CoT text may sometimes ignore input manipulations or construct elaborate but false justifications, particularly when models are presented with incorrect hints. While previous work has explored causal structures at the level of internal representations (white-box) or distributions over reasoning trajectories (via resampling), there is a need for efficient, local methods to analyze the causal relationships within a specific CoT trace. Existing black-box methods often focus on answer-level attribution or fail to account for the impact of intermediate steps on subsequent reasoning steps.

Methodology: AttriCoT

The authors propose AttriCoT, a black-box method for estimating local causal attributions within a specific CoT trace. The approach is grounded in a Structural Causal Model (SCM) and operates with linear computational complexity relative to the number of reasoning units.

1. Structural Causal Model Formulation

The method treats a CoT trace and its input as a sequence of "units" ($x_1, \dots, x_S$ for input; $y_1, \dots, y_T$ for output steps and final answer). The core assumption is an auto-regressive property where the log probability of generating a unit $y_t$ depends on the presence or absence of prior units.

• Scalarization: The unit $y_t$ is scalarized using its conditional log probability: $S_t(y_t) = \log P(y_t | x, y_{<t})$.
• Linear Structural Equation: The influence of prior units is modeled via a linear equation:
  $$S_t(y_t) = \sum_{s=1}^S \alpha_{s,t} I(x_s) + \sum_{\tau=1}^{t-1} \beta_{\tau,t} I(y_\tau) + \gamma_t + u_t$$
  Here, $I(\cdot)$ are indicator functions (1 if present, 0 if removed), and coefficients $\alpha_{s,t}$ and $\beta_{\tau,t}$ represent the causal importance of input and prior output units, respectively.

2. The AttriCoT Algorithm

The algorithm estimates these coefficients through three steps:

1. Intervention: The method performs interventions by removing specific units from the sequence. The default strategy is "Leave-One-Out" (LOO), removing one unit at a time ($N = S + T$ interventions). An enhanced variant ("AttriCoT-2x") includes "Leave-Two-Out" interventions.
2. Measurement: For each perturbed sequence, the model performs a forward pass to compute the log probabilities of the remaining output units. The change in log probability for a target unit $y_t$ serves as the target value for regression.
3. Regression: A linear regression (with optional L1 regularization) is fitted to estimate the coefficients $\alpha$ and $\beta$. This process requires $O(N)$ forward passes, maintaining linear complexity with respect to the number of units.

3. Evaluation Metric

Attribution quality is evaluated using Behavioral Faithfulness, measured by the Area Under the Perturbation Curve (AUPC).

• Procedure: Prior units are ranked by their attribution scores. The top $k$ units are removed, and the resulting decrease in the log probability of the target unit is measured.
• Metric: The AUPC aggregates these decreases as $k$ varies from 0 to 20% of the prior units. A higher AUPC indicates that the attribution method correctly identifies units that are causally critical to the generation of subsequent steps.

Key Contributions

1. AttriCoT Framework: The proposal of a black-box, local explanation method that attributes CoT steps to prior units using a structural causal model, requiring only a linear number of forward passes ($O(S+T)$).
2. Superior Faithfulness: Empirical evidence demonstrating that AttriCoT produces attributions more faithful to model behavior than alternative methods, including sentence-to-sentence masking (Thought Anchors) and prompting-based baselines.
3. Analytical Metrics: The introduction of metrics derived from attribution matrices (e.g., score density, average influence, input ratio) to reveal structural differences in reasoning across models and domains.

Experimental Results

The method was evaluated on 5 datasets (GSM8K, MATH500, GPQA-Diamond, MMLU-Pro, ZebraLogic) and 4 reasoning models (DS-Llama-8B, DS-Qwen-14B, DS-Qwen3-8B, Qwen3-8B).

• Performance: AttriCoT consistently achieved the highest AUPC scores across all models and datasets. On GSM8K, it showed improvements of 15–30% over Thought Anchors and 70–165% over prompting baselines.
• Efficiency: Unlike resampling-based methods, AttriCoT avoids the high computational cost of generating new CoT trajectories, relying instead on non-generative forward passes.
• Comparative Analysis:
  • Thought Anchors: While competitive, Thought Anchors (which uses KL divergence on token distributions) generally underperformed AttriCoT, likely because it does not measure the direct impact on the log probability of the generated unit in the same local manner.
  • Prompting: Prompting LLMs to rank units (even with In-Context Learning) yielded significantly lower faithfulness scores.
  • White-box Attention: An exploratory white-box method intervening on attention weights showed promise but did not surpass the causal intervention methods.

Analysis of Reasoning Structures

By analyzing the attribution matrices, the authors observed distinct patterns:

• Influence Decay: Importance generally decays rapidly with distance between steps but flattens into a long tail.
• Input Reliance: The "Input Ratio" metric reveals that models rely heavily on input units at the beginning of a trace. Interestingly, this reliance often increases again in the mid-to-late stages (around 50–80% of the trace), suggesting models refer back to the input context to verify or finalize answers.
• Model Differences: Larger or distilled models (e.g., DS-Llama-8B) exhibited higher score densities and different influence patterns compared to non-distilled counterparts, indicating structural variations in how they process reasoning.

Significance and Claims

The paper claims that treating CoT traces as causal objects provides a vital framework for transparency and safety. By decomposing a trace into user-defined units and fitting structural equations, AttriCoT offers a computationally efficient way to understand why specific reasoning steps are produced and whether intermediate reasoning is consequential.

The authors emphasize that their work addresses the "local" level of causality (specific traces) rather than internal mechanisms or trajectory distributions. They conclude that the resulting attribution matrices reveal suggestive patterns, such as the special influence of initial and final steps and the tendency of models to revisit inputs during verification. The authors acknowledge limitations, including the expressiveness of linear structural equations and the simplicity of the default LOO intervention, positioning future work toward developing computationally cheap white-box methods and investigating how reasoning structures depend on model families and problem difficulty.