LLMs Faithfully and Iteratively Compute Answers During CoT: A Systematic Analysis With Multi-step Arithmetics

1. Problem Statement

The paper addresses a critical question in the interpretability of Large Language Models (LLMs): Is the Chain-of-Thought (CoT) explanation a faithful reflection of the model's internal reasoning process, or is it a "post-hoc" rationalization?

Specifically, the authors investigate two sub-questions:

1. Timing of Determination: At what point during the generation process does the LLM internally "decide" on the final answer? Is it determined before the CoT begins (during the input reading phase), or is it computed iteratively as the CoT is generated?
2. Causal Faithfulness: Does the CoT text causally influence the final answer, or does the model simply generate a fluent explanation to match a pre-determined answer?

If the answer is determined before CoT generation, the CoT is unfaithful (a "post-hoc" explanation). If the answer is derived during CoT generation, the CoT is faithful.

2. Methodology

The authors employed a controlled experimental framework using synthetic multi-step arithmetic tasks to isolate reasoning mechanisms from natural language noise.

A. Dataset and Task Design

• Synthetic Arithmetic: They created a dataset of multi-hop arithmetic problems (e.g., $A = 1 + B, B = 2 + 3, A = ?$).
• Complexity Levels: Tasks were categorized into 5 levels based on:
  • #Step: Number of operations required.
  • #Stack: Number of variables that cannot be resolved immediately (requiring a "stack" or deferred resolution).
  • #Dist: Presence of distractor equations irrelevant to the final answer.
• Models Tested: Nine different LLMs were evaluated, including Qwen2.5 (7B, 14B, 32B, Math), Yi1.5, Llama 3.1/3.2, and Mistral-Nemo.

B. Linear Probing (Correlational Analysis)

To determine when the answer is encoded in the model's hidden states:

• Probes: Linear classifiers were trained to predict the value of specific variables (e.g., $A$, $B$) from the model's hidden states ($h_{t,l}$) at every token position $t$ and layer $l$.
• Metric ($t^*$): The authors defined $t^*$ as the earliest token position where the probe achieves high accuracy (threshold $\tau = 0.9$).
• Comparison: They compared accuracy in the Input phase ($t < 0$, before CoT) vs. the CoT/Output phase ($t \ge 0$).

C. Causal Intervention (Activation Patching)

To establish causality between internal states and the output:

• Technique: Activation Patching. The authors replaced hidden states from a "clean" run (solving problem $X$) with hidden states from a "different" run (solving problem $Y$) at specific grid positions (equations and layers).
• Goal: If patching the hidden state at a specific CoT step changes the final answer to match the "patched" problem, it proves that the hidden state at that step causally determines the output.
• Targets: They tested interventions on the Input part, intermediate CoT steps, and the final answer generation.

3. Key Results

A. Timing of Answer Derivation (Probing Results)

• No Pre-determination: For all models and complexity levels (except trivial cases with 0 steps), the probes failed to accurately predict the final answer during the Input phase ($t < 0$). Accuracy ($Acc_{\prec CoT}$) was consistently low (often < 30-40%).
• Iterative Computation: Accuracy ($Acc_{\succ CoT}$) spiked to near 100% only during the generation of the CoT.
  • The model computes sub-answers (e.g., calculating $B=5$) and encodes them in hidden states as it generates the text "B=5".
  • The final answer is derived incrementally, not retrieved from a pre-computed state.

B. Causal Faithfulness (Intervention Results)

• CoT Drives the Answer: Intervening on the hidden states corresponding to the CoT generation (specifically the most recent steps) successfully changed the final answer to match the patched input.
• Input Irrelevance: Intervening on the hidden states corresponding to the Input/Problem Statement (the "first pass") had almost no effect on the final answer or intermediate CoT steps.
• Recency Bias: The causal graph follows a strong recency bias. The final answer depends heavily on the immediately preceding CoT steps (e.g., the calculation of $B$), which in turn depend on the step before that. The model does not "remember" the initial problem statement in a way that directly dictates the answer without the intermediate CoT processing.

C. Robustness

• These findings held true across different model sizes (3B to 34B), different architectures (Qwen, Llama, Mistral), and varying task complexities.
• Even when models failed to solve a problem (low accuracy), the internal representation of the "correct" answer was absent, suggesting the model does not hallucinate an answer and then try to justify it; rather, if it cannot compute the answer, it does not have the representation.

4. Key Contributions

1. Evidence of Faithfulness: The paper provides strong empirical evidence that, for multi-step arithmetic reasoning, CoT is faithful. The reasoning chain is not a post-hoc rationalization but a faithful record of the model's internal, iterative computation.
2. Mechanistic Insight: It reveals that LLMs do not "know" the answer upon reading the prompt. Instead, they perform on-the-fly computation, updating their internal state with each generated token of the reasoning chain.
3. Methodological Framework: The study combines linear probing (to detect when information is present) with activation patching (to verify causal influence) in a controlled synthetic setting, offering a robust blueprint for analyzing reasoning in LLMs.
4. Recency Bias Discovery: The analysis highlights a "recency bias" in the causal flow of reasoning, where the model relies on the most recent intermediate results stored in the CoT text rather than re-reading the original problem statement.

5. Significance and Implications

• Trust in CoT: For tasks requiring step-by-step reasoning (like math or logic), users can have higher confidence that the CoT explanation reflects the actual decision-making process of the model, rather than a fabricated justification.
• Debugging and Alignment: Since the answer is computed iteratively, errors in CoT (e.g., a wrong intermediate step) directly cause the final error. This suggests that improving CoT generation (e.g., via self-correction or better prompting) is a viable path to improving model performance, as the CoT is the mechanism of computation.
• Limitations: The authors note that these results are specific to tasks requiring explicit reasoning. For simple factual recall or tasks where the answer is easily predictable from the prompt alone, the "post-hoc" behavior observed in other studies might still apply. However, for complex reasoning, the "faithful computation" mechanism dominates.

In conclusion, the paper refutes the hypothesis that LLMs pre-determine answers and merely generate CoT as a formality. Instead, it demonstrates that CoT is the engine of computation, and the model's internal state evolves faithfully alongside the generated text to arrive at the solution.