💬 NLP

Mechanistic Evidence for Faithfulness Decay in Chain-of-Thought Reasoning

This paper introduces Normalized Logit Difference Decay (NLDD), a metric that reveals a consistent "Reasoning Horizon" in Chain-of-Thought models where steps beyond 70–85% of the chain length lose faithfulness, demonstrating that accuracy alone does not guarantee genuine reasoning.

Original authors: Donald Ye, Max Loffgren, Om Kotadia, Linus Wong, Jonas Rohweder

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Donald Ye, Max Loffgren, Om Kotadia, Linus Wong, Jonas Rohweder

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a magician perform a complex trick. They pull a rabbit out of a hat, but before they do, they give you a long, detailed speech about how they are going to do it. They explain the steps, the physics, and the magic words.

Now, here is the big question: Did the rabbit actually come out because of the magic words they just said, or did the magician already have the rabbit in their pocket the whole time, and the speech was just a fancy distraction to make you think they were doing real magic?

This paper asks that exact question about Artificial Intelligence (AI) models. When an AI solves a hard math or logic problem, it often writes out a "Chain of Thought" (CoT)—a step-by-step explanation of its work. The authors want to know: Is the AI actually using those steps to figure out the answer, or is it just making up a story after it has already guessed the answer?

The New Tool: The "Confidence Drop" Test (NLDD)

To find the answer, the authors invented a new test called NLDD (Normalized Logit Difference Decay).

Think of it like this: Imagine you are driving a car, and your GPS gives you turn-by-turn directions.

Faithful Driving: If the GPS says "Turn left," and you turn left, you get to your destination. If you ignore the GPS and turn right, you get lost. The directions mattered.
Unfaithful Driving: If the GPS says "Turn left," but you were already going to turn left because you knew the way, the directions didn't actually change your path. Or worse, if the GPS says "Turn left" and you still turn left, but the GPS was actually trying to trick you into turning right, then the directions were a distraction.

The authors' test works by sabotaging the AI's explanation. They take a perfect explanation, change one step to be wrong (like telling the AI "Add 2 + 2 = 5"), and then see what happens to the AI's confidence in its final answer.

If the AI is "Faithful": When they break the explanation, the AI gets confused and loses confidence. This proves the AI was actually using the steps to think.
If the AI is "Unfaithful" (or "Anti-Faithful"): When they break the explanation, the AI's confidence doesn't drop, or it might even go up! This proves the AI didn't need the steps at all; it already knew the answer (or guessed it) before it started writing.

The Big Discovery: The "Reasoning Horizon"

The authors tested three different types of AI models (DeepSeek, Llama, and Gemma) on three types of puzzles (math, logic, and language rules). They found something surprising: AI reasoning has a "horizon."

Imagine a long rope. The first 70% to 85% of the rope is strong and useful. But the last 15% to 30%? It's just loose, dangling string.

The Sweet Spot: For all the models, the AI actually needs the first part of its explanation to solve the problem.
The Horizon: Once the AI passes a certain point (about 70–85% of the way through its explanation), writing more steps doesn't help. In fact, for some models, writing more steps actually hurts their performance. It's like the AI starts talking just to fill the silence, even though it already solved the problem.

The "Clever Hans" Problem

The paper highlights a scary phenomenon called the "Clever Hans" effect. In the 1900s, there was a horse named Hans who seemed to do math. He would tap his hoof the right number of times. But it turned out he wasn't doing math; he was reading the body language of the person asking the question.

The authors found that some AI models (specifically the Gemma model) act like Clever Hans.

They can get 99% of the answers right.
But when the authors broke the "reasoning steps," the AI didn't care.
This means the AI isn't actually reasoning through the steps; it's just memorizing patterns or guessing the answer, and the "explanation" is just a post-hoc story it tells to look smart.

The "Hidden Map" vs. The "Driver"

The paper also found a weird disconnect called the "Mapping Gap."

Imagine a driver who has a perfect map of the city in their head (the AI's internal brain), but they are driving with their eyes closed, following a random path.

The AI has the correct information inside its "brain" (the map is there).
But it isn't using that information to make the final decision.

The authors showed that even when an AI fails a task completely, it might still have the correct "map" inside its layers. Conversely, an AI might get the right answer but have a messy, confused map inside. This proves that just because an AI can show you its work, doesn't mean it did the work.

Summary

In simple terms, this paper says:

Don't trust the explanation just because it looks smart. Sometimes the AI is just making up a story after it already knows the answer.
There is a "tipping point." AI models only really need the first 70–85% of their explanation to solve a problem. The rest is often just noise.
Accuracy isn't everything. An AI can get the right answer 100% of the time but still be "lying" about how it got there.

The authors created a new tool (NLDD) to measure this, helping us understand when an AI is truly thinking and when it's just pretending.

Technical Summary: Mechanistic Evidence for Faithfulness Decay in Chain-of-Thought Reasoning

Problem Statement
Chain-of-Thought (CoT) prompting has become the standard paradigm for eliciting complex reasoning in Large Language Models (LLMs). However, a critical ambiguity remains: do these step-by-step explanations faithfully reflect the model's internal decision-making process, or are they merely plausible post-hoc rationalizations? Existing evaluations often treat faithfulness as a binary property of the final answer or rely on raw sensitivity metrics that are not comparable across different model architectures due to variations in output scaling and baseline confidence. Furthermore, it is unclear whether models genuinely utilize generated reasoning tokens to compute answers or if they rely on pre-computed priors, rendering the reasoning chain causally irrelevant.

Methodology
The authors propose a diagnostic framework centered on Normalized Logit Difference Decay (NLDD), a metric designed to quantify step-level faithfulness. The methodology involves three core components:

Counterfactual Interventions: The framework corrupts individual reasoning steps within a CoT chain (e.g., introducing arithmetic errors, entity substitutions, or depth errors) and truncates subsequent steps. This isolates the causal impact of specific tokens on the final prediction.
NLDD Metric: Unlike binary accuracy, NLDD measures the degradation in a model's confidence (logit margin) when a reasoning step is corrupted. Crucially, it normalizes this degradation against the model's intrinsic output variability (standard deviation of logits) to enable architecture-agnostic cross-model comparison.
- Positive NLDD: Indicates causal reliance; corruption reduces confidence.
- Negative NLDD: Indicates "anti-faithfulness"; corruption paradoxically increases confidence, suggesting the model ignores or is hindered by the reasoning chain.
Structural Diagnostics: To complement behavioral metrics, the authors employ:
- Representational Similarity Analysis (RSA): Measures the alignment between hidden state patterns of clean vs. corrupted chains to assess internal representational stability.
- Trajectory Alignment Score (TAS): Quantifies the geometric efficiency of the reasoning path in latent space.
- Linear Probing: Tests whether task-relevant information (e.g., stack depth, truth values) is explicitly encoded in hidden states, independent of whether the model uses it for prediction.

The framework is evaluated across three model families (DeepSeek-Coder-6.7B, Llama-3.1-8B, and Gemma-2-9B) and three tasks with varying ambiguity: Dyck-n (syntactic), PrOntoQA (logical), and GSM8K (arithmetic).

Key Results
The study identifies distinct behavioral regimes and a consistent structural phenomenon across models:

Faithful vs. Anti-Faithful Regimes:
- Faithful Regime: Models like Llama and DeepSeek exhibit high positive NLDD, indicating their predictions are causally grounded in the reasoning chain.
- Anti-Faithful Regime: Gemma-2 demonstrates high accuracy (e.g., 99% on PrOntoQA) despite significantly negative NLDD (-52.5%). In this regime, corrupting the reasoning chain actually improves the model's confidence in the correct answer, suggesting the reasoning tokens act as distractors rather than drivers.
The Mapping Gap: A dissociation was observed where models encode task-relevant structures internally (high linear probe accuracy) but fail to utilize them for the final output. For instance, Gemma achieved 82% probe accuracy on Dyck-n stack depth but 0% task accuracy, indicating the information is present but not causally utilized.
The Reasoning Horizon ( $k^*$ ): The authors identify a "Reasoning Horizon," defined as the step position where causal influence (NLDD) peaks. Beyond this point (occurring at 70–85% of chain length across tasks), additional reasoning tokens contribute negligible or negative causal influence.
- Crucially, while NLDD drops sharply beyond $k^*$ , RSA and TAS remain stable. This suggests that models continue to generate coherent internal representations and geometric trajectories even when those steps no longer influence the final prediction.

Significance and Contributions
The paper makes three primary contributions to the mechanistic understanding of LLM reasoning:

NLDD as a Faithfulness Metric: It introduces a standardized, architecture-agnostic metric that reveals whether a model's reasoning is causal or decorative. It demonstrates that high accuracy alone is insufficient to verify reasoning, as models can achieve perfect scores while being completely disconnected from their CoT explanations.
Identification of the Reasoning Horizon ( $k^*$ ): The discovery of a consistent transition point (70–85% of chain length) where causal influence decays suggests that CoT chains often contain a "theoretical pruning zone." Beyond this horizon, reasoning tokens may be post-hoc formatting or, in anti-faithful models, active interference.
Evidence of the Mapping Gap: The findings challenge assumptions in interpretability research by showing that models can maintain correct internal representations of task logic without causally utilizing them for decoding. This implies that surface-level coherence in CoT does not guarantee causal faithfulness.

The authors conclude that their multi-metric approach provides a rigorous way to measure when CoT matters, distinguishing between genuine reasoning breakdown and superficial output changes, and highlighting that for certain architectures, extended reasoning can be counterproductive.