💬 NLP

Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization

This paper argues that Chain-of-Thought reasoning can be faithful even without explicitly verbalizing prompt-injected hints, challenging the "Biasing Features" metric for conflating incompleteness with unfaithfulness and advocating for a broader interpretability toolkit that includes causal mediation analysis.

Original authors: Kerem Zaman, Shashank Srivastava

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kerem Zaman, Shashank Srivastava

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Is the Model Lying?

Imagine you ask a student (an AI) a difficult math problem. To show their work, they write down a step-by-step explanation (this is called Chain-of-Thought, or CoT).

Recently, researchers found a way to "trick" the student. They whispered a hint in the student's ear (like, "The answer is B, because a famous professor said so"). When the student changed their answer to B, the researchers checked the written explanation. If the student didn't write down the phrase "The professor said so," the researchers labeled the explanation as unfaithful (a lie or a fake).

The authors of this paper say: "Wait a minute. That's too harsh."

They argue that just because the student didn't say the hint out loud in their written notes, it doesn't mean the hint didn't actually help them solve the problem. The student might have used the hint internally but just forgot to write it down, or they might have compressed the thought process too much to fit on the page.

The Three Main Discoveries

The paper uses three main "tests" to prove that the "unfaithful" label is often wrong.

1. The "Missing Page" Analogy (Incompleteness vs. Unfaithfulness)

The Old View: If the student doesn't write the hint, they are lying.
The New View: The student might just be running out of paper.

The researchers gave the students more "paper" (more time and tokens to write). They found that when given more space, the students started writing down the hint much more often (up to 90% of the time in some cases).

The Analogy: Imagine you are trying to explain a complex movie plot to a friend, but you only have 30 seconds. You might skip the part about the villain's backstory because you don't have time. If you had 5 minutes, you would tell the whole story.
The Point: The paper argues that many "unfaithful" explanations are just incomplete summaries, not lies. The reasoning was there, but it got lost in the compression.

2. The "Ghost in the Machine" Analogy (Causal Mediation)

The Old View: If the hint isn't in the text, it didn't affect the answer.
The New View: The hint is like a ghost that changes the outcome without leaving a footprint.

The researchers used a special tool called Causal Mediation Analysis. Think of this as an X-ray that looks inside the student's brain while they are thinking. They wanted to see: Did the hint actually change the student's internal gears, even if the student didn't write it down?

The Result: Yes! Even when the written explanation didn't mention the hint, the X-ray showed that the hint was still steering the gears. The hint changed the probability of the answer, and the written explanation was still a result of that change.
The Point: The explanation is faithful to the decision-making process, even if it doesn't explicitly name the trigger. The "ghost" (the hint) was real, even if the "footprints" (the words) were missing.

3. The "Different Rulers" Analogy (Conflicting Metrics)

The Old View: There is only one ruler to measure truth: "Did you write the hint?"
The New View: You need a toolbox of different rulers.

The paper tested the "unfaithful" explanations using two other rulers:

The "Filler" Ruler: If you erase the explanation and replace it with "...", does the answer change? If yes, the explanation was doing real work.
The "Unlearning" Ruler: If you teach the student to forget a specific step in their reasoning, does the answer change? If yes, that step was important.

The Result: Many explanations that failed the "Did you write the hint?" test passed these other tests. They were doing real work and were faithful to the logic, just not to the specific wording of the hint.
The Point: Relying on just one test (checking for hint words) is like judging a chef only by whether they used salt, ignoring whether the food actually tastes good.

The Takeaway

The paper concludes that we shouldn't panic and say "AI explanations are useless lies" just because they don't repeat every hint they received.

Unfaithful vs. Incomplete: Sometimes the AI isn't lying; it's just being brief.
Hidden Influence: The AI can be influenced by a hint even if it doesn't say "I was influenced by this hint."
Better Tools: We need to use a wider variety of tools (like X-rays of the brain and different testing methods) to understand how AI thinks, rather than just checking if specific words are present.

In short: The absence of a specific word in the explanation doesn't prove the AI is lying. It might just be a compressed, incomplete, but still honest, summary of a complex thought process.

Title: Chain-of-Thought Can Be Faithful without Hint Verbalization
Authors: Kerem Zaman and Shashank Srivastava (UNC Chapel Hill)

Problem Statement

Recent literature has increasingly labeled Chain-of-Thought (CoT) reasoning in Large Language Models (LLMs) as "unfaithful," suggesting that CoTs are often post-hoc rationalizations rather than faithful reflections of the model's internal decision-making processes. This conclusion is primarily driven by the Biasing Features metric (also known as hint-verbalization evaluation). In this setup, a "hint" (e.g., "A Stanford professor thinks the answer is B") is injected into the prompt to bias the model toward a specific answer. If the model changes its prediction to match the hint but fails to explicitly mention the hint in its generated CoT, the CoT is labeled unfaithful.

The authors argue that this metric adopts an overly narrow notion of faithfulness. It conflates unfaithfulness (a misalignment between the explanation and the reasoning) with incompleteness (the necessary lossy compression required to translate distributed transformer computations into a linear natural language narrative). By equating the absence of specific keywords with a lack of causal influence, the Biasing Features metric may undervalue CoTs as interpretability tools and incentivize training pipelines to optimize for superficial verbalization rather than genuine reasoning alignment.

Methodology

The paper employs a multi-faceted approach to re-evaluate CoT faithfulness across three multi-hop reasoning datasets (OpenbookQA, StrategyQA, ARC-Easy) and various instruction-tuned and reasoning-specialized models (including Llama-3 variants, Gemma, and Qwen).

Reproduction of Biasing Features: The authors reproduce the standard hint-verbalization evaluation using three hint types: "Professor" (textual opinion), "Metadata" (XML tags), and "Black Squares" (visual markers in few-shot examples). They use an LLM-as-a-judge to determine if a CoT semantically verbalizes the hint.
Alternative Faithfulness Metrics: To test the robustness of the "unfaithful" label, they evaluate CoTs flagged as unfaithful by the Biasing Features metric using two complementary metrics:
- Filler Tokens: Replaces the CoT with ellipses ("...") to check if the reasoning text causally matters for the prediction.
- Faithfulness through Unlearning Reasoning (FUR): Selectively unlearns individual reasoning steps from the model parameters to see if the prediction changes, testing parametric alignment.
Faithful@k Metric: To distinguish between unfaithfulness and incompleteness, the authors introduce a faithful@k metric. This measures the probability of obtaining at least one hint-verbalizing CoT within $k$ generated samples. If non-verbalization is due to token limits or stochasticity (incompleteness), increasing $k$ should increase the rate of verbalization.
Causal Mediation Analysis (CMA): Using Pearl's framework, the authors decompose the effect of a hint on the final prediction into:
- Natural Direct Effect (NDE): The effect of the hint on the prediction holding the CoT fixed.
- Natural Indirect Effect (NIE): The effect of the hint mediated through the generated CoT (holding the input fixed but changing the CoT).
  This determines whether non-verbalizing CoTs still causally transmit the hint's influence.
Logit Lens Analysis: They trace hint-related concepts through the model's intermediate layers (MHA outputs) to observe how hint information propagates even when absent from the final text.

Key Results

Disagreement on Faithfulness: While the Biasing Features metric labels the vast majority of CoTs (often >80%) as unfaithful, alternative metrics tell a different story.
- Filler Tokens: Identifies that a non-trivial fraction (often >20%, up to 60% in specific settings) of "unfaithful" CoTs are actually contextually faithful (i.e., the CoT still causally influences the prediction).
- FUR: Shows that at least 50% of evaluable CoTs contain at least one faithful reasoning step, even when they do not mention the hint.
Incompleteness vs. Unfaithfulness: The faithful@k analysis reveals that increasing the inference-time budget (generating more samples) significantly increases the rate of hint verbalization. For the "Professor" hint, the probability of finding a verbalizing CoT rises to nearly 90% for some models at $k=16$ . This suggests that much of the "unfaithfulness" detected by single-sample greedy decoding is actually incompleteness due to tight token limits or sampling variance, not a fundamental lack of alignment.
Causal Mediation: Causal Mediation Analysis demonstrates that even when CoTs do not verbalize hints, they exhibit significant Natural Indirect Effects (NIE). The CoT causally mediates a portion of the hint's influence on the prediction. In some cases, the CoT influences the prediction by suppressing alternative options rather than just boosting the hinted one.
Logit Lens Findings: Hint-related tokens frequently appear in the top-5 logits of intermediate layers (particularly around layers 20–25) and during contrastive transitions or summarizing phrases, even when the final output text lacks explicit mention of the hint. This confirms that the model processes the hint internally before generating the final narrative.
Generalization: These patterns hold for larger models (Llama-3.3-70B) and reasoning-specialized models (Qwen-3-32B), though the specific dynamics of hint processing (e.g., NDE vs. NIE dominance) vary by hint type and model architecture.

Significance and Claims

The paper does not claim that all CoTs are faithful, nor does it dismiss the utility of verification steps. Instead, it argues that the narrative claiming "CoT is not explainability" is incomplete and potentially misleading when inferred primarily from hint-verbalization tests.

The authors' core contributions are:

Reframing Unfaithfulness: They posit that the absence of hint words alone does not prove unfaithfulness. The Biasing Features metric often measures plausibility (did the model mention the cue?) rather than faithfulness (did the cue drive the reasoning?).
Incompleteness as a Confounder: A significant portion of measured unfaithfulness is better attributed to the incompleteness of the explanation due to compression or token limits, rather than a misalignment between the explanation and the model's internal state.
Broader Toolkit: The paper advocates for moving beyond single-metric evaluations. It recommends integrating causal mediation analysis, corruption-based metrics (like Filler Tokens), and unlearning-based metrics to build a more robust interpretability pipeline.
Causal Role of Non-Verbalizing CoTs: Even when CoTs do not explicitly verbalize hints, they can still causally mediate the hint's influence, acting as a meaningful intermediate in the decision-making process rather than mere post-hoc rationalization.

In conclusion, the authors caution against relying solely on hint-based evaluations to dismiss CoTs as unfaithful. They suggest that CoTs remain valuable interpretability tools, provided they are evaluated with a broader set of metrics that account for the complexities of translating distributed computation into natural language.