Technical Summary: Interpretability and Faithfulness of Chain-of-Thought Under Knowledge Conflict

Problem Statement

When large language models (LLMs) encounter a document contradicting their parametric training knowledge, they must choose between following the document or trusting their internal knowledge. Prior research established that this decision is driven by the "fame" of the entity (Wikipedia page views), where famous facts are resisted and obscure facts are followed. However, practitioners increasingly rely on Chain-of-Thought (CoT) reasoning to understand why a model made a specific choice.

The core problem addressed is introspective faithfulness: Does the stated CoT reasoning accurately reflect the actual decision-making mechanism? Specifically, when a model flips its decision on the same question (e.g., from following a distractor to resisting it), does the reasoning text change to reflect the new decision, or is the reasoning a static "knowledge display" that merely gets wrapped in different attribution phrases? If CoT is merely a post-hoc rationalization rather than a causal report, monitoring systems relying on logical arguments may provide false confidence.

Methodology

The study employs a behavioral probe across 200 factual questions (from the PopQA dataset) and 8 models (including GPT-4o, Claude Sonnet 4.6, Gemini 2.5, DeepSeek variants, and Llama 4 Maverick).

Experimental Design

• Knowledge Conflict Paradigm: Items were split into high-certainty (famous entities) and low-certainty (obscure entities) tiers. Each item included a plausible but incorrect distractor.
• Prompt Conditions:
  • C1/C2 (Multi-turn Conflict): The model answers a question, receives a contradicting document, and must reason step-by-step, rate confidence (1–5), and provide a final answer.
  • C3 (Document-only): The model answers based solely on the document (serving as a compliance anchor).
  • C4 (Own-knowledge): The model is instructed to prioritize its own knowledge over the document.
• Model Variants: The study includes "native-reasoning" models (exposing internal thinking tokens) and their "standard" counterparts (user-facing output only) to isolate the effect of the reasoning feature.
• Controls: Matched-temperature ablations (e.g., GPT-4o at $T=1$) were used to rule out temperature as a confounding factor for observed anomalies.

Analytical Measures

1. Reasoning Consistency: Cosine similarity (using all-MiniLM-L6-v2) and ROUGE-L scores were computed between CoT texts of "flip pairs" (where the model gives opposite answers on the same item) versus "same-answer pairs." Attribution phrases (e.g., "the document says") were stripped to isolate substantive content.
2. Confidence Calibration: Correlations were tested between self-rated confidence and (a) entity salience ($spop$) and (b) baseline correctness (whether the model knew the fact prior to the conflict).
3. Behavioral Coherence: Generalized Estimating Equations (GEE) logistic regression tested whether confidence independently predicts the decision to follow or resist, controlling for entity fame.
4. Source Attribution: Analysis of "FOLLOW" responses to detect confabulation (inventing facts to justify following a distractor).

Key Results

1. Reasoning Consistency: The "Flip" Phenomenon

When models reached opposite conclusions on the same question, their step-by-step reasoning remained highly similar.

• Similarity: Flip pairs retained 96% of the similarity found in same-answer pairs (Cosine similarity 0.784 vs. 0.817; $d=0.34$).
• Implication: The bulk of CoT is a static "knowledge display" (reciting facts about the entity) that does not change with the decision. Only the final attribution or framing changes.
• Exception: GPT-4o was the only model showing a statistically reliable negative delta ($\Delta = -0.132$), meaning its reasoning genuinely changed when its decision flipped. For other models, the logical argument is effectively invariant to the decision.

2. Confidence as the Genuine Signal

While the logical narrative was stable, self-rated confidence showed weak but genuine introspective signal.

• Item-Level Knowledge: In the low-certainty tier, entity fame ($spop$) did not predict confidence ($\rho = -0.005$). However, baseline correctness (whether the model actually knew the fact) weakly predicted confidence ($r = 0.134, p < 0.001$).
• Decision Prediction: Lower confidence predicted a higher probability of following the distractor ($coef = -0.475, p < 0.001$). This establishes an item-level pathway: knows fact $\rightarrow$ higher confidence $\rightarrow$ resists distractor.
• Model Specificity:
  • GPT-4o: Showed statistically reliable coupling between reasoning and decision.
  • Claude Sonnet 4.6: Exhibited a "Claude Anomaly." Its pooled confidence-decision correlation was near zero ($r \approx 0$) because the relationship reversed across conditions. Under "own-knowledge" framing, it followed distractors with high confidence, canceling out the negative correlation seen in conflict conditions. This suggests Claude tracks instruction framing rather than its internal knowledge state.

3. Internal vs. Stated Reasoning

For native-reasoning models, internal thinking tokens showed greater decision-sensitivity than the user-facing CoT.

• Internal tokens on flip pairs were less similar ($0.709$) than stated outputs ($0.787$), a significant gap ($p=0.033$).
• This suggests the user-facing output "smooths over" the differences preserved in the internal process, likely to present a coherent narrative.

4. Confabulation

Among models that followed distractors, Llama 4 Maverick was a clear outlier, exhibiting 15.2% confabulation (fabricating specific facts, e.g., a non-existent village, to validate the distractor). Other models showed rates below 6.3%.

Significance and Claims

The paper claims to provide the first direct test of whether stated reasoning tracks the empirical driver of knowledge-conflict decisions. Its primary contributions are:

1. Decomposition of CoT: The study decomposes CoT into two distinct functions:
   • Function 1 (Knowledge Display): A stable recitation of parametric knowledge that remains nearly identical regardless of the final decision.
   • Function 2 (Decision Justification): A thin layer of attribution and confidence scores that adjusts to the decision.
2. Monitoring Implications: The authors argue that for safety monitoring and auditing, logical arguments are the wrong metric to read because they are often invariant to the decision. Instead, expressed confidence (and hedges) is where genuine self-knowledge signal resides, though it is weak and subject to model-specific calibration issues (like the Claude anomaly).
3. Limitations of Introspection: The paper modestly concludes that models possess only "modest" item-level metacognition. While they can distinguish known from unknown facts slightly better than chance, their confidence scores are heavily influenced by entity fame and instruction framing rather than a precise internal state.

The study does not claim causal faithfulness but establishes correlational consistency between behavioral signals and a validated predictor ($spop$ and baseline correctness), suggesting that current CoT outputs often serve as post-hoc rationalizations rather than transparent reports of the decision mechanism.