💬 NLP

Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillation

This paper challenges the assumption that Chain-of-Thought traces are inherently valid and interpretable by demonstrating that trace correctness does not reliably predict final answer accuracy and that the most performant models often generate traces that are less interpretable and more cognitively demanding for users.

Original authors: Siddhant Bhambri, Upasana Biswas, Subbarao Kambhampati

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Siddhant Bhambri, Upasana Biswas, Subbarao Kambhampati

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a brilliant but eccentric tutor to teach a group of students how to solve complex puzzles. This tutor (the Large Language Model) is famous for two things:

Getting the right answer.
Talking to themselves out loud while they solve the puzzle, explaining every single step of their thought process. This "talking out loud" is called a Chain-of-Thought (CoT) trace.

For a long time, researchers assumed that if the tutor's "talking out loud" was logical, clear, and factually correct, then the students (smaller AI models) would learn better, and the students would also be able to explain their answers clearly to you.

This paper is like a reality check that says: "Wait a minute. That assumption is wrong."

Here is the breakdown of what the researchers found, using simple analogies:

1. The "Fake It Till You Make It" Phenomenon

The researchers set up a controlled experiment. They created two types of tutors for the students:

The Honest Tutor: Always gives the correct step-by-step reasoning and the correct final answer.
The "Confident" Tutor: Gives the wrong reasoning steps (like saying "2 + 2 = 5") but somehow still manages to write down the correct final answer at the end.

The Shocking Result:
The students who learned from the "Confident" Tutor (with the wrong steps) actually got better scores on the final test than the students who learned from the "Honest" Tutor.

The Analogy: Imagine a student who memorizes the wrong formula for a math problem but somehow guesses the right answer because they've seen the answer key before. They get an 'A' on the test, even though they don't actually understand the math. The AI is doing the same thing: it's learning the pattern of how to write a solution, not necessarily the truth behind the steps.

Takeaway: Just because an AI gives you a long, detailed explanation doesn't mean the explanation is true. The AI can get the right answer even if its "reasoning" is complete nonsense.

2. The "Verbose vs. Clear" Dilemma

Next, the researchers looked at how easy these explanations were for humans to understand. They compared four types of explanations:

The Raw R1 Trace: The AI's original, unedited, rambling internal monologue. (Think of a professor who talks for 20 minutes, goes off on tangents, and uses big words, but eventually gets to the point).
The Summary: A short, clean version of that rambling.
The Explanation: A human-like rewrite of the reasoning.
The "Correct" Trace: The simple, logical steps the researchers created.

The Shocking Result:

For the AI's Performance: The Raw R1 Trace (the rambling, messy one) made the AI perform the best.
For Human Understanding: The Raw R1 Trace was rated as the worst. Humans found it confusing, exhausting, and hard to follow. It was like trying to read a novel written in a language you barely know.
For Human Understanding: The simple, logical "Correct" traces were rated as the easiest to understand, but training the AI on these made the AI perform worse on the actual test.

The Analogy:
Imagine a chef who makes a delicious meal (the correct answer).

Method A: The chef explains the recipe by shouting every single ingredient, mixing technique, and random memory of a farm they visited in 1998. You get the best meal, but you are too confused to learn how to cook it yourself.
Method B: The chef gives you a simple, clear recipe card. You can easily learn to cook, but the meal you make isn't quite as good as the chef's.

3. The Big Disconnect

The paper concludes that there is a massive gap between what helps the AI think and what helps humans understand.

For the AI: It thrives on messy, verbose, and sometimes factually incorrect "thinking" traces. It uses them as a crutch to find the right answer, even if the crutch is broken.
For Humans: We need clear, concise, and truthful explanations to trust the AI.

Why Does This Matter?

Currently, many AI companies are training their models to "think" by showing them these long, messy traces, assuming that if the AI explains itself well, it's being honest.

This paper warns us:

Don't trust the explanation just because it's long. An AI can be confidently wrong in its reasoning but right in its answer.
Don't assume the AI's "thought process" is for you. The way the AI "thinks" to get the best score is often the exact opposite of what a human needs to understand the answer.

The Final Lesson:
We need to stop treating the AI's internal "thinking" as a transparent window into its logic. Instead, we should treat the AI's reasoning as a black box that produces good results, and then build a separate, simpler system to translate those results into something humans can actually understand and trust.

In short: The AI is a genius who solves puzzles by talking to itself in a confusing way. If we want to learn from it, we shouldn't try to understand its messy internal monologue; we should just ask it for the answer and then have a human (or a simpler AI) translate that answer into plain English for us.

1. Problem Statement

Recent Large Reasoning Models (LRMs), such as DeepSeek R1, utilize Chain-of-Thought (CoT) traces—intermediate reasoning steps generated before a final answer—to improve performance. These traces are increasingly used as supervision signals for Knowledge Distillation (KD) to train smaller models.

The paper challenges a prevailing, implicit assumption in the field: that CoT traces emitted at inference time are both semantically correct (factually accurate) and interpretable to end-users.

The Gap: While traces are used to boost model accuracy, there is little evidence that the correctness of the reasoning steps correlates with the correctness of the final answer, nor that these traces are actually understandable to humans.
The Core Question: Must CoT traces be semantically correct and interpretable to end-users to enhance LLM task performance?

2. Methodology

The authors designed a controlled experimental framework to decouple trace semantics from final answer accuracy and to evaluate human interpretability.

A. Rule-Based Problem Decomposition (Dataset Construction)

To create verifiable ground truth for intermediate steps, the authors focused on Open Book Question Answering (QA) tasks (specifically temporal reasoning).

Decomposition: Complex QA problems were broken down into two structured sub-steps:
1. Classification: Identifying the type of temporal relation (e.g., "during," "overlap").
2. Information Retrieval (IR): Identifying the specific fact in the text that satisfies the relation.
SFT Dataset Creation: They constructed Supervised Fine-Tuning (SFT) datasets with four distinct trace types:
1. Correct Traces: Verifiably correct classification and IR steps paired with the correct final answer.
2. Incorrect Traces: Verifiably incorrect classification and IR steps (hallucinated facts or wrong relations) paired with the correct final answer.
3. R1 Traces: Verbose, unstructured traces generated by DeepSeek R1.
4. Summaries & Explanations: LLM-generated (GPT-4o-mini) summaries and post-hoc explanations of R1 traces.

B. Experimental Setup

Models: Fine-tuned Small Language Models (SLMs) including Llama-3.2-1B, Qwen3-1.7B, and larger variants (Llama-3.1-8B, Qwen3-8B).
Datasets: CoTemp QA (temporal reasoning), Microsoft MARCO QA, and Facebook bAbI QA.
Evaluation Metrics:
- Task Performance: Final answer accuracy, F1, Precision, Recall.
- Trace Accuracy: Accuracy of the Classification and IR sub-steps.
- Human Interpretability: A user study with 100 participants rating traces on a 5-point Likert scale for predictability, comprehensibility, interpretability, and faithfulness, alongside cognitive load (NASA-TLX metrics: mental demand, effort, frustration).

3. Key Contributions & Findings

Finding 1: Trace Correctness $\nleftrightarrow$ Final Answer Accuracy

The study reveals a lack of reliable correlation between the semantic correctness of intermediate traces and the accuracy of the final solution.

Incorrect Traces Work: Models fine-tuned on incorrect traces (hallucinated reasoning) often achieved higher or comparable final answer accuracy compared to models trained on correct traces.
- Example: On the CoTemp dataset, "Incorrect Trace" SFT yielded 63.88% accuracy, while "Correct Trace" SFT yielded 52.88%.
False Positives: Confusion matrix analysis showed a high rate of cases where models produced correct answers based on incorrect traces (e.g., 71.54% in bAbI QA).
Mechanism: The authors hypothesize that SLMs learn the structural patterns of the trace format (e.g., "Step 1: Classify, Step 2: Retrieve") rather than the semantic content, allowing them to bypass the reasoning logic and jump to the correct answer if the final answer is present in the training pairs.

Finding 2: Performance $\nleftrightarrow$ Interpretability

There is a negative correlation between model performance and human interpretability.

Best Performance, Worst Interpretability: Fine-tuning on DeepSeek R1 traces (verbose, unstructured) produced the highest task performance but was rated as the least interpretable by human users.
- R1 Trace Scores: Interpretability = 3.31/5; Cognitive Load (Mental Demand) = 4.65/5 (High).
Best Interpretability, Lower Performance: Algorithmically generated correct traces (structured, concise) were rated as the most interpretable but resulted in lower final solution accuracy compared to R1 traces.
- Correct Trace Scores: Interpretability = 4.86/5; Cognitive Load = 2.31/5 (Low).
Summaries/Explanations: Summarized or post-hoc explained R1 traces offered a middle ground, improving interpretability over raw R1 traces but failing to match the performance of raw R1 traces.

4. Statistical Validation

Independence Test: A Chi-squared ( $\chi^2$ ) test confirmed that trace accuracy and answer accuracy are statistically independent (Null hypothesis accepted; $p > 0.05$ ).
User Study: Mann-Whitney U tests confirmed significant differences ( $p < 0.05$ ) between R1 traces and Correct Traces regarding both interpretability and cognitive load.

5. Significance and Implications

This work fundamentally challenges current practices in LLM supervision and Knowledge Distillation:

Decoupling Objectives: The "reasoning" traces used to train models are often distinct from the "explanations" users need. Optimizing for task performance via verbose, unstructured traces may actively degrade human trust and understanding.
The "Albatross" Effect: Semantic correctness and human interpretability can act as an "albatross" (burden) to raw model accuracy. Models may perform better by ignoring the semantic validity of their own reasoning steps.
Future Directions: The authors argue for decoupling model supervision objectives from end-user-facing trace design.
- Training should focus on performance (potentially using structural or even "reasonless" traces).
- Interpretability should be handled via separate modules (e.g., post-hoc summarization or dedicated explanation generators) rather than relying on the raw inference traces.
Anthropomorphism Warning: The paper cautions against anthropomorphizing intermediate tokens as "reasoning," suggesting they may function more as statistical artifacts that correlate with answers rather than genuine logical derivations.

Conclusion

The paper demonstrates that in the pursuit of high-accuracy LLMs, the intermediate traces generated are often semantically unfaithful and cognitively burdensome for humans. The assumption that "better reasoning traces lead to better answers and better understanding" is false; in fact, the most effective traces for model performance are often the least understandable to humans. This necessitates a shift in how AI systems are designed, separating the optimization of model reasoning from the generation of human-interpretable explanations.