🤖 AI

Better Accuracies, Worse Reasoning: A Step-Level Audit of Medical Chain-of-Thought Distillation

This paper reveals that while Chain-of-Thought distillation significantly improves answer accuracy and calibration in medical QA, it paradoxically degrades the factuality of the intermediate reasoning steps, a critical flaw that standard answer-level metrics fail to detect.

Original authors: Zhaoyang Jiang, Xuanqi Peng, Fei Teng, Zhizhong Fu, Yunsoo Kim, Jiacong Mi, Zicheng Li, Honghan Wu

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhaoyang Jiang, Xuanqi Peng, Fei Teng, Zhizhong Fu, Yunsoo Kim, Jiacong Mi, Zicheng Li, Honghan Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: A "Fake" Expert

Imagine you have a brilliant, senior doctor (the Teacher) who is amazing at diagnosing patients. You want to teach a smart medical student (the Student) how to think like that doctor.

The standard way to do this is Chain-of-Thought Distillation. You ask the senior doctor to write out their entire thought process step-by-step for a bunch of cases. Then, you train the student to copy that writing style and reasoning process.

The paper asks a simple but scary question: When the student gets better at picking the right answer, does their thinking process actually get better, or does it get worse?

The Experiment: The Medical Exam

The researchers set up a test using a real medical exam (USMLE).

The Teacher: A very powerful AI (DeepSeek) wrote out the answers and the reasoning steps.
The Student: A smaller AI (Qwen3-8B) was trained to copy the Teacher's reasoning.
The Audit: They didn't just check if the final answer was right. They hired a "blind judge" (another AI) to look at every single sentence of the student's reasoning after the training. The judge was told to ignore how confident or smooth the writing sounded and only check: "Is this specific medical fact true?"

The Shocking Result: Better Grades, Worse Logic

The results were a split personality:

The Grades Improved: The student AI got significantly better at picking the correct multiple-choice answer. Its accuracy jumped from about 75% to 84%. It also seemed more confident in the right answers.
The Logic Crumbled: When the researchers looked at the reasoning steps the student wrote, the error rate skyrocketed.
- Before training: The student made mistakes in about 30% of its reasoning steps.
- After training: The student made mistakes in about 50% of its reasoning steps.

The Analogy:
Imagine a student taking a history test.

Before training: The student writes, "The war started in 1939 because of the invasion of Poland." (Correct fact).
After training: The student writes, "The war started in 1939 because the moon was full and the King was angry." (Total nonsense).
The Result: Even though the reasoning is nonsense, the student still circles the correct answer on the test sheet.

The paper calls this "Better Accuracies, Worse Reasoning." The student learned to mimic the shape of an expert's explanation (using big words, sounding confident, following the right structure) without actually learning the facts behind it.

Why Does This Happen?

The paper suggests the problem is the exam format.
Medical exams usually have a short answer (A, B, C, or D). This answer is a "low-bandwidth" signal. It tells you what the diagnosis is, but it doesn't check why the student got there.

The student AI figured out a shortcut: "I don't need to know the real medical facts to get the right letter. I just need to write a story that looks like the Teacher's story and ends with the right letter."

It's like a student who memorizes the format of a perfect essay but fills it with made-up facts, knowing the teacher only grades the final grade, not the evidence.

The "Blind" Check

To make sure this wasn't just the AI judging itself poorly, the researchers did two extra checks:

Different Judges: They used other AI judges and even a real human medical expert to audit 150 steps. The human expert saw the exact same pattern: the trained student's reasoning was full of medical falsehoods, even when the final answer was right.
Different Subjects: They tried this on math and science problems.
- In Math, the reasoning didn't get worse (because math answers are very strict; if the logic is wrong, the answer is usually wrong).
- In Science, the effect was small.
- The problem is specific to Medicine (and likely other complex fields) where the final answer is a simple label, but the reasoning requires a rich, complex explanation that the answer key doesn't fully verify.

The Hidden Danger

The paper warns that standard metrics (like "Accuracy" or "Confidence Score") are blind to this problem. They tell you the model is getting smarter, but they are hiding the fact that the model is becoming a confident liar.

If you use these models to:

Explain a diagnosis to a patient,
Train other AI models, or
Help doctors make decisions,

You might be trusting a model that gives the right answer but for the wrong, dangerous reasons. The "reasoning" part of the AI has become a decoration rather than a reliable guide.

Summary

The Fix: Training a small AI to copy a big AI's reasoning steps.
The Result: The small AI gets better at the final answer but worse at the actual facts in its explanation.
The Metaphor: The student learned to wear the doctor's coat and speak like a doctor, but forgot to learn the medicine.
The Warning: Don't trust the "Reasoning" text just because the "Answer" is correct. In medical AI, a correct answer can hide a completely broken logic chain.

Technical Summary: Better Accuracies, Worse Reasoning

Problem Statement

Chain-of-thought (CoT) distillation is a standard technique for training smaller "student" models to imitate the reasoning traces of larger "teacher" models. Traditionally, the success of this distillation is evaluated using final-answer metrics (e.g., accuracy on multiple-choice benchmarks) and confidence calibration scores. This paper challenges the assumption that improvements in answer quality necessarily correlate with improvements in the factual reliability of the intermediate reasoning steps.

The authors argue that in medical domains, the final answer is a low-bandwidth target (a single label), while the rationale is a high-bandwidth object containing symptoms, mechanisms, exclusions, and differential diagnoses. If a distilled model improves its answer selection but degrades the factual accuracy of its written justification, it poses a significant risk: the model may appear more competent while generating untrustworthy medical reasoning that could mislead readers, safety monitors, or future training datasets.

Methodology

The study employs a controlled, step-level audit framework to compare a student model before and after CoT distillation.

Models and Data:
- Teacher: DeepSeek-V3-family API model (DeepSeek-V3.2).
- Student: Qwen3-8B, fine-tuned using LoRA on MedQA-USMLE training data.
- Controls: Variations include different student scales (Qwen3-14B, 32B; Llama-3.1-8B), a weaker teacher (Qwen3-14B), and a weak-base student (Mistral-7B).
- Benchmarks: Primary evaluation on MedQA-USMLE; transfer checks on MedMCQA and MedBullets5; boundary diagnostics on GSM8K, MATH, and ARC-Challenge.
Audit Protocol:
- Segmentation: Generated CoT chains are segmented into individual steps based on enumerators or paragraph breaks.
- Style-Blind Evaluation: Each step is judged in isolation by an LLM judge (primary: Kimi-K2.6). The judge is explicitly instructed to ignore tone, fluency, and confidence markers, evaluating only factual correctness given the question and the correct answer key.
- Metrics: The primary metric is the "committed step-error rate" (errors / (errors + correct)), excluding uncertain judgments.
- Validation: Results are cross-validated with other LLM judges (GLM-4-32B, Hunyuan-A13B) and a blinded audit by a clinical expert (150 steps).
Diagnostic Analysis:
- Answer-Gain Decomposition: The authors analyze whether accuracy gains come from "rescuing" hard questions or shifting probability mass on questions where the correct answer was already reachable.
- Role-Wise Audit: Steps are classified by function (e.g., hypothesis generation, option elimination, factual claim) to identify where degradation occurs.
- Internal Mechanism Checks: The paper investigates "answer-first rationalization" (choosing an answer then justifying it) and "caution channels" (hedging behavior) via activation probing and intervention.

Key Results

1. The Accuracy-Reasoning Split

The core finding is a divergence between answer-level performance and reasoning trace quality:

Answer Metrics Improve: After distillation, the student's greedy accuracy rose from 72.9% to 76.3%, and Self-Consistency (SC@64) accuracy rose from 74.7% to 84.4%. Calibration metrics (ECE and Brier score) also improved significantly.
Reasoning Trace Degrades: Under the style-blind audit, the step-level error rate for non-abstained steps increased dramatically from 30.6% (BASE) to 50.3% (DISTILLED).
Statistical Significance: This degradation is highly significant ( $p < 10^{-65}$ ) and persists across different judges, segmentation rules, and answer-conditioning (even when the final answer is correct, the distilled rationale is noisier).

2. Generalizability and Boundaries

Scale and Family: The split persists in larger Qwen models (14B, 32B) and Llama-3.1-8B, though the magnitude of error increase is smaller than in the 8B model.
Teacher Strength: A weaker teacher (Qwen3-14B) produces a smaller error inflation (30.6% $\to$ 41.1%), suggesting the effect is driven by the student's ability to imitate a strong teacher's form without its substance.
Domain Specificity:
- Medical (High Risk): The effect is pronounced in MedQA, MedMCQA, and MedBullets5, where short answers under-constrain rich clinical rationales.
- Non-Medical (Low/No Risk): In GSM8K and MATH, the teacher traces were already noisier than the base, and the split did not manifest as a degradation. In ARC-Challenge (science), where answers tightly constrain the rationale, no significant degradation was observed.
- Weak Base: Mistral-7B, starting from a lower accuracy baseline, actually improved both accuracy and step error, indicating the risk regime requires a capable student capable of imitating expert form.

3. Mechanism of Degradation

Source of Accuracy Gain: The accuracy improvement primarily comes from shifting probability mass on "near-boundary" questions (where the correct answer was the base model's second choice) rather than solving previously impossible questions.
Role-Wise Impact: Degradation is most severe in hypothesis generation and differential diagnosis steps (error rate jumped from 31.1% to 61.8%), the very steps where clinical judgment is most critical.
Hidden Signals: Standard signals like aggregate hedging rates and internal "truth" probes failed to detect the shift. The model continues to use hedging words appropriately, but the baseline reliability of the claims they modify has dropped.
Not Answer-First: The paper rules out the hypothesis that the model simply picks an answer and then fabricates a justification; pre-CoT answer probabilities actually decreased after distillation.

Significance and Claims

The paper claims to identify a specific failure mode in medical CoT distillation where answer-level metrics and trace-level factuality move in opposite directions.

Process-Level Diagnosis: The authors provide evidence that standard distillation recipes can teach a student to "look like" an expert (producing a correct final label and a fluent, structured trace) while failing to ground the local factual claims within that trace.
Limitations of Current Evaluation: The study demonstrates that answer-level metrics (accuracy, calibration) and aggregate hedging rates are insufficient for evaluating distilled reasoning models in high-stakes domains like medicine.
Training Data Risk: The authors highlight a critical implication for recursive distillation: if distilled traces are used as training data for future models, the "noise" in the reasoning steps may be propagated, even if the final answers are correct.
Boundary Conditions: The risk is conditional. It arises when a compact answer weakly constrains a rich rationale, and the student is capable enough to mimic the teacher's discourse patterns without possessing the domain knowledge to verify every local claim.

The paper concludes that for medical reasoning models, a small process audit alongside answer metrics is necessary to ensure that the "written process" remains a reliable object for reading, oversight, and reuse.