💬 NLP

Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models

This paper demonstrates that distilling reasoning capabilities in large language models by conditioning chain-of-thought generation on the gold answer significantly degrades verifiable reasoning performance, as the resulting data encourages backward rationalization rather than genuine derivation, a flaw that standard correctness filtering fails to detect.

Original authors: Jungseob Lee, Seungyoon Lee, Suhyune Son, Dongyub Jude Lee, Sungbin Han, Sugyeong Eo, Heuiseok Lim

Published 2026-07-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jungseob Lee, Seungyoon Lee, Suhyune Son, Dongyub Jude Lee, Sungbin Han, Sugyeong Eo, Heuiseok Lim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to solve a tricky puzzle. In the world of artificial intelligence, these robots are called Large Language Models (LLMs), and they are incredibly good at talking, writing, and even solving math problems. But to get really smart at reasoning, they need to practice their "thinking process." This process is often called a "Chain of Thought," which is just a fancy way of saying the robot writes down its step-by-step logic before giving the final answer, kind of like showing your work on a math test.

Usually, when scientists train these robots, they let the robot try to solve a problem on its own. If the robot gets the right answer, they save that thinking process as a good example to study later. If it gets it wrong, they toss the example. But sometimes, the robot struggles and can't find the answer. So, a common trick is to show the robot the correct answer first and say, "Hey, here is the answer; now write a story about how you figured it out." It seems like a smart shortcut: you get a correct answer and a matching story, so you can use it for training, right?

This paper investigates a sneaky problem with that shortcut. The researchers found that when you show the robot the answer before it starts thinking, the robot learns a bad habit. Instead of actually figuring out the solution step-by-step, it starts writing a story that works backward from the answer it already knows. It's like a student who sees the answer key before taking a test and then writes an essay that justifies the answer without ever actually doing the math. The paper shows that even though the final answer is correct, the "thinking" is fake, and teaching the robot with these fake stories actually makes it worse at solving new problems on its own.

The "Cheat Sheet" Trap

The researchers set up a clever experiment to prove this. They took a very smart AI model and asked it to solve the same set of difficult math problems twice. In the first round, they hid the answer completely, forcing the model to think from scratch. In the second round, they showed the model the correct answer and asked it to write a reasoning chain that led to it. They made sure to keep only the chains that ended with the right answer in both groups, so a simple "correctness check" would pass both of them.

Then, they trained two new "student" models. One student studied only the "honest" chains where the answer was hidden. The other student studied the "cheated" chains where the answer was shown first. The results were shocking. The student who studied the honest chains got better at solving problems. But the student who studied the cheated chains actually got worse. On the hardest competition-level math problems, the cheated student lost about 27 points of accuracy compared to the honest one.

Why the "Backwards Story" Hurts

The paper explains that this happens because the model changes its behavior when it sees the answer. When the answer is hidden, the model has to search for the solution, exploring different paths like a detective looking for clues. But when the answer is visible, the model stops searching and starts "rationalizing." It writes a story that justifies the known conclusion, often stating the answer right at the very beginning of its thought process.

The researchers call this "Answer Leakage." It's like a magician who knows the trick beforehand and then tries to explain how they did it, but the explanation is just a made-up story to make the trick look real. The paper found that this bad habit is measurable even before the model is trained. If you look at the model's raw thoughts, you can see that when the answer is visible, it jumps to the conclusion much faster. This "Answer-First" habit is a warning sign: the more often a model does this, the more damage it will suffer when trained on those chains.

The Fix: Don't Show the Answer

The most important takeaway is that you cannot fix this by just checking if the final answer is correct. The paper shows that standard filters, which only look at the final result, completely miss this problem because the "cheated" chains do have the right answer. The damage is hidden inside the thinking process itself.

The solution is surprisingly simple: don't show the model the answer while it's trying to figure it out. The paper explicitly warns that even if you wait for the model to fail and then show it the answer to generate a "rescued" chain, this still comes with a real cost—a 4.0-point drop in accuracy. The best approach is to generate chains without the answer visible at all. If a pipeline must show the answer, the researchers suggest changing the instruction to "derive the answer first, then state it," rather than "explain how you got this answer." This small change in how the prompt is written stops the model from jumping to the conclusion and recovers about two-thirds of the lost accuracy.

In short, the paper proves that in the world of AI training, a correct answer doesn't always mean a correct lesson. If you teach a robot to work backward from a known truth, it forgets how to discover the truth on its own. To build smarter AI, we need to let them struggle with the unknown, rather than giving them the answer key before they even start thinking.

Technical Summary: Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation

1. Problem Statement

Standard distillation pipelines for Large Language Models (LLMs) often rely on sampling chains of thought (CoT), filtering for those that reach the correct final answer, and fine-tuning on the survivors. When sampling fails to produce a correct chain, a common remediation strategy is answer-conditioning: feeding the model the gold answer as a hint and instructing it to generate a reasoning trace that rationalizes or reaches that specific answer.

This paper challenges the implicit assumption that any chain reaching the correct answer is a valid training target, regardless of its generation process. The authors argue that answer-conditioned chains introduce a specific pathology: the model learns to rationalize backward from a known conclusion rather than derive the answer through search. Crucially, this degradation is invisible to standard correctness filters because the resulting chains are, by construction, correct.

2. Methodology: The "One-Bit Experiment"

To isolate the causal effect of answer-conditioning, the authors designed a controlled experiment where all variables are held constant except for a single binary intervention: whether the gold answer is visible to the generator during chain creation.

Setup: A single generator model ( $G$ $G$ ) is used to generate CoT chains for the same set of 935 math problems under two conditions:
1. Answer-Blind (Control): The model sees only the problem and is asked to derive the answer.
2. Answer-Leaked (Treatment): The model sees the problem and the gold answer, with an instruction to generate a chain that reaches it.
Filtering: In both arms, only chains with a correct final boxed answer are retained.
Training: Two student models are fine-tuned using an identical recipe on the filtered datasets from each arm.
Evaluation: The performance gap ( $\Delta = \text{acc}(S_{\text{blind}}) - \text{acc}(S_{\text{leaked}})$ ) is measured on verifiable reasoning benchmarks (e.g., MATH-500, AIME).

The study also includes ablation studies varying instructions (e.g., "derive first" vs. "rationalize toward"), length-matched controls, and cross-family generalization tests (training smaller or different-architecture students on leaked chains).

3. Key Contributions

1. Isolation of the Leakage Penalty

The authors demonstrate that generating chains under answer-conditioning incurs a significant accuracy penalty, even when the generator, problem set, correctness filter, and fine-tuning recipe are identical.

On MATH-500, the penalty is 16.2 points for Qwen3-8B.
On AIME (Olympiad-level problems), the penalty grows to 27.2 points.
The penalty is content-driven, not length-driven; length-matched controls show that shortening blind chains does not replicate the damage.

2. Mechanism Analysis: Off-Policy Rationalization

The paper identifies the mechanism as a shift from search to rationalization:

Answer-First Symptom: Answer-conditioned chains frequently state the gold answer within the first 20% of the reasoning block (Answer-First Rate, AFR). Blind chains rarely do this.
Off-Policy Shift: Answer-blind generation samples from $P(c|q)$ (search distribution), which is necessary for test-time inference. Answer-conditioned generation samples from $P(c|q, a)$ , collapsing the search into post-hoc justification.
Train/Test Inversion: Models trained on leaked chains generate longer chains at test time but fail to converge (termination deficit), indicating they have not learned the derivation process required when the answer is unknown.

3. Data-Centric Harm and Partial Repair

The harm is attributed to the data, not the generator's inherent capability.

Rationalized Chains Carry the Harm: Within a single leaked corpus, chains that state the answer early ("Leaked-Early") carry the majority of the penalty. A simple filter removing these chains recovers approximately two-thirds of the lost accuracy.
Instruction vs. Visibility: The primary driver of harm is the instruction to "rationalize toward" the answer, not merely the visibility of the answer. Instructions to "derive first" significantly mitigate the penalty.

4. Predictive Signature ( $\Delta$ AFR)

The authors propose a pre-training diagnostic metric: $\Delta$ AFR (the difference in Answer-First Rate between leaked and blind generations).

$\Delta$ AFR strongly correlates ( $r=0.96$ ) with the eventual leakage penalty across eight models from four different families (Qwen, DeepSeek, GLM, Nemotron).
This metric allows practitioners to screen candidate teachers before any student training occurs, predicting the penalty with a mean absolute error of 2.9 points.

4. Results and Generalization

Domain Specificity: The penalty is confined to tasks requiring multi-step derivation (Math, Code). On multiple-choice knowledge benchmarks where the model selects an option rather than deriving a solution, no penalty is observed.
Cross-Model Transfer: The harm transfers across model families. Chains generated by a Llama-Nemotron teacher degrade a Qwen3-8B student, and chains from a GLM teacher degrade a Qwen3-8B student.
Code Domain: The penalty persists in code generation (MBPP+, HumanEval+), where "correctness" is defined by test execution. Leaked code chains that pass tests still degrade student performance by ~9–10 points.
Cross-Student Transfer: The damage follows the data. Training a smaller model (Qwen3-1.7B) on leaked chains from a larger teacher (Qwen3-8B) results in a ~7 point penalty compared to blind training.

5. Significance and Practical Takeaways

The paper concludes that the standard practice of "fixing" failed sampling by conditioning on the gold answer is counterproductive for verifiable reasoning. The "correctness filter" is insufficient because it cannot distinguish between a chain derived through search and one rationalized backward from a known target.

Practical Implications:

Generate Answer-Blind: The most effective safeguard is to generate chains without showing the gold answer. If sampling fails, the data should be discarded rather than regenerated with the answer visible.
Instruction Abstraction: If answer visibility is unavoidable, the instruction must be modified to "derive first, then state the answer," which recovers roughly two-thirds of the lost accuracy.
Pre-Training Screening: Practitioners can measure $\Delta$ AFR on a small, unlabeled sample of a candidate teacher's generations to predict whether answer-conditioned distillation will harm the student.

The authors emphasize that these findings apply specifically to verifiable-reasoning distillation (math, code) where a final answer check is possible, and that the harm is a property of the data generation recipe rather than the model architecture.