Technical Summary: Diagnosing Harmful Continuation in Answer-Correct Long-CoT Training Traces

Problem Statement

Long chain-of-thought (CoT) traces are widely utilized as supervision for reasoning-oriented Large Language Model (LLM) Supervised Fine-Tuning (SFT). While existing literature acknowledges that answer-correct traces vary in training utility due to style or structure, the internal failure modes of these traces remain under-explored. Specifically, it is unclear whether a long reasoning trace remains uniformly useful once the final answer is correct.

This paper investigates a specific phenomenon in answer-correct long-CoT data: post-conclusion continuation. This occurs when the answer is already sufficiently supported by the reasoning trace, yet the generation continues with additional reasoning steps that remain part of the supervised target. The authors hypothesize that this phase may be "low-value," characterized by persistent local uncertainty and weak hidden-state progress, potentially acting as a "harmful continuation" that degrades SFT outcomes despite preserving answer correctness.

Methodology

1. Operational Partitioning and Diagnosis

To diagnose this phenomenon without assuming inherent harmfulness a priori, the authors employ a delete-only editor (Qwen3.5-27B) to process 4,780 answer-correct long-CoT trajectories from the OpenR1-Math-220k dataset.

• Operational Groups: The editor marks sentences that can be removed while preserving the reasoning necessary to recover the final answer. This partitions each trace into:
  • Retained Reasoning: The prefix supporting the final answer.
  • Editor-Removed Continuation: The post-conclusion suffix marked as removable.
• Uncertainty Diagnostics: The authors analyze answer-level entropy and Negative Log-Likelihood (NLL). They observe that while retained reasoning reduces answer NLL, the editor-removed continuation often increases answer entropy and NLL, indicating a saturation of answer support followed by unstable exploration.
• Geometric Diagnostics: Using hidden-state analysis (hidden displacement and forward progress toward the terminal state), the authors find that the editor-removed continuation exhibits weaker state movement and lower terminal-directional progress compared to retained reasoning.
• Uncertainty–Geometry Mismatch: The core diagnostic finding is a mismatch where the post-conclusion suffix remains locally costly (high uncertainty) while showing weak geometric progress.

2. Harmful Continuation Cut (HCC)

To operationalize this diagnosis without relying on a large offline editor during training, the authors propose Harmful Continuation Cut (HCC), a lightweight boundary proxy.

• Architecture: HCC uses a frozen Qwen2.5-0.5B-Instruct backbone with a cut head.
• Mechanism: It extracts sentence-level hidden states and jointly estimates:
  • A local uncertainty signal (based on entropy/NLL statistics).
  • An operational progress signal (based on hidden-state movement).
• Training Objective: The model is trained to predict the editor-identified boundary using a combination of:
  • Sequential latent regularization (variational autoencoder style).
  • Uncertainty and geometry regression losses.
  • Boundary prediction and sentence-level deletion classification losses.
• Goal: HCC approximates the editor-identified post-conclusion continuation boundary to remove it during SFT data preparation.

Key Results

1. Interventional Evidence of Harmful Continuation

The authors conduct SFT experiments on LLaMA3.2-3B and Qwen2.5-Math-7B backbones using three data processing methods: Vanilla (original), Heuristic truncation, and Editor-processed (ground truth for removal).

• Performance Gain: Removing the editor-identified post-conclusion continuation consistently improves downstream SFT performance on MATH500, AMC23, and GSM8K benchmarks compared to training on original traces.
• HCC Efficacy: HCC achieves performance comparable to the 27B editor-processed reference, and in some cases (e.g., LLaMA3.2-3B on MATH500), surpasses it. This demonstrates that a lightweight proxy can effectively recover the benefits of large-model trace editing.
• Length vs. Quality: HCC-processed traces are significantly shorter than Vanilla traces but outperform simple heuristic truncation, suggesting that the gains are due to the quality of the removed content (harmful continuation) rather than mere length reduction. Random cut baselines confirm that arbitrary shortening degrades performance.

2. Post-Training Diagnostics

Models trained on HCC-processed traces exhibit reasoning dynamics consistent with the removal of harmful continuation:

• Stability: They maintain stable answer NLL and decreasing segment entropy throughout the generation, avoiding the "entropy rebound" seen in Vanilla models.
• Geometric Progress: They show stronger token-normalized hidden displacement and reduced entropy-progress mismatch compared to Vanilla models.

3. Generalization and RL Initialization

• Cross-Domain: HCC-based SFT improves performance on non-mathematical MMLU subjects (e.g., biology, physics), suggesting the benefits of removing harmful continuation may transfer to knowledge-intensive tasks, though the authors note this does not prove the pattern exists identically in non-math traces.
• Reinforcement Learning: Models initialized with HCC-based SFT serve as stronger starting points for subsequent GRPO (Reinforcement Learning) training, consistently outperforming Vanilla-initialized models across RL steps.

Significance and Claims

The paper claims to provide the first diagnostic view of post-conclusion continuation within answer-correct long-CoT traces. Its primary contributions are:

1. Formulation: Defining a low-value phase in reasoning traces where the answer is supported but generation continues with high uncertainty and weak progress.
2. Empirical Validation: Demonstrating through answer-preserving removal that this continuation is empirically "harmful" to SFT outcomes in the studied settings.
3. Characterization: Identifying an uncertainty–geometry mismatch as the signature of this harmful phase.
4. Practical Tool: Introducing HCC, a computationally efficient (54x cheaper than the 27B editor) proxy that approximates the removal of harmful continuation, enabling scalable data cleaning for reasoning models.

The authors maintain a modest stance, clarifying that the "harmful" label is empirically supported by SFT outcomes and that the editor's labels serve as an operational intervention rather than a ground-truth oracle of intrinsic harmfulness. The work focuses on the specific setting of answer-correct long-CoT SFT, leaving finer component-level attribution and broader applicability as future work.