💬 NLP

Training Continuous Chain of Thought Models: A Tale of Two Regimes

This paper introduces C-MTP, a faster direct supervision method for training continuous Chain-of-Thought models that outperforms prior direct approaches and matches slower indirect methods on short reasoning traces, while revealing that current continuous CoT techniques struggle significantly when applied to complex tasks with longer reasoning chains.

Original authors: Varun Yerram, He He, Eunsol Choi

Published 2026-07-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Varun Yerram, He He, Eunsol Choi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to solve a tricky puzzle. In the past, the robot would just blurt out the answer, often getting it wrong because it didn't think things through. Then, scientists discovered a magic trick: if they told the robot to "think out loud" first, writing down every little step of its reasoning, the robot became much better at solving problems. This is called "Chain-of-Thought." It's like a student showing their work on a math test; the more steps they write down, the more likely they are to get the right answer.

However, there's a catch. Writing out every single step takes a lot of time and computer power. It's like asking a runner to stop and write a diary entry after every single step they take. It slows them down to a crawl. So, researchers started asking: "Can we make the robot think in a secret, compressed language? Can it do all that thinking in its head using short, dense notes instead of long sentences?" This idea is called "Continuous Chain-of-Thought." The goal is to keep the smart thinking but lose the slow, wordy talking. But here is the big question: Can we actually teach a robot to think in these secret notes without losing its smarts?

This paper, titled "Training Continuous Chain of Thought Models: A Tale of Two Regimes," dives deep into that question. The authors, Varun Yerram, He He, and Eunsol Choi, act like detectives testing two different ways to teach the robot this secret language. They call these two methods "Direct Supervision" and "Indirect Supervision."

Think of Indirect Supervision like a teacher who watches a student solve a problem step-by-step on a whiteboard. The teacher uses a special process where the robot autoregressively predicts intermediate thoughts (latents) one by one. To guide this slow, serial prediction, the teacher model (which has already solved the problem using full text) shares its internal "brain states" (activations) with the student robot. The robot learns to mimic these internal states to generate its own sequence of thoughts, one after another, like a line of dominoes falling. It's accurate but slow.

Direct Supervision, which the authors introduce as a new method called C-MTP, is more like a teacher handing the student a stack of index cards. Each card represents a chunk of the thinking process. The teacher says, "Here is the answer for this whole chunk of thinking; just learn to predict the words that go on this card." This allows the robot to learn many steps at the same time, like reading a whole page of a book instead of one letter at a time. It's much faster and more efficient.

The researchers tested these two methods on three different levels of difficulty, like climbing a mountain with three camps.

Camp 1: The Easy, Structured Path.
On simple math problems where the thinking is short and tidy (like "18 times 12 equals 216"), the new Direct Supervision (C-MTP) method was a superstar. It was faster to train, needed fewer examples to learn, and actually got better at solving problems it had never seen before compared to the old, slow Indirect method. It was like a sprinter who could run just as fast as the marathon runner but with much less training.

Camp 2: The Verbose, Chatty Path.
When the problems got a bit more complex and the thinking required long, chatty explanations (like writing full sentences for every step), the tables turned. The Indirect Supervision method took the lead. Why? Because it was designed to compress long, messy thoughts into a few short notes. The Direct method struggled here because it tried to predict every single word of the long explanation, and if it made one small mistake, the whole chain of thought fell apart. It was like trying to copy a long story by guessing every word; one typo ruins the whole paragraph.

Camp 3: The Realistic, Chaotic Summit.
Finally, the researchers tested both methods on the hardest, most realistic problems—complex math puzzles with hundreds of words of reasoning, just like a human would write. This is where the story takes a sad turn. Both methods, the fast Direct one and the careful Indirect one, crashed and burned. Their performance dropped by about 65% compared to the standard method of just letting the robot write out all its thoughts.

The authors found that on these realistic, long tasks, neither method could handle the complexity. The Direct method kept making small math errors that piled up, and the Indirect method couldn't compress the long, messy thoughts accurately enough. It turns out that the current "secret language" of Continuous Chain-of-Thought isn't quite ready for the real world yet. The paper suggests that the easy benchmarks we use to test these robots might be lying to us, making them look smarter than they really are when faced with real, messy problems.

In the end, the paper doesn't declare a single winner. Instead, it tells us that while the new Direct method is a fantastic, speedy tool for simple, structured tasks, we still have a long way to go before robots can think in secret codes for complex, real-world problems. The dream of a robot that thinks fast and smart without writing everything down is still a work in progress.

Problem Statement
Large Language Models (LLMs) achieve high performance on complex reasoning tasks by generating lengthy Chain-of-Thought (CoT) traces. However, autoregressively generating these verbose token sequences incurs significant computational costs during inference. Continuous Chain-of-Thought (ContinuousCoT) methods aim to mitigate this by replacing discrete reasoning tokens with a short sequence of dense latent representations. Existing approaches generally fall into two categories: indirect supervision, which trains models to autoregressively predict latents that are later decoded or distilled (e.g., COCONUT, CODI), and direct supervision, which attempts to map reasoning traces directly to latent spaces (e.g., CoLaR).

Indirect methods suffer from slow, sequential training due to autoregressive latent generation and often rely on complex distillation pipelines. Direct methods, while enabling parallel training, have historically struggled with optimization challenges, such as modeling complex distributions over latent spaces, and have shown performance degradation on verbose or realistic reasoning traces compared to standard CoT fine-tuning.

Methodology: C-MTP
The authors introduce C-MTP (Continuous Chain-of-Thought via Multi-Token Prediction), a novel direct supervision framework designed to simulate continuous latent reasoning more efficiently.

Core Concept: Instead of modeling latents as parameterized Gaussian distributions (as in CoLaR) or predicting them autoregressively (as in indirect methods), C-MTP treats an intermediate latent as an aggregated token embedding of a span of tokens from the original CoT trace.
Training Objective: The model is trained using Multi-Token Prediction (MTP). Given a reasoning trace $y$ , it is partitioned into non-overlapping spans of $s$ tokens. Each span is compressed into a single latent vector $z_i$ by averaging the token embeddings. The model is then trained to predict the $s$ tokens composing each latent span in parallel, conditioned on the input prompt and previous latents.
Loss Function: The objective combines a multi-token prediction loss (predicting the tokens within each span) and a standard answer generation loss. The authors also employ a CoT teacher for knowledge distillation ( $L_{KD}$ ) to guide training, though ablation studies show C-MTP remains effective without it.
Inference: During the reasoning phase, the model samples $s$ tokens at each step, aggregates their embeddings to form the next latent, and repeats until an end-of-thinking token is generated. The model then switches to standard autoregressive generation for the final answer.

Key Contributions

C-MTP Framework: A new direct supervision method that uses MTP to align continuous latent reasoning with standard pretraining objectives, enabling fully parallel training via teacher forcing.
Systematic Regime Comparison: A comprehensive empirical evaluation across three distinct CoT trace regimes:
- Structured: Compact mathematical expressions (e.g., GSM8k-Aug).
- Semi-Natural: Verbose natural language explanations (e.g., GSM8k-Aug-NL).
- Realistic: Long, complex traces generated by strong LLMs (e.g., MATH dataset).
Identification of Limitations: The paper demonstrates that while C-MTP excels in structured settings, both direct and indirect supervision methods fail to match standard CoT fine-tuning (CoT-SFT) on realistic, long-horizon reasoning tasks.

Experimental Results

Structured Traces (GSM8k-Aug): C-MTP significantly outperforms existing direct supervision methods (CoLaR) across all compression ratios. It matches or exceeds indirect supervision methods (CODI) in accuracy while being faster (approx. 2x training speedup) and more sample-efficient. C-MTP also demonstrates superior generalization to out-of-distribution problem structures compared to the sequential training of indirect methods.
Semi-Natural Traces (GSM8k-Aug-NL): On verbose traces, indirect supervision (CODI) outperforms direct methods. The authors attribute this to the "compression" nature of indirect methods, which forces the model to discard filler tokens, whereas direct methods struggle with error accumulation when forced to predict full verbose traces via teacher forcing.
Realistic Traces (MATH): On complex tasks with hundreds of reasoning tokens, both direct and indirect supervision regimes collapse, achieving only ~30–35% of the accuracy of standard CoT-SFT. Analysis reveals that errors in early reasoning steps (particularly in arithmetic expressions) cascade irrecoverably, leading to incorrect final answers.

Significance and Claims
The paper claims that current benchmarks relying on structured or semi-natural traces overstate the practical utility of ContinuousCoT methods. While C-MTP offers a more efficient and effective solution for structured reasoning tasks, the fundamental limitations of compressing long, complex reasoning traces into fixed latent budgets remain unsolved. The authors suggest that future progress may require hybrid approaches that adaptively switch between token-based and continuous reasoning, rather than relying solely on one supervision regime. The work underscores the need for evaluation on realistic, long-horizon tasks to truly assess the viability of latent reasoning.

More like this