💬 NLP

Latent Reasoning with Supervised Thinking States

The paper proposes "Thinking States," a method that improves reasoning efficiency and performance by generating and embedding latent thought tokens during the input processing stage, allowing for parallelizable training and reduced inference latency compared to traditional Chain-of-Thought.

Original authors: Ido Amos, Avi Caciularu, Mor Geva, Amir Globerson, Jonathan Herzig, Lior Shani, Idan Szpektor

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ido Amos, Avi Caciularu, Mor Geva, Amir Globerson, Jonathan Herzig, Lior Shani, Idan Szpektor

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef in a busy restaurant. You have two ways to handle a complex, multi-step recipe:

Method 1: The "Chain-of-Thought" (CoT) Way
Every time you finish a small step—like chopping an onion—you stop, grab a massive chalkboard, and write down exactly what you just did: "I have now chopped one onion. It is finely diced. I am ready for the next step." You do this for every single ingredient. By the time you get to the main course, your kitchen is cluttered with giant chalkboards, you’re running out of space, and you’re moving much slower because you’re constantly writing. This is how current AI works: it’s smart, but it’s "wordy" and slow because it has to type out its entire inner monologue.

Method 2: The "Thinking States" Way (The Paper's Method)
Instead of writing everything on a chalkboard, you use a mental notepad. As you chop the onion, you quickly summarize the "state" of your work in your head: "Onion: chopped." You don't write it down for everyone to see; you just hold that "state" in your mind and use it to guide your next move. You move faster, your kitchen stays clean, but you are still performing the same high-level reasoning.

What is this paper actually about?

The researchers have created a way for AI to "think" without having to "speak" its every thought.

Currently, if you want an AI to solve a hard math problem, you have to tell it to "show its work." This makes the AI smarter, but it also makes it incredibly slow and expensive because it generates hundreds of extra words just to process the logic.

The authors propose "Thinking States." Here is how it works in three simple steps:

The Chunking: Instead of reading a whole sentence and then thinking, the AI reads the text in small "chunks" (like bites of food).
The Mental Summary: After each chunk, a tiny, specialized part of the AI generates a quick "thought" in natural language. But—and here is the magic—it doesn't actually type those words out into the chat. Instead, it compresses those thoughts into a mathematical "summary" (a State) and injects that summary directly into its "brain" before it reads the next chunk.
The Speed Boost: Because the AI isn't wasting time printing out long paragraphs of reasoning, it can process information much faster. It’s like the difference between someone explaining their entire thought process out loud versus someone who just nods and moves to the next step.

Why is this a big deal?

It’s a "Smart Shortcut": In their tests, the AI was almost as smart as the "wordy" version (CoT) but much faster. On certain tasks, like tracking how many coins are flipped in a sequence, it was actually better at long-term memory than the wordy version.
It’s Easy to Teach: Usually, teaching an AI to "think in secret" is incredibly hard because you can't see what it's thinking to correct it. The researchers found a clever way to use existing "show your work" data to train the AI's "secret thoughts," making the training process much faster and more efficient.
It’s Efficient: It solves the "clutter" problem. The AI gets the benefits of deep reasoning without the "extra baggage" of long, unnecessary text.

The Bottom Line

This paper is teaching AI how to have an efficient inner monologue. It allows the model to keep a "running tally" of its logic in its head, making it faster, leaner, and more capable of handling long, complex problems without getting lost in its own words.

Technical Summary: Latent Reasoning with Supervised Thinking States

1. Problem Statement

Large Language Models (LLMs) achieve high accuracy on complex tasks through Chain-of-Thought (CoT) reasoning, where they generate explicit natural language rationales before providing an answer. However, CoT incurs a significant inference-time cost because the model must generate a large number of additional tokens, increasing latency and computational overhead.

Existing attempts to mitigate this include:

Continuous Latent Reasoning: Representing thoughts as continuous embeddings. While efficient, these are difficult to supervise and often require computationally expensive Backpropagation Through Time (BPTT).
Distillation/Compression: Distilling reasoning into query representations. These methods often lack the "depth" or sequential conditioning provided by CoT, as they process queries in parallel rather than through a recurrent reasoning process.

2. Methodology: The "Thinking States" Architecture

The authors propose Thinking States, a recurrent reasoning mechanism that performs reasoning while processing the input tokens, without increasing the context length.

Core Components:

Chunk-Recurrent Process: The input sequence is partitioned into $K$ non-overlapping chunks. The model processes these chunks iteratively.
Thinking Block ( $T$ ): A lightweight Transformer decoder that takes high-level representations from the backbone LLM and autoregressively generates a sequence of natural-language thought tokens ( $Z$ ).
Compression Block ( $C$ ): A Transformer encoder with a pooling layer that compresses the variable-length thought sequence into a fixed-size state ( $S$ ).
State Injection: This compressed state $S$ is added to the input embeddings of the subsequent chunk at a shallow layer (e.g., the first layer) of the backbone LLM. This allows the "thoughts" to condition all subsequent computations through the model's depth.

Training Strategy (Teacher-Forced Parallelism):
To avoid the slow and memory-intensive BPTT, the authors use Supervised Thinking States. They align existing CoT rationales to specific positions in the input text. During training, they use teacher-forcing:

They use the ground-truth reasoning steps to compute a "gold" target state.
They inject this gold state into the backbone LLM.
This allows the entire sequence to be processed in a single parallel forward pass, as all target states are known upfront.

Inference Optimization (Speculative Thinking):
To prevent sequential bottlenecks during prefill, the authors use a "speculative" algorithm. They assume most chunks require no reasoning (trivial states). They perform a parallel pass, identify only the chunks that actually generate non-trivial thoughts, and re-run only those specific segments. This significantly reduces latency.

3. Key Contributions

New Architecture: A recurrent mechanism that bridges the gap between explicit CoT and continuous latent reasoning.
Efficient Supervision: A method to train latent reasoning using natural language supervision and teacher-forcing, enabling fully parallel training.
Interpretability: Unlike purely continuous latent models, Thinking States generate natural language tokens, allowing humans to inspect the model's internal reasoning.
Length Generalization: Demonstrates that recurrent state updates allow models to handle sequences longer than those seen during training.

4. Experimental Results

The model was evaluated against CoT, iCoT, and Coconut on several benchmarks:

State Tracking (Parity & Variable Assignment): Thinking States showed superior length generalization. While CoT accuracy dropped significantly on longer sequences, Thinking States maintained near-perfect accuracy, proving it learned a true recurrent reasoning mechanism.
General Reasoning (GSM8K & 2-Hop QA):
- Accuracy: On GSM8K, Thinking States outperformed latent baselines (Coconut, iCoT) by ~8%. On 2-Hop QA, it matched or exceeded CoT performance.
- Efficiency: It achieved significant speedups (e.g., 2.66x faster than CoT on GSM8K) because it avoids generating long strings of text during inference.
Error Analysis: The authors found that Thinking States can actually be more robust than CoT in certain cases, avoiding "hallucinated" steps or overly complex computations that lead CoT astray.

5. Significance

This paper presents a major step toward efficient reasoning. It demonstrates that we do not have to choose between the accuracy of explicit CoT and the speed of latent representations. By compressing reasoning into a recurrent state that is supervised via natural language, the authors provide a scalable, interpretable, and computationally efficient framework for the next generation of reasoning-capable LLMs.