🤖 machine learning

SuperThoughts: Reasoning Tokens in Superposition

The paper introduces SuperThoughts, a method that compresses consecutive Chain-of-Thought tokens into single latent representations decoded via a Multi-Token Prediction module to achieve a 20–30% reduction in reasoning length and doubled inference throughput while maintaining competitive accuracy on complex math and reasoning benchmarks.

Original authors: Zheyang Xiong, Shivam Garg, Max Yu, Vaishnavi Shrivastava, Haoyu Zhao, Anastasios Kyrillidis, Dimitris Papailiopoulos

Published 2026-06-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zheyang Xiong, Shivam Garg, Max Yu, Vaishnavi Shrivastava, Haoyu Zhao, Anastasios Kyrillidis, Dimitris Papailiopoulos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a very difficult math problem. Usually, a smart computer (called a Large Language Model) solves this by talking to itself step-by-step, like a human writing out a long list of thoughts before giving the final answer. This is called "Chain-of-Thought" (CoT).

Think of each step in this thought process as a single word the computer writes down. To solve a hard problem, it might need to write down 500 words. For every single word it writes, the computer has to stop, think, and run a massive calculation. This is accurate, but it's slow and uses a lot of energy, like driving a car where you have to stop and restart the engine for every single meter you travel.

The Problem: Too Many Stops
The researchers behind this paper asked: "Why do we have to stop and restart for every single word? Can't the computer think about two words at once?"

Previous attempts to make the computer think in "invisible" thoughts (instead of words) failed because the computer got confused and lost its way without clear feedback. It's like trying to learn a new language by only thinking in your head without ever speaking; you might get lost because you don't know if your thoughts are correct until you say them out loud.

The Solution: SuperThoughts
The paper introduces a new method called SuperThoughts. Here is how it works, using a simple analogy:

Imagine you are a chef preparing a meal.

The Old Way (Standard): You chop one vegetable, cook it, taste it, then chop the next one, cook it, taste it. You do this one by one. It takes a long time.
The SuperThoughts Way: You chop two vegetables at the same time and put them in a special "super-bowl" (this is the Compressor). You then cook this super-bowl. When you take it out, you don't just get one cooked vegetable; you get two perfectly cooked vegetables ready to serve (this is the Main Module and the MTP Module working together).

By bundling two thoughts into one "super-thought," the computer only needs to do half as many cooking steps (forward passes) to get the same result. This theoretically doubles the speed.

The Safety Net: The "Confidence Check"
There is a catch. Sometimes, bundling two thoughts together is too hard. If the computer is trying to solve a really tricky part of the problem, it might get confused if it tries to do two steps at once.

To fix this, the researchers added a Confidence Check.

Think of it like a driver checking the road. If the road is clear and easy (an "easy" thought), the driver speeds up and takes two steps at once.
But if the road gets foggy or dangerous (a "hard" thought), the driver slows down and goes back to the old way: one step at a time.

This happens automatically. If the computer isn't 100% sure it can predict the next two words correctly, it says, "Never mind, let's just do one word now," and then tries again for the next one. This ensures the computer doesn't make mistakes just to be fast.

What They Found
The researchers tested this on several difficult math and science puzzles (like the MATH500 and OlympiadBench). Here is what happened:

Speed: The computer needed to write about 20% to 30% fewer steps to solve the problems. It was like cutting a long road trip in half by taking a shortcut.
Accuracy: Surprisingly, the computer didn't get much dumber. Its accuracy only dropped by a tiny bit (about 1 or 2 points out of 100) compared to the slow, careful method.
Size Matters: Bigger computers (with more "brain power") handled the "two-at-a-time" trick better than smaller ones.

In Summary
SuperThoughts is a way to make AI think faster without losing its smarts. Instead of taking one step at a time, it tries to take two steps at once. If it feels unsure, it automatically slows down to take one step to make sure it's right. This saves time and computing power while keeping the answers mostly correct.

Technical Summary: SuperThoughts: Reasoning Tokens in Superposition

Problem Statement

Large Language Models (LLMs) solve complex problems by generating explicit Chain-of-Thought (CoT) sequences, where each token represents a unit of computation (one forward pass). While recent models like OpenAI o1 and DeepSeek-R1 demonstrate that increased test-time compute significantly improves performance, this sequential token generation is computationally expensive.

Existing approaches attempting to bypass discrete token generation by reasoning in continuous latent spaces (e.g., COCONUT, latent reasoning) face a critical challenge: the lack of intermediate supervision. Standard CoT training relies on token-level cross-entropy loss at every step, providing dense gradient signals. When reasoning occurs in an unconstrained latent space, this supervision vanishes, leading to training instability, representational drift, and poor performance on long-horizon tasks. Conversely, methods that simply compress context or reduce discrete token counts via fine-tuning do not reduce the number of forward passes required, limiting their impact on inference throughput.

The central question addressed by this work is: Can a model be trained to reason in a richer, superposed latent space while retaining the intermediate supervision signals of discrete token generation?

Methodology

The authors propose SuperThoughts, a framework that compresses pairs of consecutive CoT tokens into single latent representations, allowing the model to decode two tokens per step. This approach halves the number of forward passes required for a reasoning chain of length $L$ (reducing it to $L/2$ steps) while preserving discrete token supervision.

Architecture

The model consists of three primary components:

Compressor: Encodes a pair of token embeddings $(c_{2i-1}, c_{2i})$ $(c_{2 i - 1}, c_{2 i})$ into a single latent vector $x_i$ $x_{i}$ . The authors explore two implementations:
- Linear Projection: Concatenates embeddings and projects them via a learnable matrix.
- Transformer Block: Processes the pair through a small Transformer layer, extracting the hidden state of the second token.
Main Module: The primary reasoning backbone (initialized from a pre-trained LLM). It processes the compressed vector $x_i$ to produce a hidden state $h_i$ and predicts the next odd-indexed token ( $c_{2i+1}$ ).
Multi-Token Prediction (MTP) Module: A lightweight module that predicts the subsequent even-indexed token ( $c_{2i+2}$ ). It takes as input the previous even token, the newly predicted odd token, and the Main module's hidden state, projecting them into a vector $x'_i$ to generate the next prediction. Both modules share the same output language modeling head.

Training Strategy

To address the instability of latent reasoning, a two-stage training protocol is employed:

Latent Distillation (Stage 1): The Compressor is trained to align the compressed latent space with the pre-trained language manifold. A "teacher" model processes the discrete token sequence, while a "student" model processes the compressed pairs. The Compressor is optimized to minimize the Smooth-L1 distance between the student's hidden states (after processing $x_i$ ) and the teacher's hidden states (after processing the corresponding token pair). This ensures the compressed representations preserve the reasoning state of the original discrete sequence.
Joint End-to-End Training (Stage 2): The Main module, MTP module, and Compressor are trained jointly using standard cross-entropy loss for all predicted tokens (both Main and MTP outputs).

Adaptive Inference

Recognizing that not all reasoning steps are equally difficult, the authors introduce a confidence-based adaptive mechanism. At each step, if the MTP module's prediction confidence (maximum probability) falls below a threshold $\tau$ , the model rejects the MTP prediction. It then falls back to standard decoding for that step, feeding the single token directly to the Main module. This allows the model to allocate more compute to "hard" tokens while maintaining efficiency on "easy" ones.

Key Contributions

SuperThoughts Architecture: A novel framework that fuses token pairs into superposed embeddings and decodes two tokens per step using a Main module and an MTP module, effectively halving the required forward passes.
Two-Stage Training Protocol: A method to stabilize latent reasoning by first aligning the compressed space via distillation before joint end-to-end training with discrete supervision.
Adaptive Inference Mechanism: A dynamic fallback system that trades throughput for accuracy on uncertain reasoning steps, preventing performance degradation on complex tasks.
Empirical Evaluation: Comprehensive evaluation on mathematical and scientific reasoning benchmarks (MATH500, AMC23, OlympiadBench, GPQA-Diamond) demonstrating significant CoT length reduction with minimal accuracy loss.

Results

The authors evaluated SuperThoughts on Qwen2.5-Math models (1.5B, 7B, and 14B variants) across multiple benchmarks:

Length Reduction: With the adaptive mechanism, SuperThoughts achieves a 20–35% reduction in CoT length.
Accuracy: The method maintains accuracy within 1–2 points of the standard discrete CoT baseline on most tasks. For instance, on the 1.5B model with $\tau=0.999$ , MATH500 accuracy reached 73.0% (vs. 72.4% baseline) with a 36% reduction in CoT length.
Scale Effects: Larger models (7B) tolerate aggressive compression better than smaller ones (1.5B). Without adaptive inference, the 1.5B model suffered significant accuracy drops (14–21 points), whereas the 7B model dropped only 5–12 points.
Comparison with HAMburger: SuperThoughts outperformed the HAMburger baseline (Liu & Zhang, 2025) at every compression level, achieving higher accuracy for the same CoT length or shorter CoT lengths for the same accuracy.
Inference Speedup: While theoretical speedup is proportional to length reduction, actual wall-clock time speedup is slightly lower due to overhead from the Compressor and MTP modules. However, this overhead diminishes as model size increases; for the 14B model, a ~33% length reduction translated to a ~28% reduction in generation time.

Significance

The paper claims that SuperThoughts successfully demonstrates that models can reason in a superposed latent space without sacrificing the stability provided by discrete token supervision. By compressing token pairs and utilizing adaptive inference, the framework offers a practical path to reducing the computational cost of long-horizon reasoning tasks. The work suggests that larger models possess sufficient per-step capacity to handle superposed tokens reliably, hinting that future scaling could allow for even greater compression (e.g., superposing three or more tokens). Furthermore, the authors note that as native MTP modules become standard in future LLMs (e.g., Qwen3-Next, MiMo), the application of SuperThoughts will become even more straightforward and effective.