💬 NLP

The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Models

This paper investigates whether language models utilize superposition to maintain multiple reasoning paths in latent chain-of-thoughts, finding that such capability only emerges in models trained from scratch, while training-free and fine-tuned regimes fail to leverage superposition due to pretraining biases toward token commitment and capacity constraints.

Original authors: Michael Rizvi-Martel, Guillaume Rabusseau, Marius Mosbach

Published 2026-04-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Michael Rizvi-Martel, Guillaume Rabusseau, Marius Mosbach

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex puzzle, like a difficult math problem or a logic riddle.

The Old Way (Discrete Thinking):
Most AI models today think like a person talking out loud. They say, "Step 1 is this," then "Step 2 is that." They have to pick one specific word to say at a time. If they are unsure, they have to guess which word to say next. This is called Chain-of-Thought (CoT). It's like walking through a maze by picking one path at a time. If you pick the wrong path, you have to go back and start over.

The New Idea (Latent Thinking & Superposition):
Researchers recently proposed a cooler idea: Latent CoT. Instead of picking just one word, imagine the AI could hold all possible paths in its mind at the same time, like a superposition of many different thoughts. It's as if the AI could walk down every path in the maze simultaneously, see which one leads to the exit, and then decide which path to actually take. This is called Superposition.

The big question this paper asks is: "Do AI models actually do this? Or is it just an illusion?"

The Experiment: Three Different Scenarios

The authors tested this idea in three different ways, like testing a new car engine under different conditions:

The "Off-the-Shelf" Test (Training-Free): They took a giant, pre-trained AI (like a finished car) and tried to force it to think in "superposition" just by mixing its internal math.
- The Result: It didn't work. The AI immediately collapsed all those mixed thoughts back into a single, specific word. It was like trying to hold a cloud in your hand; the moment you touched it, it turned into a single drop of water. The AI just picked the most likely word and ignored the rest.
The "Fine-Tuned" Test: They took a standard AI and taught it (fine-tuned it) to use these "mixed thoughts" to solve logic puzzles.
- The Result: It didn't work either. The AI learned a "cheat code." Instead of actually exploring multiple paths, it figured out how to look at the question and jump straight to the answer, skipping the thinking process entirely. It was like a student who memorized the answer key instead of learning how to solve the math problems.
The "From-Scratch" Test: They built a tiny AI from the ground up, training it only to use these "mixed thoughts" from day one.
- The Result: Success! This tiny AI actually did use superposition. It kept multiple possibilities alive in its mind while it worked through the problem. It didn't cheat; it actually explored the maze.

Why Did It Fail for the Big Models?

The paper explains that the big, pre-trained models failed for two main reasons:

The "Commitment" Habit: Big models are trained on billions of sentences to predict the next word. They get really good at saying, "Okay, I'm 99% sure the next word is 'cat'." When you give them a "mixed" thought, their brain is so trained to pick a single word that it forces the mixed thought to collapse into one word immediately. It's like a person who is so used to making quick decisions that they can't handle a moment of hesitation.
The "Shortcut" Trap: When you teach a smart, big model to use a new trick, it often looks for the easiest way to get a good grade. If it can solve the problem by ignoring the "thinking" part and just guessing the answer, it will do that. It's like a student who realizes they can pass the test by guessing the first letter of every answer, rather than actually studying.

The Takeaway

The paper concludes that superposition is real, but it's fragile.

Current AI: The big models we use today (like the ones powering chatbots) are too "hard-wired" to pick single words. They can't easily hold multiple thoughts at once. They either collapse the thoughts or cheat by finding shortcuts.
The Future: To get AI that can truly "think in parallel" (holding many ideas at once), we probably need to train small models from scratch specifically for this, rather than trying to force our current giant models to do it.

In a nutshell: The idea of AI holding multiple thoughts at once is scientifically sound, but our current "adult" AI models are too stubborn and too good at finding shortcuts to actually use it. We need to raise "baby" AI models specifically trained to keep their options open before they learn to speak in single words.

1. Problem Statement

Recent advancements in Large Language Models (LLMs) have explored Latent Chain-of-Thought (Latent CoT) reasoning, where models reason in a continuous vector space rather than discrete token space. A primary theoretical motivation for this approach is superposition: the hypothesis that models can maintain multiple candidate solutions or reasoning paths simultaneously within a single latent representation, thereby exploring the solution space more efficiently than discrete CoT (which must commit to one token at a time).

Despite theoretical arguments suggesting superposition is possible, there is a lack of empirical evidence confirming whether LLMs actually utilize this capability. The paper addresses the core research question: Does superposition actually occur in latent CoT models, or is it an illusion?

2. Methodology

The authors investigate this question across three distinct training regimes, utilizing Logit Lens (to project intermediate hidden states back to vocabulary space) and entity-level probing (to track belief evolution over reasoning steps).

Experimental Regimes:

Training-Free (Soft Thinking):
- Method: Uses "Soft Thinking" (Zhang et al., 2025), where latent thoughts are constructed as convex combinations (weighted sums) of token embeddings based on the model's logit distribution.
- Models: QwQ-32B (reasoning model) and Qwen2-1.5B (base model).
- Task: Math reasoning benchmarks (MATH500, AIME2024, GSM8K).
- Analysis: Compares entropy profiles and KL divergence between soft tokens and discrete tokens.
Fine-Tuned (Coconut):
- Method: Uses "Coconut" (Hao et al., 2024), where a pre-trained model is fine-tuned to output continuous latent thoughts via a staged curriculum (progressively replacing discrete tokens with latent ones).
- Model: GPT-2 (124M).
- Task: ProsQA (a synthetic graph-traversal task requiring multi-hop logical inference).
- Analysis: Probes internal representations to see if the model performs step-by-step reasoning or shortcuts.
From-Scratch:
- Method: A small GPT-2 style model (8 heads, 768-dim) trained entirely from random initialization using latent thoughts to solve the ProsQA task.
- Analysis: Compares belief evolution and performance with and without latent tokens.

3. Key Contributions & Results

A. Training-Free Regime: Superposition Collapses Early

Entropy Profiles: When analyzing Soft Thinking models, the Shannon entropy of the logit distributions across layers is nearly identical to discrete CoT. Both collapse to near-zero entropy at the final layers.
Intervention Experiments: Replacing a "soft" token with its discrete argmax counterpart results in negligible differences (KL divergence $\approx 10^{-4}$ , Cosine Similarity $> 0.99$ ).
Conclusion: Off-the-shelf LLMs do not maintain superposition. They process soft inputs by immediately collapsing them into a single discrete interpretation within the first few layers. The "superposition" is effectively lost before deep reasoning occurs.

B. Fine-Tuned Regime: Shortcut Learning

Latent Tokens are Unnecessary: The fine-tuned Coconut model achieves 96.6% accuracy even when fed only the question (no latent tokens), suggesting the latent reasoning steps contribute minimally to performance.
Probing Reveals Shortcuts: Entity-level probing shows that the fine-tuned model commits to the final target entity immediately at Step 0, rather than evolving beliefs through intermediate "correct next" entities.
Conclusion: Fine-tuned models do not learn to reason in superposition. Instead, they learn a shortcut: they infer the answer directly from the question embedding and simply copy it through the latent tokens.

C. From-Scratch Regime: Superposition Emerges

Genuine Superposition: Models trained from scratch do exhibit signs of superposition. Probing reveals that intermediate latent states maintain probability mass over both the "correct next" entity and "wrong neighbors," showing a genuine evolution of belief.
Necessity of Latents: Removing latent tokens from from-scratch models causes a catastrophic drop in performance (e.g., from 94.5% to 13.8%), proving the latent thoughts are doing the actual reasoning work.
Capacity Constraint: Superposition in from-scratch models is brittle. It only emerges in smaller models (2–4 layers). Larger models (8–12 layers) trained from scratch revert to learning shortcuts, ignoring the latent reasoning steps.

D. Root Causes of Failure

The paper identifies two complementary phenomena explaining why superposition fails in pre-trained/fine-tuned models:

Pretraining Bias: Autoregressive pretraining on next-token prediction forces models to commit to a single token identity in the final layers. When presented with a superposed input, the model projects it onto the nearest discrete token, collapsing the superposition.
Capacity Effects: Larger models have sufficient capacity to find "shortcut" solutions (mapping input directly to output) without needing to perform complex parallel exploration. Superposition is only favored when model capacity is constrained (smaller models) or when the model is trained from scratch without the bias of next-token prediction.

4. Significance and Implications

Debunking the "Superposition" Myth: The paper provides strong empirical evidence that current state-of-the-art latent reasoning methods (Soft Thinking, Coconut) do not leverage superposition as a mechanism for parallel reasoning. The perceived benefits often stem from shortcut learning or echo-chamber effects.
Design Guidelines for Future Models:
- Training Strategy: To achieve true superposition, models likely need to be trained from scratch on the specific reasoning task, rather than fine-tuned on pre-trained LLMs.
- Model Size: There is a "Goldilocks" zone for capacity; models that are too large tend to learn shortcuts, while smaller models are forced to utilize the latent space for reasoning.
- Re-evaluating Latent Reasoning: The authors suggest that the value of latent reasoning may not be superposition, but flexibility—the ability to represent intermediate computations that do not correspond to natural language tokens, avoiding the discretization bottleneck.
Interpretability: The work highlights the necessity of using mechanistic interpretability tools (Logit Lens, probing) to validate theoretical claims about model internals, as performance metrics alone can be misleading (e.g., high accuracy achieved via shortcuts).

In summary, the paper argues that superposition is not a default emergent property of latent CoT in current LLMs. It is a fragile capability that requires specific training conditions (from-scratch, constrained capacity) and is easily suppressed by the inductive biases of standard pretraining objectives.