💬 NLP

Thoughtbubbles: an Unsupervised Method for Parallel Thinking in Latent Space

The paper introduces Thoughtbubbles, an unsupervised transformer variant that learns to perform parallel adaptive computation in latent space by dynamically forking residual streams during pretraining, thereby achieving superior perplexity and reasoning performance with reduced computational budgets compared to standard and existing parallel methods.

Original authors: Houjun Liu, Shikhar Murty, Christopher D. Manning, Róbert Csordás

Published 2026-02-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Houjun Liu, Shikhar Murty, Christopher D. Manning, Róbert Csordás

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a very difficult puzzle. A standard computer program (like a typical AI) tries to solve it by reading the instructions one line at a time, strictly following a single path. If the puzzle gets hard, the program just keeps reading faster or uses more memory, but it can't really "stop and think" differently for the hard parts without wasting time on the easy parts.

The paper introduces a new method called Thoughtbubbles. Here is how it works, using simple analogies:

1. The Problem: The "One-Size-Fits-All" Factory

Think of a standard AI model as a factory assembly line. Every item (every word in a sentence) moves down the line, passing through the same number of machines (layers).

The Issue: Some items are simple (like the word "the"), while others are complex (like a tricky math problem). The factory treats them exactly the same. It spends the same amount of time and energy on the easy words as it does on the hard ones. This is inefficient.

2. The Solution: The "Bubble" of Parallel Thinking

The authors propose a new way for the AI to work. Instead of a single line, imagine the AI can fork (split) its thinking process.

The Fork: When the AI encounters a difficult word or concept, it doesn't just move forward. It creates a "bubble" of clones. It splits that specific thought into multiple copies that all work on the problem at the same time, side-by-side.
The Bubble: These clones form a "bubble" of extra thinking power in the middle of the network. They are like a team of experts huddled together to solve a specific tough spot in the puzzle.
The Pruning: If a thought is simple (like "the"), the AI doesn't make a bubble; it just moves it along. If a thought is too messy or unhelpful, the AI can "delete" those clones to make room for the important ones.

3. How It Learns (The "Scorecard" System)

The coolest part is that the AI learns to do this on its own while it is being trained, without anyone telling it exactly when to split or merge.

The Score: Every thought gets a hidden "score" based on how important or difficult it is.
The Decision: If a thought has a high score (it's hard), the AI says, "Let's make more copies of this to think about it longer!" If it has a low score, it says, "We don't need to waste energy on this; let's keep just one copy."
The Result: The AI naturally learns to spend more "brain power" on the confusing parts of a sentence and less on the easy parts.

4. The "Merge"

After all these clones have done their extra thinking in the bubble, they come back together. The AI averages their answers to produce one final result. It's like a group of detectives discussing a clue and then agreeing on the final conclusion.

Why Is This a Big Deal?

The paper claims three main victories:

It's Self-Taught: Unlike other methods that require humans to write out step-by-step instructions (like "think step-by-step") for the AI to learn, Thoughtbubbles learns this skill automatically just by reading text.
It's Efficient: The researchers tested this on models of various sizes. They found that their method could solve problems just as well (or better) than standard models, but it only needed half the training data. It's like getting a Ferrari engine that runs on half the fuel.
It Knows Where to Focus: The AI learned to create these "bubbles" exactly where the text was confusing or uncertain. It didn't waste time on easy words; it focused its extra energy on the hard parts.

In Summary

Thoughtbubbles is a new way for AI to think. Instead of marching in a single file, the AI can temporarily split into a team of clones to tackle hard problems, then merge back together. It learns to do this automatically, making it smarter and more efficient without needing extra human instructions.

Technical Summary: Thoughtbubbles

Problem Statement

Current Transformer architectures face a fundamental limitation: they operate with a fixed computation budget and working memory. While techniques like Chain-of-Thought (CoT) allow models to break down complex problems into steps, they rely on serially generated natural language tokens, which are inefficient and cannot be applied during pretraining. Alternative approaches, such as "pause tokens" or inserting additional residual streams, often require manual design, apply uniformly across all layers (limiting adaptivity), or rely on explicit supervision signals beyond standard language modeling. There is a need for an architecture that can dynamically allocate parallel computation in latent space unsupervised, allowing the model to "think" more about difficult tokens without explicit step-by-step instructions.

Methodology: Thoughtbubbles

The authors propose Thoughtbubbles, a Transformer variant that natively performs parallel adaptive computation in latent space. The core mechanism involves learning to "fork" or delete residual streams dynamically during the forward pass.

Core Mechanisms

Forking and Residual Streams:
- The model introduces a "forking" operation between transformer layers. For each residual stream $x^{(k)}_i$ , the model computes a "keep score" and a "fork score."
- These scores are multiplied by a "cumulative score" propagated from previous layers to determine the stream's importance.
- A top-k selection is applied to these scores to decide which streams to keep and which to fork (clone). If a stream is forked, a new residual stream is created, effectively forming a "bubble" of latent computation for that specific token.
- To maintain a source for prediction, the original (rightmost) token is always forced to be kept.
Score Attenuation:
- To train the model to assign high scores to important streams, the authors attenuate both the attention mechanism and the residual updates using the cumulative scores.
- Specifically, the attention logits and the value vectors are modulated by the scores. This forces the model to rely more heavily on streams it deems important (high scores) and less on those it intends to prune (low scores), creating a self-reinforcing loop where difficult tokens naturally accumulate more computational resources.
Positional Embeddings:
- To handle the variable number of forks, the authors modify Rotary Position Embeddings (RoPE). They apply a "partial rotation" to forked tokens proportional to the number of forks, ensuring that forks of the same token remain "close together" in the positional space.
Output Aggregation:
- At the final layer, the model decodes each residual stream separately.
- The final output distribution is a weighted average of these distributions, where weights are determined by the cumulative scores.
- For efficiency in large models (1.9B parameters), a cheaper approximation is used: averaging the residual vectors themselves before applying the final softmax projection.
Inference Strategy:
- The authors introduce dynamic forking for autoregressive generation. The inference budget ( $\kappa$ ) is scaled proportionally to the input sequence length to prevent distribution shifts where short sequences would otherwise fork excessively compared to long ones.

Key Contributions

Unsupervised Adaptive Computation: Thoughtbubbles is the first architecture to enable unsupervised, dynamic allocation of latent parallel computation. It is trained as a standard decoder-only language model using only language modeling loss, without additional supervision or explicit reasoning traces.
Performance Efficiency: The method achieves superior performance compared to standard decoders and non-adaptive parallel computation baselines (e.g., duplicated filler tokens) while using only half the training token budget at the 1.9B scale.
Scalability: The approach demonstrates consistent performance gains across model sizes ranging from 150M to 1.9B parameters. Notably, a 319M Thoughtbubbles model outperforms a 772M baseline on perplexity.
Interpretability: The model autonomously learns to allocate more computation to regions of higher uncertainty (high posterior entropy), aligning with the hypothesis that difficult tokens require more "thinking."

Results

Zero-Shot Evaluation: On a suite of zero-shot benchmarks (LAMBADA, HellaSwag, BLiMP, AI2-ARC, PIQA), Thoughtbubbles consistently outperforms parameter-matched and computation-matched baselines.
GSM8K Reasoning: Using half the token budget of the baseline, the 1.9B Thoughtbubbles model achieves competitive results on the GSM8K math reasoning dataset.
Perplexity: The model achieves the lowest perplexity across all evaluations, even surpassing larger baseline models in some cases.
Ablation Studies:
- Removing attention masking (attenuation) significantly degrades performance, confirming the necessity of the score-based modulation for training the fork/keep decisions.
- Simply averaging logits of forked tokens performs better than keeping only the rightmost token, validating the utility of the parallel streams.
- Extending forking to all layers yields only marginal gains, suggesting that early-layer forking is sufficient for most tasks.

Significance and Claims

The paper claims that Thoughtbubbles represents a significant step toward unified train-time and test-time scaling behaviors. Unlike CoT, which requires exposure to step-wise instructions during pretraining, Thoughtbubbles learns to perform adaptive computation implicitly during the standard pretraining phase.

The authors posit that this method unlocks a form of input-adaptivity in Transformers, allowing them to solve complex, multi-step problems by dynamically expanding their working memory and computational depth for specific tokens. The work suggests a future where latent adaptive computation becomes a standard, integrated component of language model pretraining, moving beyond the limitations of fixed-budget architectures and explicit, serial reasoning traces.