🤖 machine learning

Continuous Latent Contexts Enable Efficient Online Learning in Transformers

This paper demonstrates that equipping transformers with continuous latent context tokens enables them to efficiently implement online learning algorithms like weighted majority and Q-learning, allowing small models to outperform significantly larger LLMs on long-term adaptive tasks.

Original authors: Emile Anand, Abdullah Ateyeh, Xinyuan Cao, Max Dabagia

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Emile Anand, Abdullah Ateyeh, Xinyuan Cao, Max Dabagia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn a new game by watching a group of four experts play. Every round, the experts make a guess, and then you find out who was right. Your goal is to make the best guess possible over time by figuring out which expert is the most reliable.

This paper is about teaching AI models (specifically "Transformers") how to do this kind of learning efficiently, without needing to rewrite their entire "brain" (parameters) every time they get new information.

Here is the breakdown of their discovery using simple analogies:

1. The Problem: The "Short-Term Memory" AI

Standard Large Language Models (LLMs) are great at looking at a long conversation and guessing what comes next. This is called "in-context learning." However, when the game goes on for a long time (hundreds of rounds), these models struggle. They tend to forget the big picture or get confused by the sheer volume of past guesses. They act like someone trying to remember a 100-page story just by looking at the last few sentences, rather than keeping a running summary in their head.

2. The Solution: The "Continuous Scratchpad"

The authors propose giving the AI a special tool: Continuous Latent Context Tokens.

The Old Way (Discrete Tokens): Imagine the AI trying to keep a scorecard by writing down words like "Expert A is good" or "Expert B is bad." This takes up a lot of space and gets messy.
The New Way (Continuous Tokens): Imagine the AI has a tiny, invisible scratchpad where it can write numbers that aren't words. It can blend these numbers together like mixing paints.
- Instead of writing "Expert A is 80% reliable," it holds a single, smooth "color" that represents that 80% reliability.
- When a new round happens, it doesn't need to write a new sentence. It just subtly shifts the color on the scratchpad to update the score.

This "scratchpad" is compact (it takes very little space) and persistent (it stays with the AI from one round to the next).

3. The Proof: Teaching the AI Math

The researchers didn't just hope this would work; they proved it mathematically and then trained a small AI to do it.

The Weighted Majority Algorithm: They showed that with this scratchpad, a tiny AI can perfectly mimic a famous math algorithm used to track expert reliability. It's like giving the AI a calculator that automatically updates the "trust score" for every expert instantly.
Q-Learning (The Video Game Analogy): They also taught the AI to play a simple video game (a Markov Decision Process). In this game, the AI has to learn which moves give the best rewards.
- Normally, an AI needs a massive table to remember the value of every possible move.
- With the continuous scratchpad, the AI stores this entire "value table" as a few blended numbers. It can update its strategy instantly after every move, just like a human player learning from experience.

4. The Surprise: Small AI vs. Giant AI

The most surprising part of the paper is the experiment with "Frontier" models (huge, powerful AIs like DeepSeek-V3 and Qwen-3-14B).

The Setup: They asked these giant AIs to play the expert-guessing game.
The Result:
- Without the Scratchpad: The giant AIs struggled. They relied on short-term memory (like "the last person was right, so I'll guess that") and performed poorly over long sequences.
- With the Scratchpad (The "Note"): The researchers allowed the giant AIs to write a short "note" after every round to summarize what they learned.
- The Outcome: When the giant AIs were allowed to write these notes, their performance skyrocketed. They started acting like the small, mathematically perfect AI the authors built.

The Catch: The giant AIs didn't naturally know how to write a good note. They had to be prompted to do so. When they did, they realized that summarizing the past (e.g., "Expert A is 90% accurate") was much better than just remembering the raw history.

5. The Bottom Line

The paper argues that continuous latent contexts (the invisible, blended-number scratchpad) are the key to making AI efficient at long-term learning.

Small AI + Scratchpad: Can learn complex online strategies perfectly and efficiently.
Huge AI + Scratchpad: Can suddenly become much better at long-term decision-making, outperforming even larger models that don't have this "state" mechanism.

In short, the paper suggests that to make AI truly smart at learning over time, we shouldn't just make the models bigger; we should give them a better way to hold onto a summary of what they've learned, using a "continuous" internal state rather than just a long list of words.

Technical Summary: Continuous Latent Contexts Enable Efficient Online Learning in Transformers

Problem Statement

Large Language Models (LLMs) demonstrate strong in-context learning capabilities, inferring predictors from labeled examples without parameter updates. However, many interactive settings require online decision-making, where a model must adapt over long multi-turn horizons in response to feedback. Effective online learning requires maintaining a compact representation of learned information (an algorithmic state) to inform future decisions.

Standard Chain-of-Thought (CoT) prompting represents intermediate reasoning via discrete tokens, which can become prohibitively long for complex, long-horizon tasks. Recent work on continuous chain-of-thought (e.g., COCONUT) suggests that continuous latent states can carry information in "superpositions" of vocabulary tokens, potentially offering a more compact representation of context. The central question addressed by this paper is whether continuous latent context tokens can equip transformers to effectively implement online learning algorithms, specifically by storing and updating algorithmic states as linear combinations of feature embeddings.

Methodology

The paper employs a dual approach combining theoretical construction and empirical validation:

1. Theoretical Constructions

The authors provide explicit constructions of constant-depth causal transformers equipped with continuous latent context tokens (following the COCONUT architecture) to implement two foundational online decision-making procedures:

Weighted Majority Algorithm (WMA):
- Mechanism: The transformer maintains the state of WMA in a single continuous context vector. This vector acts as a superposition (linear combination) of expert embeddings, where the coefficient of each expert's embedding corresponds to the logarithm of that expert's weight.
- Operation:
  - Prediction: A soft attention head queries the state vector against expert prediction keys to compute the weighted majority prediction.
  - Update: Upon receiving ground truth, the state vector is updated via linear attention operations to increment the log-weights of correct experts (multiplicative weights update).
- Result: The construction proves that a constant-depth transformer with $O(1)$ heads can exactly recover the exponential weights update rule.
Tabular Q-Learning:
- Mechanism: The transformer implements online tabular Q-learning for Markov Decision Processes (MDPs). The Q-table is encoded as a set of continuous context tokens, where each token represents a row of the Q-table (action-indexed) as a superposition of state embeddings.
- Operation: The transformer performs two tasks per step:
  1. Evaluation: It retrieves current Q-values to select an action (implementing $\max_a Q(s, a)$ ).
  2. Update: It computes the Bellman update ( $Q(s, a) \leftarrow Q(s, a) + \alpha[r + \gamma \max_{a'} Q(s', a') - Q(s, a)]$ ) by writing terms into disjoint subspaces and summing them linearly in the residual stream.
- Result: A four-layer causal transformer is shown to implement the Q-learning update exactly at each step.

2. Empirical Validation

The authors trained small GPT-2-style transformers with continuous latent contexts using multi-curriculum objectives that did not directly supervise the latent states.

Unsupervised Learning from Expert Data: A small transformer (4 layers, ~200K parameters) was trained on expert prediction sequences. The model learned to maintain latent contexts that encoded expert reliability, achieving performance comparable to the optimal multiplicative weights algorithm.
Imitation Learning of Q-Learning: A similar small transformer was trained on trajectories generated by tabular Q-learning. The model learned to output actions consistent with the tabular greedy policy. Linear probes confirmed that the Bellman update components (reward, current value, bootstrapped value) were explicitly represented in the residual stream.
Frontier LLM Evaluation: The authors evaluated large models (Qwen-3-14B and DeepSeek-V3) on interactive expert-prediction tasks. They compared a standard protocol against a protocol allowing the model to output a short, unsupervised "note" to carry state information.

Key Results

Theoretical Feasibility: The paper demonstrates that constant-depth transformers with continuous latent contexts can theoretically implement complex online learning algorithms (WMA and Q-learning) by storing algorithmic states as superpositions in a fixed number of context tokens.
Small Model Efficiency: In synthetic online prediction tasks, the small trained transformer (4 layers, 200K parameters) outperformed significantly larger and more complex LLMs (Qwen-3-14B with 14.8B parameters and DeepSeek-V3 with 671B parameters) on long-sequence decision-making tasks.
Latent State Recovery: In the trained small models, attention mechanisms were observed to focus on the latent context tokens, and linear probes successfully decoded the algorithmic state (expert weights or Q-values) from these tokens, confirming the models learned to use the context as a persistent state.
LLM State Limitations: Frontier LLMs showed some ability to adapt to feedback but performed significantly better when allowed to output an explicit "note" carrying state information. However, these LLMs often relied on linearly growing text notes (e.g., round-by-round transcripts) rather than compact state representations. In contrast, the proposed continuous latent context approach uses a constant number of tokens regardless of sequence length.

Significance and Claims

The paper claims that continuous latent contexts provide a simple and effective persistent state for transformers to implement online learning algorithms.

Algorithmic Interpretation: The results support an algorithmic interpretation of continuous latent reasoning, suggesting that transformers can use their context not just as a passive memory buffer, but as an active, persistent online-learning state that enables the implementation of specific algorithms.
Efficiency: The approach highlights the value of continuous latent representations over discrete token-based reasoning for tasks requiring long-horizon state maintenance. It suggests that small models equipped with the right inductive bias (continuous contexts) can outperform massive models that rely on discrete token chains for state tracking.
Limitations: The authors note that their theoretical constructions are worst-case optimal and that the emergence of these specific update rules from gradient-based training on natural language data remains an open question. They also acknowledge that their study is restricted to simplified decision-making settings, and generalization to real-world reasoning is not yet established.

The work positions continuous latent reasoning as a potential solution to the scalability issues of discrete chain-of-thought in online decision-making, offering a path toward more efficient and compact state maintenance in transformer architectures.