💬 NLP

From Tokens to States: LLMs as a Special Case of World Models and the Continuous Path Beyond

This paper challenges the binary distinction between large language models and world models by arguing that LLMs are a degenerate special case of world models, proposing a continuous spectrum of architectures from token prediction to latent-space simulation while highlighting the significant research challenges in scaling to action-labelled environments and adapting transformer architectures for continuous-state prediction.

Original authors: Paul Dubois

Published 2026-06-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Paul Dubois

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Misunderstanding: Two Different Worlds?

Imagine the AI community is arguing about two different types of travelers.

Group A (LLMs): Says, "We are just word predictors. We look at the last word you typed and guess the next one. We don't know anything about the real world; we just know how words fit together."
Group B (World Models): Says, "We are reality simulators. We build a mental map of how the world works so we can plan ahead, like a chess player thinking three moves ahead."

The paper argues that these two groups are actually looking at the same thing from different angles. The author, Paul Dubois, says: "You aren't looking at two different species; you are looking at a puppy and a full-grown wolf. The puppy is just a very restricted version of the wolf."

Claim #1: The "Puppy" is actually a "Wolf" in a tiny room

The paper claims that Large Language Models (LLMs) are actually World Models, but they are stuck in a very small, boring room.

The Room (State): In a normal World Model, the "room" is the whole universe (cars, weather, people, physics). In an LLM, the "room" is just a string of words.
The Action: In a normal World Model, you can do many things (drive, jump, talk). In an LLM, the only thing you can do is add one word.
The Trick: Even though the LLM is only allowed to add words, it secretly builds a map of the world inside its brain (its hidden layers).
- The Analogy: Imagine a person who is only allowed to speak in single words. If you ask them to play a game of chess using only words like "Pawn," "King," "Check," they can't move the pieces physically. But inside their head, they are visualizing the entire chessboard perfectly. The words are just the interface, but the world model is living inside their mind.

The paper proves this by showing that models trained only on text can actually "see" things like the layout of a chessboard or the passage of time, even though they were never taught those concepts directly.

Claim #2: It's a Smooth Ramp, Not a Cliff

Many experts (like Yann LeCun) say we need to abandon word-predicting models entirely and switch to a completely new system called JEPA (a system that predicts future states in a "latent" or hidden space).

The paper says: "No, don't jump off a cliff. Just walk up a ramp."

The author describes a continuous spectrum (a smooth slide) connecting the simple word-predictor to the complex reality-simulator. You don't have to throw away the old model; you just relax the rules one by one.

Here is the path up the ramp:

The Starting Point (Standard LLM): Predicts one word at a time.
- Pros: We have infinite data (the whole internet) and a perfect tool (the Transformer architecture) for this.
Step 1: Multi-Token Prediction: Predicts a few words at once.
- Change: We stop forcing the model to guess one letter at a time. It gets a tiny bit smarter at reasoning, but we still use internet text and the same tools.
Step 2: Future Summary: Predicts a compressed summary of the future.
- Change: Instead of guessing the next word, it guesses the "gist" of the next paragraph. Still uses internet text.
Step 3: Next-Latent Prediction: Predicts the next "thought" (a hidden state) instead of a word.
- Change: The model stops outputting words and starts outputting internal concepts. This is harder to train, but we can still use internet text.
The Top of the Ramp (JEPA): Predicts the next state of the real world based on an action.
- The Big Drop: Here, the easy stuff disappears.
  - Data Problem: We can't use the internet anymore. We need data from robots, self-driving cars, or video games where we know exactly what action caused what result. This data is rare and hard to get.
  - Tool Problem: The "Transformer" tool we used for words might not work well for continuous reality. We might need a brand-new tool.

The Two Big Bottlenecks

The paper identifies two main reasons why we are stuck at the bottom of the ramp (the LLM stage) and haven't reached the top (the JEPA stage):

The Data Cliff:
- At the bottom: We have trillions of words on the internet to learn from. It's free and easy.
- At the top: We need millions of specific, labeled interactions (like a robot bumping into a wall). This is expensive and hard to collect.
The Architecture Question:
- At the bottom: The Transformer is a machine built specifically for words. It's like a wrench that fits perfectly on a square bolt.
- At the top: We need to predict continuous reality (like fluid motion). The wrench might not fit. We might need a whole new tool, but we don't know what it looks like yet.

Conclusion: Don't Burn the Bridge

The paper concludes that we don't need to throw away our current AI models.

LeCun is right that simple word-predictors can't do complex physical planning on their own.
But he is wrong to say we need to start over.

Instead, we should view the current models as the first step on a long journey. We can slowly upgrade them, step-by-step, moving from predicting words to predicting concepts, and finally to predicting reality. The destination is clear, but the path is a gradual climb, not a sudden leap.

In short: LLMs are not "fake" world models; they are just world models with their hands tied behind their backs. If we slowly untie those hands, we get to the powerful AI we want, without needing to invent a completely new universe.

Technical Summary: From Tokens to States: LLMs as a Special Case of World Models and the Continuous Path Beyond

Problem Statement

The AI community currently frames the relationship between Large Language Models (LLMs) and World Models as a binary dichotomy: LLMs are viewed as mere token predictors, while World Models are defined as systems that simulate reality to support planning. Yann LeCun (2022) has argued that achieving general intelligence requires abandoning autoregressive token prediction in favor of latent-space architectures (specifically Joint Embedding Predictive Architectures, or JEPA). This paper challenges that framing as unnecessarily binary, arguing that it obscures a natural, continuous spectrum of capabilities and architectural evolution. The core problem addressed is the lack of a formal framework that unifies LLMs and World Models, and the absence of a clear roadmap for transitioning from one to the other without discarding the practical advantages that have enabled LLMs to scale.

Methodology and Formalism

The paper employs a formal mathematical approach to redefine the relationship between LLMs and World Models, followed by a structural analysis of existing and emerging architectures.

1. Formal Embedding of LLMs
The author defines a World Model as a tuple $(S, A, T, \rho_0)$ , where $S$ is the state space, $A$ is the action space, $T$ is the transition function, and $\rho_0$ is the initial state distribution. The paper formally embeds an autoregressive LLM into this framework by making specific, restrictive choices:

State Space ( $S$ ): The set of all finite token sequences ( $V^*$ ).
Action Space ( $A$ ): The vocabulary of tokens ( $V$ ), where the only action is appending a token.
Transition Function ( $T$ ): A deterministic function that appends the chosen token to the sequence ( $s \cdot a$ ).
Policy ( $\pi_\theta$ ): The LLM itself, which predicts the probability distribution over the next token.

Under this definition, the LLM acts simultaneously as the world simulator (via the transition function) and the agent. The paper argues that LLMs are a "degenerate special case" of World Models, where the state space is restricted to discrete tokens and the transition is trivial.

2. The Continuous Spectrum Analysis
The paper constructs a spectrum running from standard Next-Token Prediction (NTP) to JEPA. It analyzes this spectrum as a sequence of steps, where each step relaxes exactly one constraint of the standard LLM formalism. The analysis tracks how each relaxation impacts two practical advantages of LLMs:

Access to internet-scale self-supervised data.
Compatibility with the Transformer architecture (co-designed for discrete tokens).

Key Contributions

1. The Containment Claim
The primary theoretical contribution is the proof that LLMs $\subset$ World Models. The paper demonstrates that World Models are a strict generalization of LLMs, not a replacement. This reframes the research goal from "abandoning LLMs" to "progressively relaxing LLM constraints."

2. Identification of Five Constraints
The paper isolates five specific constraints that define the LLM special case within the broader World Model class:

State Representation: Restricted to discrete token sequences (vs. arbitrary continuous/latent states).
Action Granularity: Restricted to one micro-token per step (vs. semantic agent actions).
State Persistence: State must be re-derived from context every pass (vs. explicit stored variables).
Training Supervision: Relies on next-token prediction on raw text (vs. paired state-action-next state tuples).
Layer Architecture: Restricted to Transformers co-designed for discrete tokens.

3. The "Data Cliff" and "Architecture Gap"
The paper identifies two critical bottlenecks that emerge as one moves along the spectrum:

The Data Question: While early spectrum steps (NTP, Multi-Token Prediction, Future Summary) can utilize internet-scale text, the final step to JEPA requires paired (observation, action, next observation) data from instrumented environments. This data is orders of magnitude scarcer ( $\sim 10^9$ samples) than internet text ( $\sim 10^{13}$ tokens).
The Architecture Question: The Transformer architecture is highly optimized for discrete tokens. As prediction targets move to continuous latent spaces (Next-Latent, JEPA), the Transformer's inductive bias becomes less optimal. The paper suggests the field may lack a "crystallized" architecture for continuous-state prediction analogous to the Transformer for text.

4. Empirical Reinterpretation of Internal Representations
The paper synthesizes mechanistic interpretability findings (e.g., OthelloGPT, Llama-2) to argue that "world models" already exist inside LLMs. These models encode rich, linear representations of board states, geographic space, and time in their hidden activations, despite being trained only on token sequences. This supports the claim that the "world model" resides in the latent activations, while tokens serve merely as the interface.

Results and Evidence

The paper does not present new experimental results but synthesizes existing research to support its theoretical claims:

OthelloGPT & Chess Models: Demonstrate that hidden activations encode full board states with >99% accuracy, proving world-state representation is possible within token-predicting architectures.
Multi-Token Prediction (MTP): Cites Gloeckle et al. (2024) and DeepSeek-V3, showing that predicting $k$ tokens simultaneously improves reasoning without sacrificing data or architecture advantages.
Future Summary & Next-Latent: Cites Mahajan et al. (2026) and Teoh et al. (2026) to show intermediate architectures that predict compressed futures or latent states, bridging the gap between NTP and JEPA.
Chain-of-Thought (CoT): Reinterprets CoT not as a new paradigm, but as a mechanism to increase the planning horizon within the existing token-state world model, effectively moving along the spectrum's granularity axis.

Significance and Claims

The paper claims its significance lies in dissolving the false binary between "token predictors" and "world models." By framing LLMs as a constrained subset of World Models, it suggests a gradient ascent path toward general intelligence rather than a discontinuous jump.

Incremental Transition: The path forward is not to discard LLMs but to incrementally relax constraints. Intermediate steps (MTP, Future Summary, Next-Latent) offer a way to improve world-model capacity while retaining the scalability of internet-scale data and Transformer architectures.
Clarification of the Frontier: The paper posits that the true research frontier lies in the transition to JEPA, where both data availability and architectural fit collapse simultaneously. It suggests that solving the "action-conditioned world model" problem requires not just more data, but potentially a new architectural primitive for continuous-state prediction.
Re-evaluation of LeCun's Critique: The paper argues LeCun is correct that current LLMs cannot perform closed-loop planning in the physical world, but incorrect that this requires abandoning the token-prediction paradigm entirely. Instead, the spectrum view shows that CoT and latent-space prediction are natural extensions of the same underlying formalism.

In conclusion, the paper asserts that the destination (action-conditioned latent-space world models) is clear, but the path is a continuous spectrum of architectural and data evolution, not a binary choice between two opposing technologies.