🤖 machine learning

Learn from your own latents and not from tokens: A sample-complexity theory

This paper theoretically and empirically demonstrates that training generative models to predict their own latent representations, rather than raw tokens, achieves constant sample complexity relative to hierarchical depth, thereby offering a significant data-efficiency advantage over traditional supervised or token-level self-supervised learning.

Original authors: Daniel J. Korchinski, Alessandro Favero, Matthieu Wyart

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniel J. Korchinski, Alessandro Favero, Matthieu Wyart

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Data Glutton"

Imagine you are trying to teach a robot to understand English. To do this, you feed it every book, website, and tweet ever written (trillions of words). This is how current AI models (like the ones that write essays or chat with you) learn. They look at individual words ("tokens") and try to guess the next one.

However, a human child learns to speak fluently with a tiny fraction of that data. They don't need to read the entire internet; they just need to talk to their parents and friends.

The paper asks: Why is the AI so hungry for data? The authors suggest it's because the AI is trying to learn the wrong things. It's staring at the raw "pixels" or "letters" (tokens) instead of understanding the underlying concepts (latents) that those letters represent.

The Analogy: The Russian Nesting Dolls

To explain their theory, the authors use a model called the Random Hierarchy Model (RHM). Think of this as a set of Russian nesting dolls or a family tree.

The Leaves (Tokens): At the very bottom, you have individual letters or sounds.
The Middle Layers (Latents): These letters combine to form words, which form phrases, which form sentences. Each level is a "latent" (a hidden concept).
The Top (Root): The final meaning or the "parent" of the whole structure.

The Old Way (Token-Level Learning):
Imagine you are trying to figure out the structure of this family tree, but you are only allowed to look at the bottom layer (the leaves). To understand how the great-grandparents are related, you have to trace a path all the way down to the leaves and back up.

The Problem: As the tree gets deeper (more levels of meaning), the signal gets weaker and weaker. It's like trying to hear a whisper from the top of a skyscraper while standing at the bottom; the noise drowns it out.
The Cost: To learn a deep hierarchy this way, you need an exponential amount of data. If the tree has 4 levels, you might need millions of examples. If it has 10 levels, you might need more data than exists in the universe.

The New Way (Learning from Your Own Latents):
Now, imagine you are allowed to look at the middle layers of the tree. Instead of guessing the next letter, you guess the next concept.

The Solution: If you are at level 2, you predict level 3. You don't have to look all the way down to the leaves. The signal is strong and clear because you are working with "neighbors" in the hierarchy.
The Result: You can learn the whole structure with a constant amount of data, regardless of how deep the tree is. You don't need millions of examples; you just need enough to see the pattern once or twice.

The Three Experiments

The authors didn't just do math; they built three things to prove this works:

The Clustering Algorithm (The Detective):
They built a simple computer program that acts like a detective. It looks at groups of words, sees which ones behave similarly (synonyms), and groups them together to form the next level of the hierarchy.
- Result: It successfully rebuilt the entire family tree using a tiny amount of data, proving that if you look at the right level, the job is easy.
The Stacked Neural Network (The Ladder):
They built a deep neural network where each layer is a small team. The first team groups the raw letters into words. The second team groups those words into phrases. Each team teaches the next team.
- Result: This network learned just as efficiently as the simple detective algorithm. It proved that deep networks can learn hierarchies efficiently if they are structured to predict their own internal representations.
The "Data2vec" Analysis (The Hidden Genius):
They looked at a popular AI method called Data2vec. This method already tries to predict its own internal representations. The authors showed that Data2vec is implicitly doing the same thing as their "Ladder" network.
- Result: Data2vec is already a hierarchical learner! It doesn't need to be stacked in complex ways (like some other recent theories suggest); it naturally discovers the layers of the hierarchy on its own. This means we might not need to build complicated new architectures; we just need to use methods that already exist but understand why they work.

The Core Takeaway

The paper claims that predicting raw data (tokens) is inefficient because the signal gets diluted over long distances. Predicting your own internal concepts (latents) is efficient because the signal stays strong.

Token Learning: Like trying to solve a 1,000-piece puzzle by looking at the dust on the table. You need a million tries to find the right piece.
Latent Learning: Like looking at the picture on the puzzle box. You can solve it with just a few tries.

The authors conclude that the reason biological learners (like humans) are so data-efficient is likely because our brains are constantly predicting their own internal models of the world, rather than just memorizing raw sensory input. By shifting AI training to focus on "learning from your own latents," we can potentially build smarter AI that learns with far less data.

Technical Summary: Learn from your own latents and not from tokens: A sample-complexity theory

Problem Statement

Generative models, including diffusion models and large language models (LLMs), have achieved remarkable empirical success but rely on training data orders of magnitude larger than what biological learners require. Frontier LLMs are trained on $10^{13}$ – $10^{14}$ tokens, far exceeding the exposure of a child to reach adult-level competence. While hypotheses such as multimodality and grounding have been proposed, this work investigates the hypothesis that learning is more efficient not at the level of raw tokens, but in an abstract latent space.

Specifically, the paper addresses the theoretical gap in understanding self-supervised learning (SSL) methods that predict their own latent representations (e.g., data2vec, JEPA) versus those that predict raw tokens (e.g., masked language modeling, diffusion). Key questions include:

By how much does predicting one's own latents improve data efficiency compared to token-level prediction?
Is there a theoretical benefit to explicitly stacking such methods into multi-scale hierarchies (e.g., H-JEPA)?

To answer these, the authors utilize the Random Hierarchy Model (RHM), a simplified probabilistic context-free grammar (PCFG) that generates visible tokens via a tree of hidden latent variables of depth $L$ . The RHM allows for tractable sample-complexity analysis of compositional structure.

Methodology

The authors analyze three distinct approaches to learning the latent hierarchy of the RHM:

Iterative Latent Clustering (ILC): An algorithmic approach where the model recovers the hierarchy level-by-level. At each level $\ell$ , the algorithm estimates empirical context vectors for tuples of tokens and clusters them to identify synonyms (tuples sharing the same parent). This constructs the latent labels for level $\ell+1$ , which then serve as inputs for the next level.
Stacked Latent-Clustering (SLC) Network: A differentiable, end-to-end neural architecture mirroring the ILC algorithm. It consists of a stack of $L-1$ modules, each containing a Predictor (which predicts cousin tokens to estimate context vectors) and a Clusterer (which maps prediction vectors to discrete latent codes via contrastive objectives). The network is trained via gradient descent, with options for global backpropagation or local learning rules (stop-gradients between modules).
Theoretical Analysis of data2vec: The authors analyze the popular SSL method data2vec, which uses a teacher-student setup to predict latent representations of masked inputs. They propose a theoretical framework where the teacher's targets implicitly carry learned latent information, allowing the student to perform hierarchical latent prediction inductively.

Key Contributions and Results

1. Sample Complexity Bounds

The paper establishes a rigorous separation in sample complexity between token-level objectives and latent-prediction objectives:

Token-level SSL (Supervised or Masked): To recover the latent tree of depth $L$ , token-level objectives require a number of samples scaling as $P \sim m^{L+1}$ (where $m$ is the number of production rules per symbol). This is exponential in the depth $L$ because the signal must propagate through the entire hierarchy, with each unresolved rule attenuating the correlation signal by a factor of $m$ .
Latent Prediction (ILC, SLC, data2vec): The authors prove that predicting one's own latents reduces the sample complexity to $P \sim m^3$ $P \sim m^{3}$ (up to logarithmic factors). This scaling is independent of the depth $L$ .
- Mechanism: Once a level is recovered, both the conditioning object and the prediction target are lifted to that latent level. The problem at each subsequent level becomes a local synonym-clustering problem with the same statistical strength as the first step, avoiding the exponential accumulation of noise.

2. Algorithmic and Neural Validation

ILC: Numerical experiments confirm that the iterative clustering algorithm recovers the full non-root hierarchy with accuracy scaling as $m^3$ , independent of $L$ .
SLC: An end-to-end trained SLC network achieves the same $m^3$ scaling. Notably, the authors show that this scaling holds even when using strictly local learning rules (stop-gradients between modules and no global backpropagation), suggesting biological plausibility.
data2vec: The authors provide the first sample-complexity analysis of data2vec. Empirical results show that data2vec on the RHM also scales as $m^3$ . Theoretically, they argue that data2vec implicitly performs hierarchical latent prediction: as the teacher updates its weights, the targets for the student progressively incorporate higher-level latent representations, effectively lifting the prediction task up the hierarchy without explicit architectural stacking.

3. Implications for Hierarchical Stacking

The results suggest that explicit stacking of latent-prediction modules (as in H-JEPA) is largely redundant. A single latent-prediction module (like data2vec) is capable of implicitly discovering and utilizing multi-scale hierarchical structures, achieving the optimal $m^3$ sample complexity without the need for complex, manually stacked architectures.

Significance

The paper places the intuition that "token-level prediction is suboptimal" on a firm quantitative footing. By demonstrating that learning from one's own latents can yield exponential gains in sample efficiency over token-level objectives, the work provides a theoretical justification for the shift toward latent-space prediction in generative modeling.

Furthermore, the analysis of data2vec suggests that current state-of-the-art self-supervised methods may already be implicitly hierarchical, challenging the necessity of explicit hierarchical supervision schemes. The authors conclude that developing synthetic data models like the RHM is crucial for designing and theoretically understanding new architectures before scaling them to natural language and images.

The work also highlights the potential for biologically plausible learning rules (local updates, predictive coding) to achieve optimal sample complexity, bridging the gap between machine learning efficiency and biological learning constraints.