📊 statistics

Task Ecologies and the Evolution of World-Tracking Representations in Large Language Models

This paper establishes a theoretical framework demonstrating that autoregressive language models evolve world-tracking representations by minimizing a Jensen-Shannon excess term tied to training ecology equivalence classes, a principle validated through small-scale experiments to predict specific failure modes and recovery mechanisms in representational selection.

Original authors: Giulio Valentino Dalla Riva

Published 2026-04-08

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Giulio Valentino Dalla Riva

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to predict the next word in a sentence. You might wonder: Does this robot actually "understand" the world, or is it just a fancy parrot memorizing patterns?

This paper, Task Ecologies and the Evolution of World-Tracking Representations in Large Language Models, tries to answer that question. Instead of looking at massive, complex AI models (which are like black boxes), the author uses tiny, simple AI models as "laboratory mice." By watching these small models, we can see exactly how they learn to represent reality.

Here is the core idea broken down into simple concepts and analogies.

1. The "Map" vs. The "Territory"

Imagine the world is a vast, complex territory with mountains, rivers, and cities. The AI doesn't see the territory directly; it only sees "signs" (words).

The Territory: The actual state of the world (e.g., "It is raining," "The code has a bug," "The sentence is grammatically correct").
The Map: The internal representation the AI builds to predict the next word.

The paper asks: How detailed does the AI's map need to be to be perfect?

2. The "Ecology" (The Training Environment)

The author uses the word Ecology to describe the specific set of tasks the AI is trained on.

Analogy: Think of the AI as a fish. If you only train the fish in a tank with only blue water and no rocks, it will never learn to navigate around rocks. Its "ecology" is limited to blue water.
The Finding: The AI only learns to distinguish things that are necessary to solve the tasks it sees. If the training data never asks the AI to distinguish between two very similar-looking but logically different situations, the AI will merge them into one bucket. It won't "know" the difference because it never needed to.

3. The "Perfect Map" (Zero Excess)

The paper proves a mathematical rule about the "perfect" map.

The Rule: The most efficient map is one that keeps separate only the things that actually matter for the tasks, and merges everything else.
The Metaphor: Imagine you are packing for a trip.
- Too Simple: You pack one giant suitcase with everything mixed up. You can't find your socks. (High error).
- Too Complex: You pack a separate box for every single sock, every button, and every thread. (Wasteful, high complexity).
- Just Right (Ecologically Veridical): You pack one box for "socks," one for "shirts," and one for "shoes." You don't separate "left socks" from "right socks" unless the task specifically requires it. This is the Minimum-Complexity Solution.

The paper shows that if an AI is trained perfectly, it naturally settles on this "Just Right" map. It stops distinguishing things that don't help it predict the next word.

4. The "Freeze" vs. The "Adaptation"

The paper checks if this theory holds true for different types of AI.

Frozen Models (The Static Map): If you take a finished AI and just use it (without changing its brain), it acts like a fixed map. The theory works perfectly here.
In-Context Learning (The Flexible Reader): If you give the AI a prompt to "act like a doctor," it changes its behavior. But the paper argues this doesn't actually change the map inside its brain; it just changes which part of the map it looks at.
Fine-Tuning (The New Map): If you retrain the AI on a new task, you are effectively giving it a new brain. The old rules don't apply because the "map" itself has changed.

5. The "Two-Worlds" Problem (Training vs. Reality)

This is one of the most important insights.

The Training World: The data the AI sees while learning (e.g., Wikipedia articles).
The Evaluation World: The real-world tests we give it later (e.g., "Write code that doesn't crash" or "Solve this math problem").

The Problem: Sometimes, the Training World doesn't force the AI to learn a distinction, but the Evaluation World does.

Analogy: Imagine you train a driver only on empty highways. They learn to drive fast. But then you put them in a city with traffic lights. They fail because the "highway ecology" never taught them to stop for red lights.
The Solution: The paper suggests that if we want the AI to learn these missing distinctions, we need to "inject" the evaluation tasks into the training process. We have to mix the "city driving" data into the "highway" training so the AI is forced to build a map that includes traffic lights.

6. The "Simplicity Pressure" (Why AI Makes Mistakes)

Why do AI models sometimes fail to understand complex logic?

The Theory: AI models have a natural "laziness" or "simplicity pressure." They prefer to merge things together to save energy (complexity).
The Result: They will happily merge two concepts if the "reward" for keeping them separate is small.
Example: If distinguishing between "a cat" and "a dog" helps predict the next word 99% of the time, the AI learns it. But if distinguishing between "a cat" and "a slightly different shade of cat" only helps 0.01% of the time, the AI will likely merge them and treat them as the same thing. This is why AI can be "hallucinating" or missing subtle details—it's optimizing for the simplest map that works mostly well.

7. The "Evolution" of AI

Finally, the paper looks at AI not as a single robot, but as a population.

Analogy: Think of AI models like species in nature.
- Mutation: Engineers tweak the code or data.
- Selection: We pick the models that perform best on tests.
- Heredity: The next generation of models is built from the winners.
The Insight: Over time, this evolutionary process pushes the population of AI models toward better "maps." If a model fails a test (like a code check), it gets discarded. The survivors are the ones that learned the necessary distinctions. This explains why, over time, AI gets better at things we care about, even if the individual training steps are just about predicting words.

Summary

This paper argues that AI models are not magic; they are efficient map-makers.

They build maps based strictly on what they are asked to do (the Ecology).
They make their maps as simple as possible, merging details that aren't strictly necessary.
If they fail at a task, it's often because the training data didn't force them to draw that specific line on the map.
To fix this, we must change the training environment (the ecology) to force them to learn those missing distinctions.

By using tiny, observable models as "laboratory mice," the author proves these rules mathematically, giving us a clear blueprint for how to build better, more "truthful" AI in the future.

1. Problem Statement

The paper addresses a fundamental question in the theory of Large Language Models (LLMs): Does autoregressive next-token training drive models to develop internal representations that accurately track the underlying "world" (latent states), or do they merely learn distinctions that are locally useful for prediction?

While recent debates (e.g., the "Platonic Representation Hypothesis") suggest models converge toward a shared statistical model of reality, others argue this is a scale artifact or that models only learn local neighborhoods. The author seeks to isolate and mathematically define the conditions under which an LLM's internal encoding of world states becomes ecologically veridical (i.e., it preserves exactly the distinctions required by the training task ecology and merges only those that are irrelevant).

2. Methodology: The "Model Organism" Approach

Instead of analyzing massive, opaque frontier models, the author employs a model organism methodology using small, frozen autoregressive transformers (inspired by Karpathy's microgpt).

Experimental Setup: The "world" consists of finite latent states (e.g., languages, text corpora like Alice in Wonderland, Dante's Commedia, and the Communist Manifesto).
Observability: Because the models and ecologies are small and finite, the author can directly observe and enumerate:
- Induced partitions of world states (which states the model merges vs. separates).
- Exact finite-ecology quantities (entropy, Jensen-Shannon divergence).
- Population-level selection trajectories.
Theoretical Framework: The paper adapts the Ecological Veridicality framework from evolutionary perception (Dalla Riva, 2026) to the autoregressive setting. It treats the training corpus as a "task ecology" $\mu$ that defines equivalence classes of world states.

3. Key Contributions and Theoretical Results

A. Static Optimality: The Decomposition Theorem

The paper proves that for any encoding $p$ of latent world states, the Bayes-optimal next-token cross-entropy loss ( $L^*_D$ ) decomposes exactly into two terms:
$L^*_D(p) = H(Y | C, W) + \text{Excess}(p)$

Irreducible Term: $H(Y | C, W)$ is the conditional entropy of the next token given the context and the true world state (the theoretical lower bound).
Jensen–Shannon Excess: The second term is a weighted sum of Jensen–Shannon (JS) divergences within the cells of the encoding partition.
Key Insight: The excess loss vanishes if and only if the encoding preserves the training ecology's equivalence classes. If the model merges two states that the ecology distinguishes, it incurs a strictly positive excess loss.

B. Architectural Applicability

The paper determines when this "fixed-encoding" analysis applies to specific transformer architectures:

Frozen Dense & MoE Transformers: Satisfy the "cognitive impenetrability" condition. Their weights are fixed, so the induced encoding is invariant across tasks (contexts).
In-Context Learning (ICL): Does not enlarge the model's separation set. ICL selects which context to use but cannot create new distinctions the frozen weights cannot already make.
Per-Task Adaptation: Breaks the premise. Fine-tuning or LoRA changes the weights per task, meaning the encoding is no longer fixed, invalidating the static ecological veridicality theorem for that specific lineage.

C. Minimum-Complexity Veridicality

Among all encodings that achieve zero excess loss (i.e., are ecologically veridical), the paper identifies the minimum-complexity solution:

The optimal encoding is the quotient partition $W / \sim_\mu$ .
This partition merges all and only the states that are equivalent under the training ecology.
Any finer encoding (preserving unnecessary distinctions) incurs higher representational complexity (entropy) without improving the Bayes-optimal loss.
Simplicity Pressure: Under a regularized objective (Loss + $\beta \times$ Complexity), distinctions with low predictive gain (low JS divergence) relative to their complexity cost are preferentially merged. This predicts a specific failure mode: models shed low-gain distinctions first.

D. Dynamic Evolution and Two-Ecology Extension

The paper models the LLM ecosystem as a Darwinian population where:

Units of Selection: Whole trained models (lineages) are the replicators.
Selection Pressure: Lineages are selected based on performance across a deployment ecology.
Two-Ecology Gap: There is often a mismatch between the Token Ecology (next-token prediction) and the Evaluation Ecology (benchmarks, user preferences).
- Some distinctions (e.g., code correctness, logical consistency) have weak token-level signatures but are heavily weighted in evaluation.
- Ecology Injection: Post-training (fine-tuning, RLHF) acts as a mechanism to inject the evaluation ecology into the token objective. The paper derives a threshold ( $\alpha^*$ ) showing how much injection is required to "rescue" a distinction that the base token ecology would otherwise merge.

4. Empirical Results

Using the microgpt model organism, the author validates the theory with exact finite-ecology checks:

Static Decomposition: Verified that the measured excess loss perfectly matches the calculated Jensen–Shannon excess term.
Split-Merge Threshold: Confirmed that as a complexity penalty ( $\beta$ ) increases, the model merges distinctions exactly when their predictive gain falls below the theoretical threshold derived in Theorem 55.
Off-Ecology Failure: Models trained on specific languages (e.g., English, French) show increased error and divergent behavior when probed with unseen languages (e.g., Voynich manuscript), validating the prediction that merged distinctions lead to positive excess on refined ecologies.
Two-Ecology Rescue: Experiments on bracket balancing in Lisp code showed that while the base token ecology failed to learn the structure, injecting the evaluation signal (via post-training or specific data mixes) successfully "rescued" the distinction, with the population converging toward the injected ecology's requirements.

5. Significance and Implications

Theoretical Precision: The paper moves beyond vague claims of "world models" to provide a precise, information-theoretic definition of when and why models track the world. It proves that topological convergence (agreeing on which states are distinct) is guaranteed under specific conditions, even if geometric convergence (how far apart those states are in vector space) is not.
Failure Modes: It predicts that models will fail on distinctions that are:
1. Low-gain for the token objective (simplicity pressure merges them).
2. Not separated by the training ecology but required by the deployment ecology (the "gap" problem).
Methodological Shift: It advocates for using small, observable models as "laboratory organisms" to test representational theories, rather than relying solely on scaling laws of opaque systems.
Niche Construction: The paper highlights that LLMs are not just passive observers; they reshape the world (via generated text in training data), creating a feedback loop where the "truth" they track is partially constructed by their own outputs.

In summary, the paper establishes that autoregressive training selects for representations that are ecologically veridical up to the limits of model capacity and simplicity pressure, and that inter-model selection and post-training are necessary to recover distinctions that the raw token objective misses.