🤖 machine learning

Emergent Models: Intelligence from Tiny Substrates

This paper introduces Emergent Models, a machine learning paradigm where simple, open-ended dynamical substrates like cellular automata are evolved to solve tasks through emergent computational behaviors, demonstrating that such minimal systems can achieve latent universality and global generalization beyond traditional differentiable architectures.

Original authors: Giacomo Bocchese, Nicola Giacobbo, Etienne Guichard, James Wiles, Akshaj Devireddy

Published 2026-08-17

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Giacomo Bocchese, Nicola Giacobbo, Etienne Guichard, James Wiles, Akshaj Devireddy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of artificial intelligence as a giant library. For decades, the librarians have been trying to build the perfect bookshelf. They've stacked millions of books (data points) onto shelves (neural networks) and taught the shelf to guess what book comes next based on the ones before it. This works great if you ask about books you've already seen, but if you ask about a story that hasn't been written yet, the shelf often just guesses randomly or makes up nonsense. This is because the shelf is really good at memorizing patterns but bad at understanding the rules that make the story work in the first place.

Enter a different idea: what if, instead of building a giant bookshelf, we built a tiny, self-driving robot that learns to solve problems by moving around? This is the world of "Emergent Models." Think of it like a game of "Life" played on a computer screen. You have a grid of cells that can be on or off. You give them a simple rule, like "if two neighbors are on, turn on." Then, you watch what happens. Sometimes, incredibly complex patterns—like spaceships or gliders—emerge from those simple rules. The big question scientists are asking is: Can we use these tiny, rule-based robots to learn math and logic better than our giant bookshelves? Can a system with just a few hundred "parts" figure out the secret recipe for a cake, rather than just memorizing a list of ingredients? This paper dives into that question, exploring whether simple, evolving systems can be the key to smarter, more adaptable AI.

The Tiny Substrates That Think

The authors of this paper introduce a new way of thinking about machine learning called Emergent Models (EMs). Imagine you have a blank canvas (the "substrate") and a single, tiny paintbrush that follows one simple rule, like "move one step right and change color." In traditional AI, we try to paint a perfect picture by adjusting millions of brushstrokes at once. In this new approach, we set up the canvas and the brush, then let them run for a while. We don't tell the brush what to paint; we just watch to see if, after running for a while, the pattern that emerges happens to solve a problem, like adding two numbers together.

The core idea is that intelligence might not need a massive, complex brain. Instead, it might emerge from a simple, repeating process, much like how a flock of birds moves in a perfect formation without a single bird giving orders. The "brain" here is just a tiny set of rules running over and over again on a grid of data.

The Magic of "Latent Universality"

The paper proposes a fascinating concept called Latent Universality. Imagine you have a universal remote control (the "program") and a TV (the "substrate"). In most AI, you have to buy a new TV for every different show you want to watch. But in Latent Universality, the TV is fixed. You only change the remote control (the initial state of the grid). The authors suggest that if you have the right kind of TV (a specific type of mathematical grid), you could theoretically program it to do any calculation just by changing the starting pattern on the screen, without ever changing the TV's internal wiring.

They prove mathematically that this is possible in theory. If you have a grid that goes on forever and a simple rule, you can set the starting pattern to act like a calculator, a chess player, or a translator. The "program" is just the starting picture you draw on the grid.

The Experiments: Tiny Robots in Action

Since building a perfect, infinite grid is hard, the researchers tested this idea with a "zoo" of tiny, simple models. They didn't try to build a super-intelligent AI; they just wanted to see if these tiny systems could learn basic rules and then apply them to new, bigger numbers.

1. The Math Wizards (EM43)
They created a one-dimensional strip of cells (like a long line of dominoes) with a simple rule. They taught it to do math like $x + 1$ or $x \times 2$ .

The Result: It worked perfectly! The model learned the rule on small numbers (like 1 to 30) and then applied it flawlessly to huge numbers (up to 1,000) that it had never seen before.
The "Aha!" Moment: When they looked at how the model solved the problem, they saw beautiful, repeating geometric patterns moving across the grid. They call this "geometric grokking." It's as if the model didn't just memorize the answer; it discovered a visual dance that works for any number. This is something traditional AI struggles to do, especially with repeating patterns like "what is the remainder when you divide by 4?"

2. The Game of Life (GoL-EM)
Next, they used Conway's Game of Life, a famous cellular automaton where cells live or die based on their neighbors. They tried to teach this system to do math and balance a pole (CartPole).

The Result: It could learn simple math like $x + 1$ , but it was very finicky. Small changes to the starting pattern caused the whole system to crash. It was like trying to balance a house of cards in a windstorm. While it could balance a pole for a while, it wasn't very stable. The researchers found that searching for the right starting pattern was almost like guessing randomly; the system was so sensitive that it was hard to "train" it in the usual way.

3. The Continuous Learners (CEM1D & CEM2D)
Finally, they tried models that used smooth, continuous numbers (like real-world physics) instead of just "on/off" switches.

The Result: These were much better at controlling things. The 1D version could balance a pole very well, and the 2D version could learn to navigate a world where food sometimes turned into poison.
The Catch: Even these smart models had trouble. Sometimes they would get stuck spinning in circles or freezing up. They learned to adapt to the changing environment, but they weren't perfect. The researchers noted that these models are still very new and much less reliable than the giant AI models we use today.

What This Means (and What It Doesn't)

The paper is very clear about what it has and hasn't achieved. It does not claim to have built a better AI than the ones you see in movies or on your phone. In fact, it admits that these tiny models are currently much harder to train and less powerful than the big, complex ones.

However, it does suggest something exciting: that the secret to true intelligence might be iteration (repeating a simple step over and over) rather than size (having millions of parameters). The authors found that when these tiny models did learn, they learned the rule, not just the answer. This allowed them to guess correctly on numbers they had never seen, a skill called extrapolation.

They also introduce a new way of thinking about AI efficiency. They suggest that the best models might be the ones that use very few "hard" parts (the rules) but run them for a long time (many steps). This is the opposite of today's big AI, which uses a massive amount of "hard" parts but only runs them once.

The Bottom Line

This paper is a foundational step, like the first time someone built a working steam engine. It's not a car yet, and it's definitely not a spaceship. But it proves that a simple, repeating system can learn rules and apply them to new situations. The researchers show that with just a few hundred parameters, a system can learn to do math and control simple robots.

The big takeaway is that we might be overcomplicating AI. Instead of building bigger and bigger brains, maybe we should be building smaller, simpler systems that are allowed to think for a longer time. The paper suggests that the path to smarter AI might lie in these "emergent" behaviors, where intelligence grows out of simple rules, just like life grows out of simple chemistry. It's a hopeful, playful, and slightly wild idea that the future of intelligence might be found in the smallest, simplest things.

Technical Summary: Emergent Models: Intelligence from Tiny Substrates

Problem Statement

Traditional deep learning, particularly feed-forward neural networks (FFNNs), often struggles with global generalization and extrapolation. While universal approximation theorems guarantee that FFNNs can approximate functions arbitrarily well on compact domains, they typically fail to capture the underlying generative rules of data over open domains. Consequently, models trained on bounded ranges often cannot extrapolate beyond their training distribution (e.g., ReLU networks become asymptotically linear, and sigmoidal networks converge to constants).

Furthermore, while recent work in reasoning LLMs and recurrent architectures suggests that iterative computation improves generalization, these models remain tied to massive parameter counts and often exhibit brittleness in long-sequence extrapolation. The paper posits that the "Bitter Lesson" of AI—leveraging compute through general methods rather than top-down engineering—combined with insights from artificial life, suggests a need for substrates governed by simple, local rules where complex behaviors emerge through iteration. The core question is whether minimal, local-recursive computational substrates can achieve high global expressivity and extrapolation with significantly fewer parameters than current neural architectures.

Methodology

The Emergent Model (EM) Framework

The authors formalize Emergent Models (EMs) as a machine learning paradigm where modeling is treated not as learning a closed-form input-output map, but as the emergence of computational behaviors within simple dynamical systems. An EM is defined as a tuple $M = (S, f, H, E, D, \oplus)$ :

Latent State ( $S$ ): A configuration space (discrete or continuous) acting as memory.
Update Rule ( $f$ ): A local transition function (e.g., cellular automaton rule) iterated over the latent state.
Halting Condition ( $H$ ): A predicate determining when the iteration stops.
Interface ( $E, D, \oplus$ ): An encoder, decoder, and combining operator that link external inputs/outputs to the latent state.
Program ( $p$ ): A non-perturbed initial state. For a given input $x$ , the initial state is formed as $s_0 = p \oplus E(x)$ .

Training Approach:
Unlike gradient-based backpropagation, EMs are trained via evolutionary search (Genetic Algorithms). The search space includes:

Soft Parameters: The program state $p$ (the initial configuration). In some settings, this can be updated via state retention (Lamarckian inheritance) across macrosteps.
Hard Parameters: The update rule $f$ , halting condition $H$ , and interface parameters. These are fixed during inference but optimized during training.
The objective is to find a configuration where the evolved trajectory, decoded at halting, solves the target task.

Theoretical Foundation: Latent Universality

The paper introduces the concept of Latent Universality, a regime where the substrate ( $S, f, H$ ) and interface ( $E, D, \oplus$ ) are fixed, and only the program $p$ varies to realize any partial computable function.

Theorem 1: The authors prove that for binary string inputs/outputs, there exists an Emergent Model where a fixed substrate and interface suffice to compute any partial computable function by varying only the finite-support program state. This aligns with the concept of a Universal Turing Machine where the program and input are encoded into the initial tape configuration.

Experimental Substrates

The authors evaluate a "zoo" of minimal EM instantiations to test the framework's viability without engineering a single competitive architecture:

EM43: A 1D discrete cellular automaton with 4 states per cell and a lookup-table rule.
GoL-EM: Conway's Game of Life (fixed rule) where only the initial glider configuration (program) is optimized.
CEM1D: A continuous-valued 1D lattice with a local differential equation update rule.
CEM2D: A continuous 2D lattice with fast activation and slow memory fields, featuring trainable port locations.

Key Results

1. Arithmetic Extrapolation (EM43)

Task: Learning simple integer functions ( $x+1, 2x, x \mod 4, \text{round}(x/3)$ ) trained on $[0, 30]$ and tested on $[30, 1000]$ .
Performance: EM43 achieved 100% exact-match accuracy on all single-input tasks, including periodic functions like $x \mod 4$ and oscillatory functions like $x \cdot (1 + x \mod 2)$ .
Observation: The models exhibited "geometric grokking," where generalization was realized through stable, interpretable space-time patterns (e.g., shifting markers) that remained valid as input scales increased.
Limitation: The two-input multiplication task ( $a \cdot b$ ) failed to converge within the compute budget, suggesting current search limitations or expressivity constraints for complex interactions.
Efficiency: The successful models used only 70–110 parameters (140–220 bits of information), trained on just 30 examples.

2. Control Tasks (GoL-EM & CEM1D)

CartPole:
- GoL-EM: Achieved returns significantly above random baselines (mean ~171, best ~417) but struggled with robustness (54.5% non-halting rate) and state retention caused instability due to debris accumulation.
- CEM1D: A continuous 1D model with state retention achieved returns near the maximum (734.93), outperforming GoL-EM. However, it relied heavily on angular velocity and failed to integrate cart position effectively, eventually drifting out of bounds.
Meta-Life (Adaptation):
- Task: Agents must forage for food or avoid poison in a changing environment without prior knowledge of the phase.
- Results: Both CEM2D and a dense memory-augmented RNN (mRNN) baseline achieved high meta-scores, indicating successful online adaptation.
- Comparison: The mRNN converged faster and achieved slightly higher scores (0.82 vs 0.71). CEM2D required longer training but demonstrated that local continuous dynamics could support memory and adaptation.
- Failure Modes: Both models exhibited "rotating attractors" (spinning in place) when resources were scarce, a failure attributed to the training objective's diminishing returns and the difficulty of recovering from a stuck state.

Key Contributions

Formalization of Emergent Models: A rigorous framework treating dynamical systems as ML models where the "program" is an initial state in a latent space, decoupled from the fixed update rule and interface.
Proof of Latent Universality: A theoretical proof demonstrating that a fixed substrate and interface can compute any computable function solely by varying the initial program state, establishing a necessary condition for global expressivity.
Empirical Validation of Minimal Substrates: Demonstration that local-recursive computation at the scale of tens to hundreds of parameters can achieve exact extrapolation on arithmetic tasks and support control/adaptation behaviors, challenging the notion that massive parameter counts are required for generalization.
Geometric Grokking: Identification of a phenomenon where generalization manifests as stable, interpretable geometric patterns in the space-time evolution of the substrate, rather than opaque weight adjustments.
The $q$ -Ratio Conjecture: A hypothesis that the ratio of parameters to temporal depth ( $q = N_{params} / \bar{T}$ ) governs the trade-off between generalization and trainability. Low $q$ (simple rules, deep iteration) favors algorithmic compression and extrapolation but creates irregular search landscapes; high $q$ (large networks, shallow iteration) favors memorization.

Significance and Claims

The paper positions itself as foundational work rather than a proposal for a competitive architecture. Its primary significance lies in:

Widening the Design Space: Moving beyond differentiable feed-forward maps to explore simple, open-ended physical-like substrates.
Reframing Intelligence: Suggesting that intelligence may emerge from the interaction of simple local rules and initial conditions (an "artificial physics") rather than the explicit engineering of neural architectures.
Modest Claims: The authors explicitly state that their models are not yet competitive with mature task-specific methods. They acknowledge limitations in training efficiency, search scalability (especially for complex tasks like multiplication), and robustness in control settings.
Future Directions: The work calls for the realization of the latent-universal regime in practice and the development of optimization methods specifically tailored for the non-convex, non-local landscapes of low- $q$ recursive models.

The paper concludes that while the current state of local-recursive modeling is comparable to deep learning two decades ago, the ability of tiny substrates to extrapolate and adapt suggests a promising, under-explored path toward efficient and generalizable machine learning.