🧬 biology

Self-orthogonalizing attractor neural networks emerging from the free energy principle

This paper demonstrates that self-orthogonalizing attractor neural networks naturally emerge from the free energy principle applied to random dynamical systems, yielding biologically plausible, multi-level Bayesian active inference dynamics that optimize predictive accuracy and model complexity without requiring explicitly imposed learning rules.

Original authors: Tamas Spisak, Karl Friston

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tamas Spisak, Karl Friston

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine your brain (or a smart computer) as a vast, bustling city. In this city, there are millions of tiny workers (neurons) constantly talking to each other. Usually, we think these workers need a boss to tell them what to do or a teacher to show them how to learn. But this paper proposes something different: the city organizes itself.

The authors, Tamas Spisak and Karl Friston, use a big idea called the Free Energy Principle to explain how this self-organization happens. Think of "Free Energy" not as a battery, but as a measure of surprise. If the city is constantly surprised by what's happening outside (like a sudden storm or a new building), it's in a state of high "free energy." To survive and stay stable, the city wants to minimize this surprise.

Here is how the paper explains the magic of this self-organizing city, broken down into simple concepts:

1. The "Self-Organizing" City

Imagine a flock of birds. No single bird is the leader telling the others where to fly. Instead, each bird just follows simple local rules (stay close to neighbors, don't crash). Yet, the whole flock moves as one beautiful, coordinated unit.

This paper says the brain works the same way. It doesn't need a central programmer to set up its learning rules. Instead, every tiny part of the network tries to reduce its own "surprise" by predicting what its neighbors are doing. When everyone does this locally, a global pattern emerges. These patterns are called Attractors.

The Analogy: Think of an attractor like a valley in a hilly landscape. If you roll a ball (a thought or a memory) anywhere on the hill, it will naturally roll down into the nearest valley and stay there. The brain creates these "valleys" for things it recognizes, like a face or a word.

2. The Magic Trick: "Self-Orthogonalizing"

The most exciting discovery in this paper is how these "valleys" (attractors) arrange themselves.

Usually, if you teach a computer to recognize two similar things (like the letters "O" and "Q"), it might get confused because the patterns look too much alike. But this paper shows that the brain's self-organizing system naturally pushes these patterns apart until they are orthogonal.

The Analogy: Imagine you have a set of flashlights in a dark room.

Bad Organization: You point all the flashlights at the same spot. The beams overlap, and it's hard to tell which light is which.
Orthogonal Organization: The paper says the brain automatically angles the flashlights so they point in completely different directions (like the X, Y, and Z axes). They don't overlap.

By doing this, the brain creates a "clean" map of the world. It learns to store memories in a way that they don't muddy each other. This makes it incredibly good at generalizing—if you see a slightly blurry "O," the brain knows exactly which "valley" to roll into because the "O" valley is perfectly distinct from the "Q" valley.

3. Learning Without a Teacher

How does the city learn these patterns? The paper describes a simple, local rule that happens automatically:

Hebbian Learning (The "Yes" Rule): If two workers talk to each other often, they get closer.
Anti-Hebbian Learning (The "No" Rule): If the workers are already explaining what's happening, they stop reinforcing each other to avoid redundancy.

The Analogy: Imagine a group of friends trying to describe a movie.

If two friends say the same thing, they stop repeating it (to save energy).
If one friend says something new that the others didn't predict, they pay attention to it.
Over time, the group creates a perfect, non-redundant summary of the movie where every friend contributes a unique piece of the puzzle. This is how the network learns to be efficient and avoid "catastrophic forgetting" (forgetting old things when learning new ones) because the "valleys" are so well-organized they don't crash into each other.

4. Dreaming and Sequences

The paper also shows what happens when the data comes in a specific order (like a story or a sequence of numbers).

Random Data: The network builds stable, static "valleys" (like recognizing a face).
Sequential Data: The network builds paths between the valleys.

The Analogy: If you teach the network the sequence "1, 2, 3," it doesn't just store three separate valleys. It builds a slide that naturally flows from the "1" valley to the "2" valley, and then to the "3" valley. Even if you turn off the lights (remove the input), the network can "dream" or "replay" this sequence on its own, rolling from 1 to 2 to 3 automatically. This is how the brain might learn to walk, speak, or remember a story in order.

5. Why This Matters for AI and Brains

The paper claims this isn't just a cool math trick; it's a fundamental way nature works.

For Brains: It explains how the brain can be so robust. Even if you have noisy, blurry input (like seeing a face in the fog), the brain's "valleys" are so well-organized that it can still guess the right answer. It also explains why we don't forget old memories when we learn new ones—the "valleys" are spaced out perfectly.
For AI: Current AI often needs massive amounts of data and specific instructions to learn. This paper suggests that if we build AI that follows these self-organizing rules, it could learn more efficiently, handle noise better, and remember things longer without needing a human to constantly retrain it.

Summary

In short, this paper argues that you don't need to program a brain or a smart AI with complex rules. If you just give it a simple goal—to minimize surprise—it will naturally organize itself into a highly efficient system. It will create distinct, non-overlapping memories (orthogonal attractors), learn sequences like a story, and even "dream" to reinforce what it has learned, all by itself.

Technical Summary: Self-Orthogonalizing Attractor Neural Networks Emerging from the Free Energy Principle

Problem Statement
Attractor dynamics are fundamental to complex systems, including the brain, serving as mechanisms for memory, decision-making, and motor control. However, a foundational gap exists in understanding how such self-organizing attractor networks emerge from first principles without explicitly imposed learning or inference rules. Traditional models often rely on ad hoc learning rules (e.g., Hebbian learning, contrastive divergence) that lack a unified derivation from the thermodynamic or inferential constraints governing biological systems. The paper addresses the need to derive adaptive self-organization—where a network not only operates as an attractor system but also "builds itself" through local interactions—from the Free Energy Principle (FEP).

Methodology
The authors formalize the emergence of attractor networks by applying the Free Energy Principle to a "universal partitioning" of random dynamical systems, specifically utilizing the concept of deep particular partitions.

Theoretical Framework:
- Deep Particular Partitions: The system is decomposed into a hierarchy of "subparticles" (micro-scale units), each possessing internal states ( $\sigma_i$ ), sensory states ( $s_{ij}$ ), and active states ( $a_{ij}$ ). These subparticles are separated from external states by a Markov blanket, ensuring conditional independence ( $\sigma_i \perp \sigma_j | s_{ij}, a_{ij}$ ).
- Generative Model: The internal states are modeled as continuous Bernoulli variables (truncated exponential distributions) with a priori biases. The coupling between subparticles is defined by a weight matrix $J$ .
- Variational Free Energy (VFE) Minimization: The core derivation assumes that to persist, the system must minimize variational free energy. This minimization is performed locally at each node (subparticle) regarding its internal states and couplings.
Derivation of Dynamics:
- Inference: Minimizing VFE with respect to internal states yields a stochastic update rule. The expected value of the state follows a Langevin function (a sigmoid activation), driven by the sum of a prior bias and weighted inputs from neighbors. This results in a continuous-state stochastic Hopfield network behavior.
- Learning: Minimizing VFE with respect to the couplings (weights) yields a local learning rule. This rule takes the form of a contrastive update: $\Delta J_{ij} \propto \sigma_i \sigma_j - \hat{\sigma}_i \sigma_j$ . The first term is Hebbian (correlation-based potentiation), and the second is anti-Hebbian (predictive error-based depression). Crucially, this rule does not require separate "clamped" and "free-running" phases (unlike Contrastive Divergence) but uses instantaneous correlations.
Simulation Protocol:
- The authors implemented the derived rules in Python and JAX.
- Simulations 1–2: Trained on correlated patterns (images of handwritten digits) to test orthogonalization and generalization under varying precision (inverse temperature) and evidence strength.
- Simulation 3: Trained on sequential data to test the emergence of asymmetric couplings and sequence dynamics.
- Simulation 4: Tested resistance to catastrophic forgetting via "free-running" (zero bias) spontaneous activity.
- Simulation 5: Evaluated scalability and memory capacity up to 50,000 nodes.
- Simulation 6: Applied to the Olivetti faces dataset (4096 nodes) to test performance on naturalistic, high-dimensional data.

Key Contributions

Derivation of Self-Organizing Rules: The paper demonstrates that attractor networks, including their learning and inference dynamics, emerge naturally from the FEP applied to deep particular partitions, obviating the need for externally imposed learning rules.
Emergent Self-Orthogonalization: A central finding is that the derived learning rule inherently drives the network toward approximately orthogonal attractor representations. By minimizing model complexity (redundancy) alongside predictive accuracy, the network penalizes overlapping attractors. This results in a set of attractors that span the input subspace efficiently, resembling "projector neural networks" (Kanter-Sompolinsky) which offer maximal memory capacity.
Unified Inference and Learning: The framework unifies inference and learning as two facets of VFE minimization. Inference updates states to match evidence, while learning updates couplings to minimize future surprise, all occurring simultaneously in a continuous-learning regime.
Generalization of Boltzmann Machines: The derived networks generalize conventional Boltzmann machines.
- With random data, they converge to symmetric couplings (Boltzmann-like).
- With sequential data, they develop antisymmetric couplings, creating non-equilibrium steady-state (NESS) dynamics with solenoidal (rotational) flows. This allows the network to learn and recall temporal sequences (heteroclinic cycles) without altering the underlying energy landscape's stationary distribution.
Robustness to Catastrophic Forgetting: The stochastic nature of the dynamics allows the network to spontaneously "replay" attractors during free-running phases. This activity reinforces existing attractors, preventing the loss of learned representations when new data is introduced, effectively mitigating catastrophic forgetting.

Results

Orthogonal Basis Formation: Simulations confirmed that when trained on correlated inputs, the network's attractors become approximately orthogonal (e.g., reducing correlation from $r=0.77$ to $r=-0.19$ ). This orthogonalization was most effective in a "balanced regime" of intermediate precision.
Generalization: The orthogonalized attractors enabled the network to generalize to unseen data (one-shot learning) by combining quasi-orthogonal basis states, outperforming standard Hopfield networks in reconstruction tasks from noisy inputs.
Sequence Learning: When presented with sequential data, the coupling matrix developed a significant antisymmetric component. This allowed the network to spontaneously generate the learned sequence (1→2→3→...) in the absence of external input, driven by probability currents along iso-energy contours.
Scalability and Capacity: The network demonstrated memory capacities substantially exceeding the classical Hopfield bound ( $\sim 0.14N$ ), approaching the projector-network limit ( $K \approx N$ ) due to emergent orthogonalization. Runtime scaling was confirmed to be $O(N^2)$ , consistent with the analytical complexity.
Face Recognition: On the Olivetti faces dataset, the network successfully reconstructed identities from heavily corrupted inputs by integrating noisy sensory evidence with learned attractor-based priors, demonstrating robust Bayesian inference.

Significance and Claims

The paper claims to provide a unifying theory for self-organizing attractor networks that bridges neuroscience and artificial intelligence.

For Neuroscience: The findings offer a normative explanation for why biological neural networks might exhibit Hebbian/anti-Hebbian plasticity, spontaneous activity, and orthogonal representations. It suggests that the brain's ability to perform active inference and maintain stable memories without explicit programming arises from the fundamental imperative to minimize free energy. The emergence of orthogonal attractors is proposed as a signature of FEP-driven systems, potentially detectable in large-scale brain data (e.g., resting-state networks).
For Artificial Intelligence: The framework offers a principled alternative to backpropagation and contrastive divergence. It enables continual learning and resistance to catastrophic forgetting through spontaneous dynamics. The derived learning rule is computationally efficient (no free-running phase) and naturally supports sequence learning and generalization.
Theoretical Implications: The work posits that the separation of the coupling matrix into symmetric (perception/energy minimization) and antisymmetric (action/planning/solenoidal flow) components may reflect a deeper formal correspondence with the dual objectives of the FEP.

The authors maintain a modest stance regarding limitations, acknowledging that the $O(N^2)$ memory footprint is prohibitive for very large networks without sparse approximations, and that the current simulations are limited to single-layer architectures. They suggest that future work should explore hierarchical deep partitions and implementation on thermodynamic or memristive hardware to leverage the local, stochastic nature of the updates.