🤖 machine learning

Topological Simplification in Predictive Coding Networks

This paper employs persistent homology to demonstrate that predictive coding networks exhibit a distinct topological simplification pattern where later collapse of connected components correlates with lower reconstruction error and superior performance compared to standard MLPs, highlighting the unique role of bidirectional dynamics in shaping the compression-reconstruction tradeoff.

Original authors: Adam Shaw, Jiayu Li, Michael Sperling, Michael Kim, Alvin Jin

Published 2026-08-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Adam Shaw, Jiayu Li, Michael Sperling, Michael Kim, Alvin Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a giant, complex machine turns a messy pile of raw ingredients into a perfect, organized meal. In the world of artificial intelligence, this machine is a "neural network," a computer program designed to learn from data. Scientists have long been fascinated by how these networks "see" the world. They don't just look at pixels or numbers; they transform the data into a hidden shape, a kind of invisible landscape where similar things are close together and different things are far apart.

To study these invisible landscapes, researchers use a branch of math called "topology." Think of topology as the study of shapes that can be stretched or squished but not torn. In this language, a solid ball is the same shape as a cup because you can mold one into the other, but a ball is different from a donut because you can't make a hole without tearing the dough. Another key idea is "predictive coding." Imagine a detective who constantly guesses what will happen next, then checks their guess against reality. If the guess is wrong, they adjust their theory. This is how some advanced AI models work: they don't just look at data once; they constantly predict, check, and refine, trying to understand the data so well they can even recreate it from scratch. The big question is: as these detective-AIs get smarter, how does the shape of their understanding change? Do they flatten the world into a simple line, or do they keep the complex curves intact?

This is exactly what a team of researchers from the University of Southern California set out to investigate in their 2026 paper, "Topological Simplification in Predictive Coding Networks." They wanted to see how these "detective" AI networks handle the messy shapes of data as they process it layer by layer. They trained these networks on two types of puzzles: a synthetic game made of floating disks and holes (designed to be topologically tricky) and the famous MNIST dataset of handwritten numbers.

The researchers discovered that these networks act like a series of filters that gradually smooth out the rough edges of the data. As the data moves deeper into the network, the complex shapes get simpler. Specifically, the network "collapses" separate groups of data points into single, unified blobs. This is called "topological simplification." However, the timing of this collapse is everything. The team found that smaller networks, which have fewer "brain cells" (neurons), rush to simplify the data very early in the process. It's like a student who quickly memorizes a few key facts to pass a test but forgets all the details. In contrast, larger, more powerful networks hold onto the complex shapes for much longer, delaying the simplification until the very end.

There is a catch, though. The researchers found a strong link between when the network simplifies the data and how well it can recreate the original image. Networks that simplified too early lost important information, making their "reconstructions" (their attempts to draw the original image back from memory) blurry and poor. The networks that waited longer to simplify the shapes were much better at rebuilding the data. In fact, they found a clear pattern: the later the network simplified the topology, the better it was at reconstruction.

Perhaps the most surprising finding was how these "detective" networks compared to standard, one-way AI networks (called MLPs). Even when given the exact same size and training, the detective networks (Predictive Coding Networks) held onto the complex shapes of the data for about 3.6 layers longer than the standard networks before finally simplifying them. This suggests that the constant back-and-forth "guessing and checking" process of predictive coding forces the network to keep more of the world's complexity alive for longer, preserving the ability to reconstruct the original scene.

In short, the paper suggests that there is a trade-off. If you want a small, fast network, it will simplify the world quickly, but it might lose the ability to recreate the details. If you want a network that can perfectly reconstruct what it sees, it needs the capacity to keep the world's complex shapes intact for as long as possible. The study uses a mathematical tool called "persistent homology" to measure these changes, acting like a ruler that counts how many holes and islands exist in the data at every step. The results show that while all deep networks eventually flatten the world to make sense of it, the "detective" style of learning keeps the map detailed for much longer, offering a better balance between understanding and remembering.

Technical Summary: Topological Simplification in Predictive Coding Networks

Problem Statement
While persistent homology has been extensively used to analyze topological simplification in feedforward neural networks (where topological features like connected components and loops are progressively collapsed to create separable representations), it remains unclear how this phenomenon behaves in architectures that must simultaneously support inference and reconstruction. Predictive Coding Networks (PCNs) are hierarchical, bidirectional models that perform inference through iterative minimization of a global energy functional. Unlike feedforward classifiers, PCNs must preserve sufficient topological information to support approximate inversion (reconstruction). This paper investigates whether PCNs exhibit topological simplification similar to feedforward networks and how architectural constraints influence the tradeoff between topological collapse and reconstruction fidelity.

Methodology
The authors employ persistent homology to track the evolution of representation topology across layers in PCNs trained on two datasets:

Synthetic Dataset: A topologically complex 2D manifold consisting of nine disjoint disks ( $M_a$ , $\beta_0=9$ ) embedded within a region containing nine holes ( $M_b$ , $\beta_1=9$ ). This dataset requires significant topological transformation for classification.
Real-World Dataset: MNIST, treated as point clouds in $\mathbb{R}^{784}$ .

The study utilizes the JAX-based PCX library to train PCNs with varying hidden-layer widths, architectural configurations (uniform, increasing/decreasing, bottleneck), and activation functions (ReLU, Leaky ReLU, tanh). Models are trained to high accuracy (≥99.9% on synthetic, ≥95% on MNIST).

Two primary metrics are introduced to quantify the findings:

Center of Mass of Topological Simplification (COM): A scalar metric quantifying the "timing" of topological collapse. It calculates the expected depth at which Betti numbers (specifically $\beta_0$ and $\beta_1$ ) decay, treating simplification events as irreversible. A lower COM indicates earlier collapse.
Mean Reconstruction Distance (MRD): A metric quantifying reconstruction fidelity. It measures the average Euclidean distance between inverted outputs and their nearest same-class neighbors in the training set, capturing whether inversions lie near the correct data manifold.

The analysis restricts persistent homology computations to single classes to avoid conflating class topologies.

Key Results

Model Size and Simplification Timing: There is a strong positive correlation between model size (sum of hidden-layer widths) and the timing of topological simplification (COM). Smaller PCNs collapse connected components significantly earlier than larger models. This relationship holds across activation functions, with Spearman correlations ranging from $\rho \in [0.72, 0.79]$ . ReLU networks show the highest sensitivity to capacity, where added width delays collapse more substantially than in Leaky ReLU or tanh networks.
Simplification-Reconstruction Tradeoff: A strong negative correlation ( $\rho = -0.58$ ) exists between the timing of simplification (COM) and reconstruction error (MRD). Architectures that simplify topology later (higher COM) achieve better reconstruction quality. Early collapse corresponds to the loss of structural information necessary for inversion, leading to poorer generative performance.
PCNs vs. Feedforward Networks (MLPs): When compared to matched Multilayer Perceptrons (MLPs) trained to the same accuracy, PCNs consistently delay topological simplification. On average, PCNs collapse connected components 3.6 layers later than their MLP counterparts. This suggests that the recurrent, bidirectional dynamics of predictive coding impose constraints that preserve topological structure longer to facilitate reconstruction.
Consistency Across Data: These trends were observed in both the synthetic manifold and the high-dimensional MNIST dataset, despite the latter lacking known ground-truth topology.

Significance and Claims
The paper claims that persistent homology offers a useful quantitative lens for analyzing the compression–reconstruction tradeoff in PCNs. The findings extend topological analyses of learned representations beyond feedforward networks, highlighting that:

Topological simplification is not uniform; it is governed by model capacity and the specific dynamics of the architecture.
The recurrent nature of predictive coding inference shapes when the tradeoff between compression (simplification) and reconstruction is resolved, forcing the network to preserve topological features deeper into the network compared to standard feedforward models.
There is a direct link between the timing of topological collapse and the ability of the network to invert its representations, suggesting that excessive early simplification compromises the generative capabilities of the model.

The authors maintain a modest scope, noting that while the trends hold for the tested architectures and datasets, future work is needed to test these hypotheses across broader datasets and to develop more refined, reconstruction-aware topological metrics for real-world semantic fidelity.

Technical Summary: Topological Simplification in Predictive Coding Networks

More like this