🤖 machine learning

On the Infinite Width and Depth Limits of Predictive Coding Networks

This paper investigates the infinite width and depth limits of Predictive Coding Networks, deriving stable parameterizations that prevent training explosions and demonstrating that Predictive Coding gradients converge to Backpropagation gradients in wide networks, thereby offering a biologically plausible local learning mechanism for deep architectures.

Original authors: Francesco Innocenti, El Mehdi Achour, Rafal Bogacz

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Francesco Innocenti, El Mehdi Achour, Rafal Bogacz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a giant, multi-layered team of robots how to solve a puzzle. In the world of artificial intelligence, the standard way to do this is called "backpropagation." It's like a strict manager who stands at the very end of the assembly line, looks at the final mistake, and then runs all the way back to the beginning, shouting specific instructions to every single robot about exactly how they messed up. It works incredibly well for computers, but it is a method that differs from how nature operates. Real brains don't have a central manager shouting instructions backward through time; instead, every neuron only knows what its immediate neighbors are doing.

Enter "Predictive Coding," a theory suggesting that brains learn more like a group of friends trying to guess the weather. Each person (neuron) makes a prediction based on what their neighbors say, feels a little "error" if their guess is wrong, and adjusts their own activity to fit in better. Once everyone stops arguing and reaches a calm agreement (equilibrium), they all tweak their connections slightly to do better next time. This method is biologically realistic because everyone only talks to their neighbors, but for a long time, scientists weren't sure if it could handle the massive, deep networks needed for modern AI. Could this local, chatty method actually scale up to the size of a supercomputer, or would it fall apart?

This paper dives into that exact question by looking at what happens when you make these networks infinitely wide (adding tons of neurons side-by-side) and infinitely deep (adding tons of layers on top of each other). The researchers, Francesco Innocenti, El Mehdi Achour, and Rafal Bogacz, wanted to know: Can we tune the "knobs" of these predictive coding networks so they stay stable and learn effectively, just like the standard backpropagation networks we use today?

Their investigation reveals a surprising twist. They found that for predictive coding to work on a massive scale, it must use the exact same settings and scaling rules as the standard backpropagation method. If you try to use the "standard" settings that most people use in software like PyTorch, the predictive coding network's output will explode into chaos as it gets wider, much like a microphone left too close to a speaker. However, if you switch to a specific, mathematically precise set of rules (known as "mean-field" or "maximal update" parameterizations), the network stabilizes.

Under these stable conditions, the paper shows that as the network gets incredibly wide, the way predictive coding learns becomes almost indistinguishable from backpropagation. In fact, for networks that are much wider than they are deep (think of a wide, shallow pancake rather than a tall, thin tower), the "local" updates of predictive coding align perfectly with the "global" updates of backpropagation. The authors proved this mathematically for simple linear networks and confirmed it with experiments on complex, non-linear models like Convolutional Neural Networks (CNNs) and Transformers.

The study also rules out a few exciting possibilities. It suggests that the "fast" learning speed some researchers saw in deeper-than-wide networks using older settings was actually an illusion caused by instability; those settings break down as the network grows. Furthermore, while the paper suggests that the brain might use a local learning rule similar to predictive coding to achieve the power of backpropagation, it notes that this only works efficiently if the brain is "much wider than it is deep"—a hypothesis that aligns with current biological understanding of the cortex.

Ultimately, this work doesn't just say "predictive coding works"; it draws a strict boundary around how it can work. It tells us that to scale this biologically plausible method to the size of modern AI, we can't just tweak it randomly; we have to adopt the same rigorous mathematical scaling that makes standard AI work. This is a crucial step forward, suggesting that the brain might indeed be using a local, energy-minimizing dance to solve the same complex problems that our global, backpropagation-trained computers solve today.

Technical Summary: On the Infinite Width and Depth Limits of Predictive Coding Networks

Problem Statement

Predictive Coding (PC) is a biologically plausible alternative to standard backpropagation (BP) where neurons minimize local prediction errors (an energy function) before updating weights. While recent reparameterizations have improved the training stability of deep PC networks (PCNs), the theoretical basis for scaling these methods to large widths and depths remains unclear. Specifically, it is unknown whether PC can be scaled to match the stability and feature-learning capabilities of BP in the infinite-width and infinite-depth limits, or if standard PCNs suffer from instability (e.g., exploding outputs) as model size increases. Furthermore, the conditions under which PC gradients converge to BP gradients in these limits require rigorous characterization.

Methodology

The authors conduct a theoretical and empirical analysis of PCNs in the joint limits of infinite width ( $N \to \infty$ ) and infinite depth ( $L \to \infty$ ).

Theoretical Framework:
- General Parameterization: The study adopts a general "abcd" parameterization scheme for linear Multi-Layer Perceptrons (MLPs) and Residual Networks, defining scaling exponents for preactivations ( $a_\ell$ ), weight initialization variance ( $b_\ell$ ), learning rate ( $c$ ), and network output ( $d$ ).
- Equilibrated Energy: For linear networks, the authors utilize the "equilibrated energy" ( $F^*$ ), which is the PC energy minimized with respect to activities. They show that for linear networks, $F^*$ is a rescaled version of the standard Mean Squared Error (MSE) loss: $F^*(\theta) = L(\theta) / s(\theta)$ , where $s(\theta)$ is a weight-dependent rescaling factor.
- Desiderata: The analysis seeks parameterizations that satisfy three main criteria (adapted from BP literature):
  1. Stability: Layer preactivations and network predictions must remain bounded (neither vanish nor explode) as $N \to \infty$ .
  2. Feature Learning: Features must evolve non-trivially (non-lazy learning) during training.
  3. Depth Stability (for ResNets): Hidden activations must remain stable as $L \to \infty$ , and each residual block must contribute $\Theta(1/L)$ to feature evolution.
Derivation:
- The authors derive the specific scaling exponents required to satisfy the above desiderata for PCNs.
- They analyze the asymptotic behavior of the rescaling factor $s(\theta)$ and the gradients of the equilibrated energy.
- They prove that under these stable parameterizations, the PC gradients converge to BP gradients as the width grows, with the error term depending on the depth-to-width ratio ( $L/N$ ).
Empirical Validation:
- Experiments were conducted on linear and nonlinear architectures, including MLPs, Convolutional Neural Networks (CNNs), and Transformers.
- Models were trained on datasets ranging from toy tasks to CIFAR-10, Fashion-MNIST, and ImageNet-1k.
- The study measured the cosine similarity between PC gradients (computed at activity equilibrium) and BP gradients across varying widths and depths.

Key Contributions

Equivalence of Stable Parameterizations:
The paper proves that the set of parameterizations that ensures both stability and non-trivial feature learning for PCNs is identical to that of BP. Specifically, for linear MLPs and Residual Networks, the "Mean-Field" or "Maximal Update" ( $\mu$ P) parameterization is the unique solution that satisfies the stability and feature-learning desiderata for PC. This implies that standard PCNs (using standard parameterizations like PyTorch's defaults) are inherently unstable at large widths, causing outputs to explode.
Convergence to Backpropagation:
The authors demonstrate that under these stable parameterizations, the gradients computed by PC converge to the gradients of BP as the network width becomes much larger than its depth ( $L/N \to 0$ ).
- For linear MLPs at fixed depth, the relative error between PC and BP gradients scales as $O(N^{-1})$ .
- For linear Residual Networks in the joint limit, the relative error scales as $O(L/N)$ .
- This convergence holds for both the objective functions and the gradients, implying that in the wide limit, PC effectively implements BP.
Instability of "Fast Saddle-to-Saddle" Regime:
The paper identifies that a previously observed regime where PC exhibits faster "saddle-to-saddle" dynamics than BP (in deep, narrow networks) relies on the Standard Parameterization (SP). The authors show that this regime is unstable with respect to both width and depth, violating the stability desiderata. Thus, the speed-up comes at the cost of numerical instability in large-scale models.
Empirical Generalization:
Despite the theoretical analysis being derived for linear networks, experiments show that the high gradient alignment between PC and BP persists in nonlinear architectures (CNNs, Transformers) trained with different optimizers (Adam) and loss functions, provided the networks are much wider than deep and activity equilibrium is reached.

Results

Theoretical: The derivation confirms that the output scaling exponent $d=1/2$ is critical for non-lazy feature learning in PC, just as in BP. The rescaling factor $s(\theta)$ approaches 1 as $N \to \infty$ (for MLPs) or as $L/N \to 0$ (for ResNets), causing the PC objective to converge to the MSE loss.
Experimental:
- Gradient Alignment: Cosine similarity between PC and BP gradients approaches 1.0 as width increases, particularly for ResNets and Transformers where $L/N$ is small.
- Nonlinear Models: High alignment was observed in 32-layer Residual MLPs, 10-layer CNNs, and 12-block Transformers.
- Failure of Standard Parameterization: Models using standard parameterizations (SP) showed exploding outputs and poor gradient alignment as width increased, confirming the theoretical instability.

Significance and Claims

The paper claims to provide the first principled constraints on the parameterizations that are scalable with PC. Its primary significance lies in two areas:

Constraining Scalability: It establishes that for PC to be scalable to large models, it must adopt the same parameterization rules as BP (specifically $\mu$ P). This suggests that "heuristic" reparameterizations used in recent deep PC work must align with these theoretical limits to be viable.
Biological Plausibility of BP: The results suggest a mechanism by which the brain (arguably a network that is much wider than it is deep) could implement the global credit assignment of backpropagation using only local learning rules. If the brain operates in a regime where $L/N \to 0$ , PC could theoretically converge to BP, allowing for efficient learning without non-local information propagation.

The authors remain modest regarding the limitations, noting that the theoretical proofs rely on linear networks and exact activity convergence. They acknowledge that the computational cost of reaching activity equilibrium in nonlinear networks remains a practical bottleneck, though they suggest that analog hardware implementations might eventually address this. They do not claim that PC is superior to BP in all regimes, but rather that in the scalable, wide limit, the two algorithms become mathematically equivalent under the correct parameterization.