Technical Summary: Understanding Sample Efficiency in Predictive Coding

Problem Statement

Predictive Coding (PC) is a biologically plausible framework for cortical learning that minimizes prediction errors through local weight updates. While recent empirical studies suggest PC offers superior sample efficiency and reduced catastrophic interference compared to Backpropagation (BP), particularly in online and continual learning regimes, a principled theoretical explanation for these advantages has remained elusive. Existing evidence is largely confined to small-scale models, and the specific mechanisms by which PC avoids the "interference" that plagues BP—where updates correct errors in one direction while inadvertently introducing errors in directions the model already predicts accurately—are not analytically understood. This paper seeks to provide a rigorous analytical account of when and why PC outperforms BP, specifically by quantifying learning efficiency through the metric of "target alignment."

Methodology

The authors employ Deep Linear Networks (DLNs) as a tractable analytical framework to derive closed-form expressions for learning dynamics. DLNs are chosen because, despite being linear, they exhibit non-linear learning dynamics useful for understanding deeper architectures.

1. Theoretical Derivation:

   • The authors derive the continuous-time weight dynamics for both BP and PC.
   • For BP, they show that the change in prediction ($\partial \hat{y}/\partial t$) is a sum of terms where the output residual $r$ is pre-multiplied by weight-dependent matrices ($W_{L:l+1}W_{L:l+1}^\top$). These matrices distort the direction of $r$, introducing interference.
   • For PC, they derive the weight dynamics based on the minimization of a specific energy function involving a scaling matrix $S$. They demonstrate that the term $S^{-1}$ in the PC update acts as a preconditioner that cancels out the weight-dependent distortion terms found in BP, thereby attenuating interference.

2. Metric Definition:

   • Target Alignment: Defined as the cosine similarity between the output prediction error ($r$) and the change in the model's prediction ($\Delta \hat{y}$) induced by a weight update. Perfect alignment (value of 1) implies the update moves the prediction directly toward the target without interference.

3. Proposed Modifications:

   • Adaptive Learning Rates: The authors propose layer-specific learning rates ($\alpha_l$) scaled by the inverse of the dot product of pre- and post-inference activities to guarantee optimal alignment for single-sample learning.
   • Weight Update Rescaling: For batch training, they propose multiplying weight updates by a decorrelation matrix $A_l$ (the inverse of the batch-averaged activity correlation matrix) to remove inter-sample interference.

4. Empirical Validation:

   • Experiments were conducted on synthetic regression tasks using randomly initialized DLNs with varying depths (1 to 8 layers) and widths.
   • Both linear and non-linear (ReLU) autoencoders trained on MNIST were used to test the generalizability of findings.
   • Comparisons were made between standard BP, standard PC, and their "rescaled" counterparts (with adaptive rates or decorrelation matrices).

Key Contributions

• Analytical Explanation of Interference: The paper derives that BP introduces interference due to weight-dependent distortion of the error signal, whereas PC inherently attenuates this interference via the scaling matrix $S$. This explains why PC exhibits higher target alignment.
• Conditions for Superiority: The authors identify that PC's advantage is most pronounced in deep, narrow, and pre-trained networks where weight matrices are poorly conditioned (i.e., far from orthogonal), leading to significant interference in BP.
• Guaranteed Optimal Alignment:
  • Theorem 1: Proves that PC with layer-specific learning rates (scaled by activity norms) achieves perfect target alignment ($TA=1$) for single-sample learning, independent of architecture or weight conditioning.
  • Theorem 2: Proves that PC with weight update rescaling (using a decorrelation matrix) achieves perfect target alignment for batch learning, provided the batch size is smaller than the smallest layer width.
• Sample Efficiency: The paper demonstrates that higher target alignment translates to more sample-efficient learning over full training trajectories, allowing PC to reach lower error rates with fewer steps compared to BP.

Results

• Target Alignment: In randomly initialized networks, PC consistently achieves higher target alignment than BP. This gap widens significantly as network depth increases and when using Kaiming initialization (which causes activity norm decay), whereas Norm-preserving initialization maintains high alignment for both but PC still retains an edge in deep, narrow settings.
• Training Trajectories: In "whole training" experiments, PC converges faster and reaches lower final errors than BP. This benefit is most evident in deep networks (8 hidden layers) and narrow architectures.
• Rescaling Effects:
  • Online Learning (Batch Size = 1): Rescaling learning rates provides minimal benefit at the start of training because estimating the Fisher Information Matrix (or activity correlations) with a single sample diminishes geometric advantages. However, PC with rescaling eventually outperforms standard BP.
  • Batch Learning: Rescaling weight updates significantly improves performance. Rescaled PC reaches minimum error with far fewer training steps than standard PC, standard BP, or rescaled BP.
• Non-linear Models: Preliminary experiments on non-linear autoencoders trained on MNIST show that the benefits of PC and the efficacy of rescaling (especially with larger batch sizes) persist, suggesting the theoretical insights extend beyond linear models.
• ResNets: The analysis extends to linear Residual Networks, showing that while skip connections improve baseline conditioning, PC still outperforms BP, and layer-specific scaling guarantees perfect alignment.

Significance and Claims

The paper claims to provide a mechanistic understanding of the higher learning efficiency observed in PC over BP in previous works. By linking PC's performance to the concept of target alignment and its proximity to natural gradient descent, the authors argue that PC's ability to minimize interference is the primary driver of its sample efficiency.

The work suggests that to maximize PC's effectiveness, one should:

1. Use weight initializations that preserve activity norms across layers.
2. Employ layer-specific learning rates or weight update rescaling to guarantee interference-free learning.

The authors remain modest regarding the biological plausibility of their proposed rescaling mechanisms, noting that while they serve as optimal performance baselines, biological systems may employ other strategies (like activity normalization) to achieve similar goals. The paper concludes that these findings can guide the parametrization of PC for effective learning and offer a theoretical bridge between cortical learning theories and modern deep learning optimization.