Closed-form predictive coding via hierarchical Gaussian filters

Technical Summary: Closed-form Predictive Coding via Hierarchical Gaussian Filters

Problem Statement

Predictive coding (PC) offers a biologically grounded alternative to backpropagation (BP) for training artificial neural networks, characterized by local computation, Hebbian-like updates, and the absence of a global error signal. However, scaling PC to modern deep architectures has been hindered by two critical limitations:

1. Computational Inefficiency: Current PC networks rely on iterative gradient descent to relax activations toward equilibrium (typically requiring $T \approx 32\text{--}128$ steps per sample), making them significantly slower than BP on sequential hardware.
2. Performance Degradation: As network depth increases, performance degrades sharply.

The authors trace both issues to a specific simplification in existing deep PC implementations: the fixation of the precision matrix to the identity. This simplification discards the precision-weighted prediction errors required by the variational derivation of PC. Consequently, these networks lose the property that makes PC distinct in the linear case—formal equivalence to natural-gradient descent—and fail to adaptively scale error signals or manage uncertainty across layers.

Methodology

The paper proposes reformulating predictive coding networks as generalized Hierarchical Gaussian Filters (gHGF). This approach replaces iterative state inference with closed-form, one-shot variational updates.

Core Architectural Shifts:

• From Fixed to Dynamic Precision: Unlike standard PC networks that treat precision as a fixed hyperparameter (identity matrix), the gHGF treats precision ($\pi$) as a first-class belief state. The model infers both the posterior mean ($\mu$) and precision ($\pi$) for every node at each step.
• Volatility Coupling: The framework introduces "volatility coupling," where a parent node modulates the rate of change (log-variance) of its child. This allows prediction errors to explicitly modulate precision updates via Volatility Prediction Errors (VOPE), a mechanism absent in standard value-coupled PC.
• Generative Model: The network is modeled as a hierarchy of belief nodes. Continuous state nodes are generated as Gaussians with means determined by non-linear value couplings (e.g., ReLU) and variances determined by volatility parents. Binary output nodes generate predictions via a logistic sigmoid, with surprise measured as binary cross-entropy.
• Inference and Learning:
  • Inference: Instead of iterative relaxation, the quadratic expansion of the variational energy allows for an analytic stationary point, reducing inference to a single update sweep.
  • Weight Updates: Synaptic weights (value-coupling strengths) are updated via local Hebbian rules. The authors propose three variants, with the primary benchmark using a precision-weighted rule: $\alpha^* = \alpha + \eta \delta_\ell g(\mu_{\ell+1}) \cdot \pi_\ell$. This incorporates inferred precision as a natural gradient weight.

Key Contributions

1. Reformulation of PC: The authors provide a reformulation of deep PC networks as hierarchical Gaussian filters, replacing iterative inference with closed-form variational updates.
2. Precision Learning: They demonstrate that precision can be endogenously inferred online as posterior beliefs coupled to the hierarchy, rather than treated as fixed hyperparameters.
3. Hebbian-Compatible Updates: The method yields weight update rules that incorporate precision as a natural gradient weight, maintaining biological plausibility.
4. Performance Benchmarks: The approach achieves state-of-the-art convergence on FashionMNIST, approaching backpropagation in wall-clock cost while converging in fewer epochs.

Experimental Results

The authors evaluated their method (HGF) against Multilayer Perceptrons (MLP) trained with backpropagation and vanilla Predictive Coding Networks (PCN) on the FashionMNIST dataset.

• Direct Comparison (FashionMNIST): Across various depth ($d \in \{2, 8\}$) and width configurations, the HGF converged faster and achieved higher test accuracy than vanilla PCN. While BP remained competitive, HGF approached BP's performance with significantly fewer epochs.
• Online Learning: In a strictly online regime (batch size 1), HGF converged to a lower steady-state error than both BP and PCN baselines.
• Data Efficiency: On small dataset subsets ($n \in \{60, 3000\}$), HGF outperformed both baselines, demonstrating superior data efficiency.
• Concept Drift: In a non-stationary environment where class labels were periodically permuted, HGF maintained lower steady-state error and recovered more rapidly from drift events compared to baselines, leveraging its accumulated belief states without resetting precision.
• Precision Learning Dynamics: On a synthetic spiral classification task, networks with volatility coupling (precision learning) showed a hierarchical increase in expected precision, effectively gating learning and inference. However, the authors note that at extreme depths (depth 16), the fixed-precision baseline still outperformed the volatility-coupled variant, suggesting that while precision learning stabilizes error signals, the resulting inter-layer precision imbalances require further investigation.

Significance and Claims

The paper establishes that closed-form variational inference with online precision learning provides a tractable foundation for deep predictive coding networks. By resolving the need for iterative relaxation and global error signals, the method retains the biological and interpretative advantages of PC while addressing its primary scalability bottlenecks.

The authors modestly frame their contribution as a step toward bridging the gap between theoretical PC and practical deep learning. They acknowledge that:

• The current implementation processes samples sequentially rather than in batches, which accounts for most of the remaining wall-clock gap to BP.
• The framework has not yet been extended to convolutional architectures or large-scale natural image datasets (e.g., CIFAR, ImageNet).
• The trade-off between the fidelity of the variational derivation (dynamic precision) and engineering choices for scaling (e.g., ePC, $\mu$PC) requires further controlled comparison.

Ultimately, the work argues that dynamic precision learning is essential for deep PC, offering a principled handle on uncertainty and enabling Hebbian-compatible plasticity without the computational overhead of iterative inference.