Predictive Coding Graphs are a Superset of Feedforward Neural Networks

Here is an explanation of the paper "Predictive Coding Graphs are a Superset of Feedforward Neural Networks" using simple language and creative analogies.

The Big Idea: From a Straight Line to a Web

Imagine you have a standard Feedforward Neural Network (FNN). Think of this like a conveyor belt in a factory.

Raw materials (data) enter at the start.
They move in one direction: Station A $\rightarrow$ Station B $\rightarrow$ Station C.
At each station, a worker (a neuron) does a specific job and passes the item to the next.
Once the item reaches the end, you get a final product (a prediction).
The Catch: If a mistake happens at Station C, the only way to fix it is to stop the whole line, send a message all the way back to Station A to tell them what went wrong, and then restart. This "sending the message back" is called Backpropagation. It works well, but it's rigid and requires the whole factory to stop and talk to each other in a very specific order.

Now, meet the paper's hero: Predictive Coding Graphs (PCGs).
Think of a PCG not as a conveyor belt, but as a busy, chaotic open-plan office.

Everyone (every node) is talking to everyone else.
Instead of just waiting for the next item, every worker is constantly guessing what the next item should look like based on what they see.
If your guess is wrong, you feel a "tension" (error). You adjust your guess until the tension goes away.
The Magic: In this office, you don't just talk forward. You can talk backward, sideways, or even talk to yourself. The whole office adjusts together to find the best solution.

The Paper's Two Big Discoveries

The author, Björn van Zwol, proves two main things that connect these two very different worlds.

1. The "Quiet Time" Discovery (Testing Phase)

The Analogy: Imagine the factory (FNN) and the office (PCN) are both trying to assemble a toy.

During Training (Learning): The office is loud. People are arguing, guessing, and adjusting their positions to figure out how to build the toy. The factory is quiet, just following a strict script.
During Testing (Inference): This is when the toy is finished and you just need to show it to a customer.
- The paper proves that when the office (PCN) is just "showing off" the final toy, it behaves exactly like the factory (FNN). The chaotic guessing stops, and the workers line up in a straight line, passing the toy down the chain just like the factory.
- Why it matters: This means PCNs are just as good at making predictions as the standard networks we use today. They aren't a "worse" alternative; they are a "different" one that ends up doing the same job perfectly.

2. The "Master Blueprint" Discovery (The Superset)

The Analogy: Imagine the Factory (FNN) is a specific type of house: a bungalow. It has a front door, a hallway, and a back door. It's simple and straight.

The paper proves that the PCG is not just another house; it is the entire city of architecture.
The PCG is a universal blueprint that can build any house.
If you want a bungalow (a standard FNN), you just take the PCG blueprint and tape over the windows and doors that aren't needed. You "mask" the extra connections.
But because the PCG is a "superset," it can also build a house with a spiral staircase (loops), a house with a secret tunnel to the back (backward connections), or a house where the kitchen talks to the bedroom (lateral connections).

The "Superset" Concept:

FNNs are a small circle inside a big circle.
PCGs are the big circle.
Every FNN is a PCG, but not every PCG is an FNN.

Why Should We Care? (The "So What?")

1. It's More "Biological" (Like the Human Brain)
Our brains don't work like conveyor belts. Neurons in the brain talk to each other in loops and webs. Backpropagation (the standard training method) is mathematically clever but biologically weird (how does the brain send a "negative error signal" backward through time?). PCGs work more like how our brains actually process information: by constantly predicting and correcting errors in real-time.

2. It Unlocks New Architectures
Because PCGs allow "backward" and "sideways" connections, they can solve problems that standard networks struggle with.

Skip Connections: You might have heard of "ResNets" (Residual Networks), which are famous for being very deep and smart. They use "skip connections" (jumping over layers). The paper shows that these are just a specific, easy-to-make version of a PCG.
The Future: If we can train these weird, loop-filled networks (which standard methods can't do easily), we might discover new types of AI that are more robust, efficient, or creative.

3. The Trade-off
There is a catch. Running a "conveyor belt" (FNN) is fast. Running a "busy office" (PCG) where everyone talks to everyone takes more time and computing power. The paper admits this: PCGs are slower to run right now. But, the author suggests that the extra flexibility might be worth the extra time, just like a complex Swiss Army knife is more useful than a simple knife, even if it takes longer to open.

Summary in One Sentence

This paper proves that Predictive Coding Graphs are the "Swiss Army Knife" of neural networks: they include all the standard "knives" (Feedforward Networks) we use today as simple special cases, but they also have the potential to be much more complex, flexible, and brain-like tools for the future of AI.

Here is a detailed technical summary of the paper "Predictive Coding Graphs are a Superset of Feedforward Neural Networks" by Björn van Zwol.

1. Problem Statement

Predictive Coding Networks (PCNs) are neuroscience-inspired probabilistic latent variable models that offer a biologically plausible alternative to backpropagation (BP) for training neural networks. While PCNs have been shown to converge to the computations of Feedforward Neural Networks (FNNs) during inference, and Predictive Coding Graphs (PCGs) have been proposed to allow arbitrary network topologies (including loops and non-hierarchical structures), the precise mathematical relationship between these frameworks remained formally undefined.

Specifically, two gaps existed in the literature:

PCN vs. FNN Equivalence: While it was believed that PCNs converge to FNNs, a rigorous proof establishing them as mathematically equivalent during the testing (inference) phase was lacking. Consequently, the applicability of the Universal Approximation Theorem (UAT) to PCNs lacked formal justification.
PCG vs. PCN Relationship: It was unclear how PCGs (which allow arbitrary graph topologies) relate to standard hierarchical PCNs. Specifically, whether PCGs constitute a strict mathematical superset of PCNs, and how their energy functions and dynamics align, had not been formally proven.

2. Methodology

The author employs rigorous mathematical proofs to establish set-theoretic relationships between these neural network architectures. The methodology involves:

Formal Definitions: Defining FNNs, PCNs, and PCGs based on their energy functions ( $E$ $E$ ), activity rules (node updates), and learning rules (weight updates).
- FNN: Defined by a standard feedforward activity rule $a^\ell = f(W a^{\ell-1})$ .
- PCN: Defined by minimizing an energy function $E_N = \sum (\epsilon^\ell)^2$ , where $\epsilon$ is the prediction error.
- PCG: A generalization where nodes and weights exist on an arbitrary graph, defined by a global energy function $E_G$ .
Backward Induction: Used to prove that minimizing the PCN energy function during testing yields the exact same node activations as the FNN feedforward pass.
Weight Matrix Partitioning: Used to demonstrate that a PCG weight matrix can be partitioned into block matrices. By constraining specific blocks to zero (masking), the PCG reduces exactly to a hierarchical PCN structure.
Equivalence of Dynamics: Showing that the objective functions ( $E_N$ vs. $E_G$ ) and the resulting activity/learning rules are identical under specific weight constraints.

3. Key Contributions

A. Proof of PCN-FNN Equivalence During Testing

The paper provides a formal proof (Theorem 1) that PCNs are equivalent to FNNs during the testing phase.

Mechanism: By minimizing the energy function $E_N$ with respect to node activations, the system converges to a state where the prediction error $\epsilon^\ell = 0$ for all layers. This condition forces the PCN activity rule to become $a^\ell = f(\sum w a^{\ell-1})$ , which is identical to the FNN forward pass.
Implication: This proves that the Universal Approximation Theorem (UAT) applies to PCNs. Since FNNs are universal function approximators and PCNs compute the exact same function during inference, PCNs inherit this theoretical guarantee.

B. Proof of PCGs as a Superset of PCNs

The paper proves (Theorem 2) that PCGs are a mathematical superset of PCNs.

Mechanism: By defining a specific weight matrix structure for a PCG where non-hierarchical connections (backward, lateral, and skip connections) are set to zero, the PCG's energy function $E_G$ becomes equivalent to the PCN energy function $E_N$ (up to a constant).
Implication: PCGs generalize PCNs. While PCNs are restricted to hierarchical layers, PCGs allow for arbitrary topologies, including loops and non-hierarchical connections, while retaining the ability to replicate PCN behavior when the graph is constrained.

C. Unification of Network Topologies

The work unifies the understanding of modern deep learning architectures:

Skip Connections: The paper visualizes how ResNet-style skip connections are simply a specific non-zero block in the PCG weight matrix.
Recurrence: It distinguishes the recurrence in PCGs (occurring in "inference time" via iterative minimization, similar to Hopfield networks) from the recurrence in RNNs (occurring in "data time" via weight sharing).

4. Results

Theoretical Validation: The paper establishes that PCNs are not just approximations of FNNs but are mathematically equivalent during inference. This solidifies the theoretical foundation for using PCNs in ML tasks.
Superset Hierarchy: The paper establishes the hierarchy: FNNs $\subset$ PCNs $\subset$ PCGs.
- FNNs are a special case of PCNs (during testing).
- PCNs are a special case of PCGs (with hierarchical masking).
Universal Approximation: As a corollary, PCGs (when configured hierarchically) are universal function approximators.
Topological Flexibility: The results confirm that PCGs can train arbitrary graph structures using Inference Learning (IL), a capability that Backpropagation (BP) cannot easily handle due to its reliance on strict feedforward chains for gradient calculation.

5. Significance

Bridging Neuroscience and ML: The paper strengthens the link between biologically plausible models (Predictive Coding) and standard Deep Learning. It demonstrates that the "biological" approach is not just an alternative but a generalization of standard neural networks.
Justification for Non-Hierarchical Architectures: By proving PCGs are a superset, the paper provides a theoretical basis for exploring non-hierarchical, looped, or recurrent structures in ML that are difficult to train with BP. This opens the door for studying how network topology (beyond just depth) affects learning performance.
Theoretical Rigor: The paper addresses a gap in the literature where PCNs were often treated as heuristic or experimental. By providing formal proofs for equivalence and universality, it moves PCNs from a niche biological model to a robust mathematical framework for machine learning.
Practical Implications: While PCGs offer greater topological freedom, the paper notes a computational trade-off: inference in PCGs is $O(N^2 T)$ (where $T$ is inference steps) compared to $O(LM)$ for FNNs. However, the potential benefits of richer topologies (like those found in ResNets or biological brains) may outweigh this cost, suggesting a new direction for efficient, biologically-inspired AI.

In summary, this paper fundamentally repositions Predictive Coding Graphs as a comprehensive framework that encompasses traditional feedforward networks, offering a mathematically rigorous path to explore more complex, biologically plausible neural architectures.