🧬 biology

Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry

This paper proposes a biologically plausible supervised learning framework called Correlative Information Maximization that resolves the weight symmetry problem and emulates realistic neural structures by optimizing correlative mutual information between layer activations using coordinate descent.

Original authors: Bariscan Bozkurt, Cengiz Pehlevan, Alper T Erdogan

Published 2026-03-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Bariscan Bozkurt, Cengiz Pehlevan, Alper T Erdogan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Problem: The "Magic Mirror" of the Brain

Imagine you are trying to teach a robot how to recognize a cat. In the world of modern AI, we use a method called Backpropagation. Think of this like a teacher standing behind the robot, whispering corrections into its ear.

How it works: The robot makes a guess ("Is that a cat?"). The teacher checks the answer, calculates the mistake, and sends a signal backwards through the exact same wires the robot used to make the guess, telling it exactly how to adjust its internal knobs (weights).
The Biological Flaw: In our actual brains, neurons don't have "magic mirrors." If a signal travels from Neuron A to Neuron B, there isn't a separate, identical wire going from B back to A with the exact same strength. Our brains are messy, one-way streets that somehow manage to learn. The fact that AI needs these perfect "symmetric" wires to learn is the biggest reason scientists think our current AI models don't work like real brains.

The Solution: The "Two-Way Prediction Game"

This paper proposes a new way for neural networks to learn that doesn't need those magic mirror wires. The authors call it Correlative Information Maximization (CorInfoMax).

Instead of a teacher whispering corrections from the back, imagine the network is a team of detectives playing a game of "Forward and Backward Prediction."

The Analogy: The Detective Squad

Imagine a multi-story building where each floor is a layer of the neural network.

Floor 1 (Input): Sees the raw data (a blurry image).
Floor 2 (Middle): Tries to guess what Floor 3 will see.
Floor 3 (Output): Tries to guess what Floor 2 saw.

The Rule of the Game:
Every floor wants to be as "surprised" as possible by its neighbors, but also wants to be able to predict them perfectly.

Forward Prediction: Floor 2 looks at Floor 1 and says, "Based on what I see here, I predict Floor 3 will look like this."
Backward Prediction: Floor 2 also listens to Floor 3, who says, "Based on what I see here, I predict Floor 2 looked like this."

The goal isn't just to get the right answer at the top; it's to make sure every floor can predict its neighbors in both directions. If Floor 2 can't predict Floor 3, or if Floor 3 can't predict Floor 2, the "connection" between them is weak, and they need to adjust their internal knobs.

The Magic Trick: Why We Don't Need Symmetric Wires

In the old "Backpropagation" method, the wire going up (Forward) and the wire coming down (Backward) had to be identical twins.

In this new CorInfoMax method, the "Forward" prediction and the "Backward" prediction are two different games played with two different sets of rules.

The wire going up is trained to minimize the error of the forward guess.
The wire coming down is trained to minimize the error of the backward guess.

Because these are two separate optimization problems, the wires don't need to be identical. They can be totally different! This solves the "Weight Symmetry" problem. It's like having a one-way street for traffic going up and a different one-way street for traffic coming down; you don't need the roads to be mirror images of each other to get traffic flowing.

The Biological "Hardware": The Three-Room Apartment

The paper also shows that when you force a computer to learn using this method, the resulting structure looks exactly like a real biological neuron.

Imagine a neuron not as a simple lightbulb, but as a three-room apartment:

The Living Room (Soma): This is the main decision-maker. It fires the signal.
The Basement (Basal Dendrites): This room receives mail from the floor below (forward input).
The Attic (Apical Dendrites): This room receives mail from the floor above (backward/top-down input).

In the real brain, these rooms talk to each other. The paper shows that the math of "Correlative Information Maximization" naturally creates this exact three-room structure.

The "Forward" signal comes in the basement.
The "Backward" signal comes in the attic.
The Living Room compares the two. If the basement says "It's a cat" and the attic says "It's a dog," the Living Room gets confused and adjusts the wiring to fix the mismatch.

This matches real biology perfectly, where neurons have distinct parts for receiving different types of signals.

The "Noise" and the "Silence"

The paper also introduces a concept called Lateral Inhibition.

Analogy: Imagine a crowded party where everyone is shouting. To hear the important conversation, everyone else needs to shut up.
In the Network: If one neuron is very active, it sends a signal to its neighbors to "calm down." This forces the network to be sparse (only a few neurons fire at a time). This is how the brain saves energy and creates clear, distinct memories. The math in the paper naturally creates these "shut-up" signals (interneurons) without anyone having to program them explicitly.

Why This Matters

It's Biologically Plausible: It explains how the brain might learn without needing impossible "magic wires" that connect forward and backward paths perfectly.
It Works: The authors tested this on standard image recognition tasks (like identifying handwritten numbers or clothes). Their "biological" network performed just as well as standard AI, proving you don't need the "magic mirror" to get smart results.
It's a New Blueprint: Instead of forcing biology to fit our current AI models, this model forces our AI to fit the rules of biology, resulting in a more realistic and potentially more efficient way to build intelligent machines.

Summary in One Sentence

The paper suggests that instead of a teacher whispering corrections from the back (which requires impossible mirror-wires), the brain learns by having every layer constantly guess what its neighbors are thinking in both directions, a process that naturally creates the complex, three-part structure of real neurons and solves the mystery of how we learn without symmetry.

1. Problem Statement

The paper addresses two fundamental challenges in the intersection of neuroscience and deep learning:

The Weight Symmetry Problem: The standard backpropagation (BP) algorithm requires that the backward path (error propagation) uses synaptic weights that are the exact transposes of the forward path weights. This "weight transport" is considered biologically implausible, as biological neurons lack a mechanism to copy weights from feedforward to feedback pathways.
Biological Realism: Conventional artificial neurons are oversimplified. Biological neurons (specifically pyramidal neurons) have complex structures with distinct compartments (soma, basal dendrites, apical dendrites) and utilize lateral inhibition. Existing biologically plausible models often struggle to reconcile these structural details with effective supervised learning.

2. Methodology: Correlative Information Maximization (CorInfoMax)

The authors propose a new normative framework based on Correlative Information Maximization (CorInfoMax) to derive network structure and learning rules.

A. The Objective Function

Instead of minimizing prediction error directly, the framework maximizes the Correlative Mutual Information (CMI) between activations of consecutive layers ( $r^{(k)}$ and $r^{(k+1)}$ ), augmented with a Mean Square Error (MSE) loss for the final output layer.
The total objective $J$ is:
$J = \sum_{k=0}^{P-1} I^{(\epsilon_k)}(r^{(k)}, r^{(k+1)}) - \frac{\beta}{2} \|y_T - r^{(P)}\|^2_2$
Where $I^{(\epsilon)}$ is the CMI, defined via the determinant of correlation matrices. Crucially, the CMI has two equivalent mathematical forms:

Forward Prediction Form: Maximizes the dependence of $r^{(k+1)}$ on $r^{(k)}$ (minimizing forward prediction error).
Backward Prediction Form: Maximizes the dependence of $r^{(k)}$ on $r^{(k+1)}$ (minimizing backward prediction error).

B. Solving the Weight Symmetry Problem

The core innovation lies in utilizing both forms of the CMI simultaneously for adjacent layers:

The Forward Prediction term drives the learning of feedforward weights ( $W_{ff}$ ).
The Backward Prediction term drives the learning of feedback weights ( $W_{fb}$ ).
Because these are derived from two distinct optimization problems (predicting $k+1$ from $k$ vs. predicting $k$ from $k+1$ ), the resulting weights $W_{ff}$ and $W_{fb}$ are naturally asymmetric. There is no requirement for $W_{fb} = W_{ff}^T$ , thus resolving the weight transport issue without needing random feedback weights or complex alignment mechanisms.

C. Network Architecture and Dynamics

By applying Projected Gradient Ascent to optimize layer activations under polytopic constraints (e.g., non-negativity), the authors derive a biologically plausible network structure:

Multi-Compartment Pyramidal Neurons: The dynamics naturally map to a three-compartment model:
- Soma ( $u^{(k)}$ ): The cell body potential.
- Basal Dendrites ( $v_B^{(k)}$ ): Receive feedforward input ( $W_{ff}$ ).
- Distal Apical Dendrites ( $v_A^{(k)}$ ): Receive top-down feedback ( $W_{fb}$ ) and lateral inputs.
Lateral Inhibition & Autapses: The maximization of unconditional entropy (to prevent dimensional collapse) introduces lateral connections. These are decomposed into:
- Autapses ( $D^{(k)}$ ): Self-connections.
- Lateral Inhibition ( $O^{(k)}$ ): Mediated by SST interneurons, enforcing sparsity or non-negativity constraints.
Learning Dynamics (Equilibrium Propagation): The network uses a time-contrastive learning approach (similar to Equilibrium Propagation). It operates in two phases:
1. Free Phase: The network runs with $\beta=0$ to reach equilibrium.
2. Nudged Phase: The output is "nudged" toward the target label ( $\beta > 0$ ), and weights are updated based on the difference in neural activity between the two phases.

3. Key Contributions

Principled Solution to Weight Symmetry: The paper demonstrates that maximizing CMI using two alternative expressions naturally yields asymmetric forward and backward prediction networks, eliminating the need for weight transport.
Emergence of Biological Structure: The optimization of the information-theoretic objective naturally results in a network architecture featuring multi-compartment pyramidal neurons, apical/basal dendrites, and lateral inhibitory interneurons, aligning with experimental neuroscience findings.
Normative Framework: It provides a unified theoretical basis where network structure, dynamics, and learning rules emerge from a single objective (information maximization) rather than being hand-crafted.
Performance: The method achieves competitive performance on standard benchmarks compared to other biologically plausible algorithms and standard backpropagation.

4. Experimental Results

The authors evaluated the framework on MNIST, Fashion-MNIST, and CIFAR-10 (and CIFAR-100 for deeper networks) using Multi-Layer Perceptrons (MLPs).

Accuracy:
- MNIST: CorInfoMax achieved 97.62% (non-negative constraint) and 97.71% (sparsity constraint), comparable to Backpropagation (98.27%) and superior to other biologically plausible methods like Equilibrium Propagation (97.61%) and Predictive Coding.
- CIFAR-10: Achieved 51.86%, outperforming Equilibrium Propagation (49.28%) and Feedback Alignment (50.75%), though slightly below standard Backpropagation (53.96%).
Weight Asymmetry: The authors measured the cosine angle between forward weights and the transpose of backward weights. The angle remained close to 90 degrees (orthogonal) throughout training, confirming that the weights are indeed asymmetric and do not converge to a symmetric state.
Robustness: The framework performed well under different activation constraints (non-negative unit hypercube $B_{\infty,+}$ and sparse $L_1$ -norm ball $B_{1,+}$ ).

5. Significance

This work represents a significant step toward bridging the gap between artificial intelligence and biological plausibility.

Theoretical Impact: It challenges the necessity of backpropagation's weight symmetry, suggesting that the brain may utilize bidirectional predictive coding based on information maximization rather than error backpropagation with tied weights.
Neuromorphic Potential: Since the learning rules are local (dependent on pre- and post-synaptic activities and local error signals) and the architecture mimics biological neurons, this framework is highly suitable for implementation on future neuromorphic hardware, potentially offering energy-efficient deep learning solutions.
Unifying Perspective: It unifies concepts of information theory, predictive coding, and multi-compartmental neuron modeling into a single, coherent supervised learning framework.