Learning in Deep Networks under Dale's Constraint

Technical Summary: Learning in Deep Networks under Dale's Constraint

Problem Statement
Deep learning has become a primary framework for systems neuroscience, yet the standard backpropagation algorithm faces significant hurdles in biological plausibility. A central obstacle is Dale's constraint: biological neurons are either purely excitatory or purely inhibitory, and synapses cannot change sign. Furthermore, neuronal activity (firing rates) is inherently non-negative. Standard backpropagation relies on signed activations, signed error signals, and sign-unconstrained synaptic weights to propagate gradients. Existing biologically plausible learning models (e.g., predictive coding, feedback alignment) often fail to simultaneously satisfy Dale's law and the non-negativity of activity while recovering exact gradient updates. Specifically, many models allow neurons to represent mixed-sign values or rely on global error signals that violate local synaptic constraints.

Methodology
The authors propose a novel neural architecture and learning rule that strictly adheres to Dale's constraint and non-negative activity while supporting backpropagation-like learning.

1. On-Off Neural Circuit Motif:

   • The core building block is a small circuit motif composed of excitatory and inhibitory neurons.
   • This motif maps two non-negative inputs ($x_1, x_2 \ge 0$) to two non-negative output channels, denoted $y^+$ (On) and $y^-$ (Off).
   • The circuit implements a "soft" difference operation. If $x_1 > x_2$, the On channel activates; if $x_2 > x_1$, the Off channel activates.
   • A signed quantity $d = x_1 - x_2$ is represented not by a single mixed-sign neuron, but by the pair of non-negative channels: a positive difference is encoded in $y^+$, and a negative difference is encoded in $y^-$ with the same magnitude. At most one channel is active at a time (or both are zero if the difference is below a threshold).

2. Network Architecture:

   • Standard deep networks are constructed by replacing each hidden mixed-sign neuron with an on-off motif.
   • The network operates in two streams:
     • Bottom-Up (BU): Computes feedforward activations using the on-off motifs.
     • Top-Down (TD): Propagates feedback signals for learning. For every BU motif, there are two corresponding TD motifs (one for the BU On channel, one for the BU Off channel).
   • The TD stream also uses paired non-negative channels to represent positive and negative synaptic updates.

3. Local Hebbian Learning Rule:

   • Learning is driven by a local Hebbian update that combines presynaptic BU activity with paired TD feedback signals.
   • For a synapse with presynaptic activity $x_j$, the update is determined by two TD channels:
     • Activity in the positive TD channel ($\bar{\partial}_i$) drives a Hebbian increase in synaptic weight.
     • Activity in the negative TD channel ($\partial_i$) drives a Hebbian decrease.
   • The effective update is $\Delta W_{ij} = \eta x_j (\bar{\partial}_i - \partial_i)$.
   • Crucially, the synaptic weight $W_{ij}$ itself has a fixed sign (excitatory or inhibitory), but its magnitude is updated based on the difference between the two non-negative TD signals.

Theoretical Analysis
The authors provide a theoretical proof showing that under specific conditions, this local learning scheme exactly recovers the backpropagation update:

• Assumptions: The BU network consists of linear operations followed by ReLU nonlinearities, and the BU and TD weights are aligned (symmetric).
• Mechanism: The TD propagation satisfies the same recursion as backpropagation error signals ($\delta_l$). Specifically, the difference between the two non-negative TD channels ($\bar{\partial}_l - \partial_l$) propagates backward exactly as the signed error signal $\delta_l$ does in standard backpropagation, gated by the ReLU derivative (which corresponds to the active BU channel).
• Result: Consequently, the local Hebbian update $\eta (\bar{\partial}_l - \partial_l) h_{l-1}^T$ is mathematically equivalent to the gradient descent update $-\eta \frac{\partial L}{\partial W_l}$.

Experimental Results
The model was evaluated on image classification benchmarks:

• Controlled Experiments (MNIST, Fashion-MNIST, CIFAR-10):
  • The symmetric Hebbian model (where BU and TD weights are aligned) achieved test accuracy comparable to standard backpropagation trained networks of the same effective width.
  • Variants with "weak symmetry" (noisy alignment) and "learned motif" weights also performed well, suggesting the architecture is robust to imperfect alignment and does not require fixed unit-magnitude internal weights.
  • The on-off model outperformed standard ReLU MLPs of comparable size, suggesting the paired-channel representation offers computational benefits beyond just doubling the neuron count.
• Scaling (Tiny ImageNet):
  • On the more challenging Tiny ImageNet benchmark (200 classes), the on-off model with local Hebbian learning achieved 42.31% top-1 accuracy.
  • This significantly outperformed recent biologically inspired baselines like Self-Contrastive Forward-Forward (SCFF, 35.7%) and standard convolutional baselines of matched width (35.58%).
  • The results indicate the architecture scales effectively to deeper, convolutional networks.

Significance and Claims
The paper claims to resolve a long-standing discrepancy in biologically plausible learning: how to implement effective credit assignment in networks constrained by Dale's law and non-negative activity.

• Biological Plausibility: The model demonstrates that signed learning signals can emerge from structured interactions between non-negative neural populations (On/Off channels) without requiring neurons to fire negatively or synapses to change sign. This aligns with biological evidence of on-off pathways in the retina and visual cortex, as well as recent findings of vectorized dendritic feedback.
• Computational Efficacy: Contrary to the assumption that biological constraints necessarily degrade performance, the on-off architecture not only matches but often exceeds the performance of standard "vanilla" networks of comparable size. The authors suggest that the paired-channel representation provides an efficient way to encode visual features (e.g., complementary edge polarities or object-background contrasts), offering a computational advantage alongside biological realism.
• Limitations: The authors acknowledge that the model is not a complete biological model of cortical learning. It requires a doubling of neurons (or channels) compared to standard networks and relies on the alignment of BU and TD pathways, though they show this alignment need not be exact.

In summary, the work provides a concrete mechanism for backpropagation-like learning in Dale-constrained networks, bridging the gap between the computational power of gradient-based learning and the physical constraints of biological neural circuits.