Scalable Equilibrium Propagation via Intermediate Error Signals for Deep Convolutional CRNNs

Technical Summary: Scalable Equilibrium Propagation via Intermediate Error Signals

Problem Statement
Equilibrium Propagation (EP) is a biologically plausible learning rule for Convergent Recurrent Neural Networks (CRNNs) that estimates gradients using only local, temporally separated neuron states, avoiding the non-biological requirements of Backpropagation Through Time (BPTT). While EP has shown promise as a candidate for neuromorphic on-chip training, its application has been historically restricted to shallow architectures. As network depth increases, EP-trained CRNNs suffer from the vanishing gradient problem. In deep networks, backward-propagated error signals attenuate across layers during simulation, failing to sufficiently perturb the equilibrium states of intermediate neurons. This leads to a collapse in gradient computation and prevents the network from converging, resulting in poor performance in deeper models (e.g., VGG-11/13) compared to shallow ones (e.g., VGG-5).

Methodology
To address the scalability limitations of EP, the authors propose a novel framework that integrates intermediate learning signals to enhance information flow and neuron dynamics. The core methodology involves modifying the scalar primitive function $\Phi$ used in the EP training process by adding auxiliary loss terms at specific intermediate layers. The framework introduces two distinct augmentation strategies:

1. Local Error (LE) Signals: This approach incorporates pseudo-targets $\hat{y}^*_i$ at selected intermediate layers. The neuron states $\xi^t_i$ are projected via a randomly initialized matrix $B_i$ to generate intermediate readouts $\hat{\xi}^t_i$. A local loss $L_{Aug}$ is computed between these readouts and the pseudo-targets, providing direct supervision to intermediate layers.
2. Knowledge Distillation (KD): This approach utilizes a pre-trained "teacher" model. Intermediate layers of the student network (the EP-trained CRNN) are aligned with the corresponding logits of the teacher model. The loss $L_{Aug}$ is computed between the student's intermediate outputs and the teacher's outputs, scaled by a temperature parameter $\tau$.

The total loss function for the augmented EP framework is defined as:
$$L_{Total} = L_{EP}(\xi^t_{out}, \hat{y}) + \kappa \sum_{i \in \Upsilon} L_{Aug}(\xi^t_i, \hat{y}^*_i)$$
where $\kappa$ is a scaling factor and $\Upsilon$ is the set of layers receiving intermediate signals. Theoretical analysis (Theorem 3.1) demonstrates that under specific Lipschitz continuity conditions, this augmented framework retains convergence guarantees to a fixed point. Furthermore, Theorem 3.2 establishes that the gradient estimation remains an unbiased approximation of the true gradient (with an error bound of $O(\beta^2)$) as the nudging factor $\beta \to 0$.

The authors also explore weighted knowledge distillation, introducing learnable linear mapping weights to align student and teacher logits, though empirical results indicated this specific variant incurred higher memory costs with slightly lower performance than the standard KD approach.

Key Contributions

• Scalable EP Framework: This work presents the first EP framework capable of successfully training deep Convolutional CRNNs (surpassing the previous five-layer limit), effectively bridging the performance gap between shallow and deep architectures.
• Empirical Evidence of Vanishing Gradients: The paper provides empirical evidence demonstrating that vanishing gradients in deep EP networks occur during both the minimization of the scalar primitive function and the gradient computation process, leading to convergence failures.
• Integration of Intermediate Signals: To the authors' knowledge, this is the first integration of knowledge distillation and local error signals within the EP framework.
• Theoretical Convergence: The study provides a theoretical proof (Theorem 3.1) ensuring that the augmented neuron dynamics converge to stable states, preserving the biological plausibility of the EP update rule.

Results
The proposed framework was evaluated on the CIFAR-10 and CIFAR-100 datasets using deep VGG architectures (VGG-7, VGG-10/11, VGG-12/13).

• Performance: The augmented EP framework achieved state-of-the-art performance for EP-trained networks on these datasets. For CIFAR-10, the VGG-7-KD model achieved 89.67% test accuracy, and the VGG-7-LE model achieved 88.99%, outperforming prior EP studies restricted to VGG-5. On CIFAR-100, the VGG-12-KD model achieved 64.75% test accuracy.
• Convergence Behavior: Local Error (LE) signals demonstrated faster convergence, reaching over 70% accuracy before epoch 15, whereas Knowledge Distillation (KD) required more than 45 epochs to reach comparable levels, likely due to the need for a warm-up phase.
• Efficiency: Despite the addition of intermediate signals, the computational and memory overhead remained marginal compared to standard EP. Crucially, the memory consumption of the augmented EP framework was significantly lower than BPTT (e.g., BPTT consumed ~18.1x more GPU memory than the KD-augmented EP for VGG-13 on CIFAR-100).

Significance and Claims
The paper claims that the introduction of intermediate learning signals is a significant advancement in the scalability of Equilibrium Propagation. By enabling the training of significantly deeper architectures without sacrificing the biologically plausible, local nature of the learning rule, the work suggests that EP can be extended to practical, complex applications. The authors highlight that this approach maintains the suitability of EP for neuromorphic and on-chip training, where hardware constraints necessitate efficient, local learning mechanisms. The work positions itself as a complementary advancement to other recent efforts (such as those using residual connections) to solve the vanishing gradient problem in EP, paving the way for future research into applying EP to large-scale vision and language models.