Technical Summary: Adaptive Multi-Scale Goodness Aggregation for Forward–Forward Learning

1. Problem Statement

The Forward–Forward (FF) algorithm, introduced by Hinton, offers a biologically plausible alternative to backpropagation by utilizing two forward passes (one on real data, one on negative data) to train layers independently. This approach eliminates the need for storing intermediate activations for gradient computation, thereby reducing memory usage and avoiding the "weight transport" problem. However, the baseline FF algorithm suffers from significant performance gaps compared to backpropagation-based models (e.g., ~92% vs. ~98% on MNIST).

The authors identify that this performance gap stems not from the core FF concept, but from four specific, overly simplistic design choices in the original implementation:

1. Single-Scale Goodness: The original algorithm computes "goodness" (a measure of activation quality) as a single scalar (sum of squared activations), failing to capture information at different levels of abstraction.
2. Static Negative Sampling: Negative samples are generated via random shuffling, resulting in training signals that are either trivially easy or destructively hard, lacking a curriculum-based progression.
3. Fixed Thresholds: A single, static threshold ($\theta$) is applied across all layers and throughout the entire training process, ignoring the fact that activation distributions change with network depth and training progress.
4. Static Learning Rates: The use of a fixed learning rate fails to account for the instability of early training stages where goodness scores and thresholds are still settling.

2. Methodology: Adaptive Multi-Scale Goodness Aggregation (AMSGA)

The paper proposes AMSGA, a framework that retains the core layer-wise, non-backpropagation structure of FF while replacing the four weak design choices with adaptive mechanisms.

A. Multi-Scale Goodness Aggregation

Instead of a single goodness metric, AMSGA computes goodness at three scales and combines them using depth-dependent weights:

• Local Goodness ($g_{local}$): The mean squared activation per neuron. This captures fine-grained, low-level features, weighted heavily in early layers.
• Intermediate Goodness ($g_{inter}$): Neurons are grouped into partitions, and the mean squared activation is averaged within groups. This provides a stable mid-level signal.
• Global Goodness ($g_{global}$): The mean squared activation across all neurons and samples, capturing the overall energy of the layer. This is weighted more heavily in deeper layers.
• Weighting Scheme: The final goodness $g(h, l)$ at layer $l$ is a linear combination:
  $$g(h, l) = w_{local}(l) \cdot g_{local} + 0.35 \cdot g_{inter} + w_{global}(l) \cdot g_{global}$$
  where weights shift linearly from favoring local structure in early layers to global activity in deeper layers.

B. Adaptive Negative Mining with Curriculum Learning

The selection of negative samples is dynamically adjusted based on training progress ($p = \text{current epoch} / \text{total epochs}$):

• Early Stage ($p < 0.3$): Focuses on "boundary proximity," selecting negatives close to the decision threshold to ensure stable initial learning.
• Middle Stage ($0.3 \le p < 0.7$): Expands the difficulty band and introduces "very hard" negatives (those with high goodness scores) at a 40% rate to refine representations.
• Late Stage ($p \ge 0.7$): Aggressively refines the model by widening the band and increasing the sampling rate of very hard negatives to 50%.
• Safety Mechanism: If the selection criteria yield too few samples, the algorithm guarantees a minimum batch size by adding the hardest remaining samples.

C. Layer-Dependent Adaptive Thresholds

The threshold $\theta$ is no longer a global constant. It is adapted based on both layer depth ($l$) and training progress ($p$):
$$\theta(l, p) = \theta_0 \times \left(1 + 0.15 \frac{l}{L}\right) \times (1 + 0.3 p)$$
This ensures that deeper layers (which handle more abstract features) and later training stages (where representations are more refined) utilize higher thresholds, maintaining proper calibration of the decision boundary.

D. Learning Rate Scheduling

To prevent instability during the initial settling of goodness scores and thresholds, AMSGA employs a warm-up phase (linearly increasing the learning rate for the first 10% of training) followed by cosine annealing for the remainder. This is critical in FF to avoid large, destabilizing updates before the system stabilizes.

3. Key Contributions

1. Novel Aggregation Mechanism: Introduction of multi-scale goodness aggregation that leverages local, intermediate, and global signals with depth-dependent weighting.
2. Curriculum-Based Training: Implementation of an adaptive negative mining strategy that evolves sample difficulty in sync with training progress.
3. Dynamic Calibration: Replacement of fixed thresholds with layer- and stage-dependent adaptive thresholds.
4. Stabilization Strategy: Integration of warm-up and cosine annealing specifically tailored to the unique dynamics of local learning rules.
5. Performance Validation: Demonstration that these modifications significantly close the performance gap between FF and backpropagation without sacrificing the algorithm's memory efficiency or biological plausibility.

4. Experimental Results

Experiments were conducted on MNIST and Fashion-MNIST using Multi-Layer Perceptron (MLP) architectures.

• MNIST:

  • Baseline FF: 92.00% test accuracy.
  • AMSGA: 94.45% test accuracy.
  • Improvement: +2.45%.
  • The model achieved a train-test gap of <0.1%, indicating strong generalization.

• Fashion-MNIST:

  • Baseline FF: 81.00% test accuracy.
  • AMSGA: 84.50% test accuracy.
  • Improvement: +3.5%.

• Computational Efficiency:

  • AMSGA introduces minimal computational overhead. The time per epoch increased only slightly (e.g., from 2.5s to 2.7s on MNIST) despite the added complexity of the aggregation and mining logic.
  • The method retains the core FF advantages: no backward pass, no global gradient storage, and layer-wise optimization.

5. Significance and Claims

The paper argues that the limitations of local learning rules (like FF) are not inherent to the concept but are the result of "unambitious default" design choices. By addressing how goodness is measured, how negative samples are selected, and how training dynamics are shaped, the authors demonstrate that biologically plausible learning can be made competitive with backpropagation on standard benchmarks.

The authors claim that AMSGA represents a step toward making biologically plausible learning "actually competitive, not just theoretically interesting." They emphasize that while a gap remains compared to state-of-the-art backpropagation models on harder tasks, the performance gains achieved here prove that the gap is largely due to training dynamics rather than the fundamental inability of local rules to learn complex representations. The work suggests that future improvements may lie in refining how label information is embedded and how layer-wise independence is managed, rather than abandoning the local learning paradigm.