Technical Summary: HCL-FF

Problem Statement

Deep neural networks trained via backpropagation (BP) have achieved state-of-the-art performance in vision tasks but suffer from biological implausibility, high computational demands (requiring gradient storage and backward propagation), and opaque internal representations. The Forward-Forward (FF) algorithm offers a biologically plausible alternative by training each layer independently using a local "goodness" objective (the magnitude of activations). However, standard FF approaches face two critical limitations:

1. Lack of Hierarchical Coordination: FF forces shallow layers to directly discriminate fine-grained classes, leading to suboptimal early-layer representations. Unlike BP, which naturally learns a progression from low-level cues to high-level semantics, FF lacks a mechanism to coordinate this progression across layers.
2. The Decoupling Dilemma: To prevent deeper layers from trivially inheriting the goodness signal from previous layers, FF decouples goodness from the feature representation via normalization (removing magnitude information). While this prevents overfitting on previous signals, it leaves the resulting features semantically ambiguous and unconstrained, as the optimization objective only targets activation magnitude, not the relative geometric structure of the features.

Methodology

The authors propose HCL-FF (Hierarchical and Contrastive Learning for Forward-Forward), a framework that preserves the layer-wise independence of FF while addressing the above limitations through two key innovations.

1. Hierarchical Learning (Coarse-to-Fine Strategy)

To resolve the lack of coordination, HCL-FF introduces a curriculum learning strategy where supervision is organized from coarse to fine granularity:

• Super-class Targets: Shallow layers are supervised using coarse super-class targets, while deeper layers are progressively assigned finer-grained labels, culminating in fine-grained class discrimination at the final layer.
• Hierarchy Construction: The class hierarchy is constructed via a data-driven approach. A model is first pre-trained without hierarchy, and a linear classifier is trained on the final layer's features. The classifier weights serve as class prototypes, which are then clustered using hierarchical agglomerative clustering to form a semantic tree.
• Hierarchical Channel-Wise Competitive (HierCwC) Loss: The standard FF loss is modified to compute goodness at the super-class level for shallow layers. The super-class goodness is derived by averaging the per-class goodness within each super-class group. This reduces the complexity of early-layer objectives and encourages representations to evolve from broad semantic structures to fine-grained discrimination.

2. Contrastive Learning (Semantic Grounding)

To resolve the decoupling dilemma, HCL-FF applies a supervised contrastive objective directly to the goodness-decoupled features (the normalized feature vectors $z^{(\ell)}$):

• Feature Projection: At each layer, the goodness-decoupled features are global-average-pooled and projected into a latent embedding space.
• Supervised Contrastive Loss: A standard supervised contrastive loss is applied to these embeddings, encouraging samples of the same class to cluster together and different classes to separate.
• Significance: By constraining the relational geometry of the features after goodness decoupling, this objective restores the semantic meaning lost when magnitude information is removed. It ensures that while the scale (goodness) is decoupled, the direction (semantic content) remains discriminative.

3. Overall Framework

The total loss function for each layer $\ell$ combines the HierCwC loss and the contrastive loss:
$$L_{total}^{(\ell)} = L_{HierCwC}^{(\ell)} + \lambda L_{Con}^{(\ell)}$$
where $\lambda$ is set to 1. The framework utilizes a residual Forward-Forward architecture based on Channel-Wise Convolution (CW-Conv) layers, where each layer operates independently without gradient propagation across layers.

Key Contributions

1. HCL-FF Framework: A novel FF framework that integrates hierarchical supervision and contrastive learning to address the lack of coordination and the decoupling dilemma.
2. Coarse-to-Fine Curriculum: A strategy that reduces early-layer complexity by supervising shallow layers with super-classes, enabling a natural progression of semantic abstraction.
3. Contrastive Grounding: The first application of contrastive learning specifically to goodness-decoupled features in FF, effectively preserving semantic structure without violating the layer-wise independence constraint.
4. State-of-the-Art Performance: The method achieves new state-of-the-art results among FF-based methods on multiple benchmarks.

Experimental Results

Experiments were conducted on CIFAR-10, CIFAR-100, MNIST, Fashion-MNIST, and Tiny-ImageNet.

• Performance Gains: HCL-FF achieves significant accuracy improvements over the previous best FF method (DeeperForward):
  • +5.46% on CIFAR-10 (91.68% vs. 86.22%).
  • +17.00% on CIFAR-100 (70.09% vs. 53.09%).
  • +12.51% on Tiny-ImageNet (48.46% vs. 35.95%).
• Comparison to BP: On CIFAR-10, HCL-FF surpasses standard BP-trained ResNet-20 (91.68% vs. 91.25%) and narrows the gap to parameter-matched BP variants.
• Ablation Studies:
  • Adding contrastive learning alone improved CIFAR-100 accuracy by +9.12%, confirming its role in stabilizing decoupled features.
  • Adding hierarchical learning alone improved CIFAR-100 accuracy by +6.96%, highlighting its benefit for larger label spaces.
  • Strong data augmentation was only effective when combined with hierarchical supervision, as it reduced the burden on shallow layers.
• Analysis:
  • Layer-wise Accuracy: HCL-FF shows steady accuracy gains across layers, whereas prior methods plateau early.
  • Decoupling: Linear probe evaluations show that HCL-FF maintains high accuracy even after normalization, proving it preserves semantic structure in decoupled features, unlike DeeperForward where accuracy drops significantly after normalization.
  • Visualization: t-SNE visualizations confirm that HCL-FF maintains clear class separation in both goodness responses and goodness-decoupled features.

Significance and Claims

The paper claims that HCL-FF represents a significant step toward scalable, effective, and biologically plausible learning paradigms. By resolving the fundamental decoupling dilemma and introducing hierarchical coordination, the method demonstrates that layer-wise independent training can achieve performance comparable to, and in some cases exceeding, backpropagation-based methods.

The authors note that while HCL-FF improves upon existing FF methods, it still inherits certain constraints, such as slower convergence compared to end-to-end BP and the requirement for channel counts to scale with the number of classes (though this is alleviated in shallow layers). The work suggests that combining hierarchical supervision with contrastive grounding is a viable path to overcoming long-standing challenges in non-backpropagation learning.