Sparse Goodness: How Selective Measurement Transforms Forward-Forward Learning

This paper demonstrates that introducing sparsity into the Forward-Forward algorithm's goodness function—specifically through top-k selection and adaptive entmax-weighted energy—significantly outperforms traditional sum-of-squares methods, establishing sparsity as the most critical design choice for improving learning performance.

The Gist

Imagine you are trying to teach a robot to recognize different types of clothing (like a t-shirt, a shoe, or a bag) just by looking at pictures.

For a long time, the standard way to teach AI (called "Backpropagation") was like a teacher walking through a factory, checking every single worker's mistake, and then walking all the way back to the beginning to tell everyone what to fix. This is powerful, but it's not how the human brain works.

In 2022, a genius named Geoffrey Hinton proposed a new way called Forward-Forward (FF). Instead of walking backward, the robot learns layer-by-layer as it moves forward, like a relay race. Each layer of the robot's brain has a simple rule: "Make sure the signal is Good for the right answer and Bad for the wrong answer."

The problem? The original rule for "Goodness" was very clumsy. It was like a noise meter that measured the total volume of all the neurons firing. If 1,000 neurons whispered softly, the meter said "Good!" even if none of them were actually saying anything useful.

This paper is like a team of engineers who said, "Let's fix that noise meter." They discovered that the secret to making this robot brain work isn't listening to everyone, but listening only to the loudest voices.

Here is the breakdown of their discovery using simple analogies:

1. The Old Way: The "Total Volume" Meter (Sum-of-Squares)

Imagine a crowded party. The old rule said: "If the total noise level of the room is high, that's a good party."

• The Flaw: You could have 1,000 people whispering "um, um, um," and the meter would scream "HIGH NOISE! GREAT PARTY!" But nobody is actually saying anything interesting. The signal is too diluted.

2. The New Idea: The "Top-K" Microphone

The authors proposed a new rule: Top-k Goodness.

• The Analogy: Instead of listening to the whole room, we put up a microphone that only picks up the top 5 loudest voices.
• Why it works: If the room is full of whispering, the mic stays quiet (Bad). But if a few people are shouting the correct answer, the mic picks them up loud and clear (Good).
• The Result: By ignoring the background noise and focusing only on the "stars" of the show, the robot learned 22.6% better at recognizing clothes than before.

3. The Upgrade: The "Smart DJ" (Entmax)

The "Top 5" rule is great, but it's a bit rigid. What if the answer requires 3 people to shout, and another time it needs 7?

• The Analogy: The authors introduced a Smart DJ (called Entmax). Instead of picking a fixed number of people, the DJ listens to the room and decides, "Okay, today I need to focus on the top 15% of the crowd, but I'll give them different volumes based on how important they are."
• The Result: This "Adaptive Sparsity" is the sweet spot. It's not too crowded (listening to everyone) and not too empty (listening to only one person). It found the perfect balance, pushing the robot's accuracy even higher.

4. The Secret Sauce: The "Coach" at Every Step (FFCL)

In the original setup, the robot only got a hint about what it was supposed to guess (e.g., "This is a shoe") at the very beginning of the race. By the time the signal reached the later layers, that hint was weak and blurry.

• The Fix: The authors added a Coach who stands next to every single layer of the brain.
• The Analogy: Instead of just whispering the goal at the start line, the Coach shouts "SHOE!" to every runner in the relay race. This keeps the team focused on the target the whole way.
• The Result: This simple change gave a massive boost to every method, especially the ones that were struggling.

The Big Reveal: The "Goldilocks" Zone

The most important finding of this paper is a principle they call Sparsity.

• Too Dense (Listening to everyone): The signal is muddy and confusing.
• Too Sparse (Listening to only one person): You miss important context and the signal becomes shaky.
• Just Right (Adaptive Sparsity): Focusing on the most active, relevant neurons while ignoring the rest is the key to success.

The Final Score

By combining the Smart DJ (listening to the right amount of people) and the Coach (shouting the goal at every step), the robot went from being a beginner (56% accuracy) to a master (87% accuracy) on the Fashion-MNIST test.

In short: The paper teaches us that in AI, less is often more. You don't need to process every single detail to learn well; you just need to know how to pick out the most important signals and ignore the noise.

Technical Summary

1. Problem Statement

The Forward-Forward (FF) algorithm, proposed by Geoffrey Hinton, is a biologically plausible alternative to backpropagation. It trains neural networks layer-by-layer using a local "goodness function" to distinguish between positive (correctly labeled) and negative (incorrectly labeled) data.

Despite its potential, FF performance has stagnated because the community has relied almost exclusively on a single goodness function: Sum-of-Squares (SoS), defined as the mean squared activation of a layer ($g(h) = \frac{1}{d}\sum h_i^2$). The authors argue that SoS is suboptimal because:

• It treats all neurons equally, rewarding diffuse total activity rather than discriminative peaks.
• It interacts poorly with smooth activation functions (like GELU/Swish), which produce dense, non-zero activations that dilute the signal.
• It fails to leverage the potential of sparse, discriminative representations that are central to biological neural coding.

The paper aims to systematically explore the goodness function design space to determine what activations to measure and how to aggregate them, challenging the SoS default.

2. Methodology

The authors propose a new framework centered on sparsity in the goodness function and a modified label-injection strategy.

A. Sparse Goodness Functions

Instead of aggregating all activations, the authors propose measuring only the most active neurons:

1. Top-k Goodness: Measures the mean activation of only the $k$ most active neurons ($k \approx 2\%$ of layer width). This creates a "winner-take-all" dynamic, rewarding layers for producing strong, specific peaks for positive data while ignoring the rest.
2. Entmax-Weighted Energy: To overcome the rigidity of hard Top-k selection, the authors introduce a differentiable sparse weighting mechanism using $\alpha$-entmax.
   • It maps activations to a sparse probability vector $\pi = \text{entmax}_\alpha(h)$.
   • The goodness is calculated as $\sum \pi_i h_i^2$.
   • The parameter $\alpha$ controls sparsity: $\alpha=1$ is Softmax (dense), $\alpha=2$ is Sparsemax (hard sparse), and intermediate values (e.g., $\alpha \approx 1.5$) provide adaptive sparsity, where the number of active neurons varies per input.

B. Separate Label–Feature Forwarding (FFCL)

In standard FF, labels are concatenated with the input only at the first layer. The authors adopt FFCL (Forward-Forward with Cortical Loops), where class hypotheses are injected at every layer via a dedicated linear projection.

• Mechanism: Each layer computes pure features ($h_\ell$) and a combined representation ($\tilde{h}_\ell = h_\ell + W_{label} \cdot \text{onehot}(y)$).
• Benefit: This provides a stronger, direct training signal to every layer, preventing the "dilution" of label information in deeper layers.

C. Activation Function Interaction

The study investigates how activation functions (ReLU, GELU, Swish) interact with goodness functions. They hypothesize that smooth activations (GELU/Swish) create dense activity patterns that benefit sparse goodness functions (which can pick peaks from a rich distribution) but hurt SoS (which gets diluted by many small values).

3. Key Contributions

1. Identification of Sparsity as the Key Principle: The paper establishes that sparsity in the goodness function is the single most impactful design choice in FF networks.
2. Proposal of Top-k and Entmax Goodness: Introduction of Top-k and Entmax-weighted energy as superior alternatives to SoS, with Entmax achieving the best results through adaptive sparsity.
3. Integration of FFCL: Demonstration that separate label-feature forwarding provides an orthogonal performance boost, compounding with better goodness functions.
4. Sparsity Spectrum Analysis: Discovery of an inverted-U relationship between sparsity and performance. Both fully dense ($\alpha=1$) and fully sparse ($\alpha=2$) configurations underperform; the optimal point lies at intermediate adaptive sparsity ($\alpha \approx 1.5$).
5. Activation-Goodness Interaction: Uncovering that SoS degrades with smooth activations, while sparse goodness functions thrive with them.

4. Experimental Results

Experiments were conducted on MNIST and Fashion-MNIST using a 4-layer, 2000-unit fully connected network (4×2000).

• Baseline: Standard FF with SoS and ReLU achieved 56.41% on Fashion-MNIST.
• Top-k Improvement: Switching to Top-k with Swish activation yielded 79.03% (+22.6 percentage points).
• Entmax Improvement: Using Entmax-1.5 energy with GELU achieved 85.08% (+28.7 pp).
• Combined FFCL + Entmax: The final configuration (FFCL + Entmax-1.5 + GELU) achieved 87.12%.
  • This represents a +30.7 percentage point improvement over the baseline.
  • It outperforms recent external baselines (e.g., Shah and Tripathi [2025] reported 82.84%) by a significant margin.
• Robustness: The FFCL + Contrast Top-k configuration showed remarkable robustness to the choice of $k$, varying by less than 2 percentage points across a 40x range of $k$ values.
• Scaling: While SoS performance degraded as the network size increased (from 2×500 to 4×2000), Top-k performance improved, suggesting sparse goodness functions scale better with depth.

5. Significance and Implications

• Redefining FF Objectives: The paper fundamentally shifts the design philosophy of Forward-Forward learning from "maximizing total energy" to "maximizing selective, sparse peaks."
• Biological Plausibility: The findings align with sparse coding and k-winners-take-all (kWTA) theories in neuroscience, suggesting that FF networks learn more biologically realistic representations when they focus on specific active neurons rather than total activity.
• Practical Efficiency: The results suggest that a smaller network with the correct sparse goodness function can outperform a much larger network with a suboptimal dense function.
• General Design Principle: The "focus on the signal, not the total energy" principle offers a new guideline for designing local learning rules in neural networks, applicable beyond just FF.

In conclusion, the paper demonstrates that by replacing the default Sum-of-Squares with adaptive sparse goodness functions and utilizing separate label injection, Forward-Forward learning can achieve state-of-the-art results on standard benchmarks, closing a significant performance gap with backpropagation-based methods.