🤖 machine learning

What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility

This paper argues that standard evaluation metrics for Feedback Alignment (accuracy and gradient cosine similarity) are insufficient due to silent failure modes like reference gradient collapse and aggregation masking, and proposes a new diagnostic protocol based on scale stability, reference validity, and depth utility to reliably identify non-functional credit assignment in deep networks.

Original authors: Yuren Hao, Xiang Wan, ChengXiang Zhai

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuren Hao, Xiang Wan, ChengXiang Zhai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a team of workers (a deep neural network) to solve a puzzle. The standard way to teach them is Backpropagation (BP). In this method, the boss (the output) sends a perfect, detailed memo back down the chain to every worker, saying exactly what they did wrong and how to fix it. This works great, but it requires a very specific, rigid communication system that doesn't exist in nature (like the human brain) and is hard to build on efficient computer chips.

To fix this, scientists invented Feedback Alignment (FA). Instead of sending perfect memos, the boss sends a "rough guess" using a fixed, random set of instructions. The idea is that the workers will eventually learn to ignore the noise and figure out the right direction on their own.

For the last decade, researchers have been checking if these "rough guess" methods work by looking at two things:

Did the team get a good score on the puzzle? (Accuracy)
Did the "rough guess" point in roughly the same direction as the "perfect memo"? (Cosine Similarity)

The Problem:
This paper argues that these two checks are like a broken smoke alarm. They might beep happily even when the house is actually on fire. The authors found that these standard checks can give a "green light" to methods that are actually failing to teach the deep layers of the network. They identified two "silent failures" that the standard checks miss:

1. The "Invisible Floor" Failure (Measurement Degeneracy)

The Analogy: Imagine you are trying to measure the direction of a tiny ant using a giant, clunky ruler. If the ant is so small it's smaller than the smallest mark on your ruler, your ruler just says "zero" or "noise." You can't tell if the ant is pointing North or South; you're just measuring the ruler's own limitations.

What happens in the paper:
In some modern network designs, the "perfect memo" (the reference gradient) gets crushed down to a size so tiny that it hits the computer's "numerical floor" (the smallest number the computer can handle). When this happens, the "rough guess" isn't being compared to a real direction anymore; it's being compared to computer noise. The standard check says, "Hey, the angle looks positive!" but it's actually just measuring static. It's like saying a compass is working because the needle is vibrating, not because it's pointing North.

2. The "One-Worker Hero" Failure (Aggregation Collapse)

The Analogy: Imagine a relay race with 10 runners. You want to know if the whole team is running well. Instead of timing everyone, you just add up their speeds and get an average.

Scenario A: The first runner is a superstar (very fast), but the other 9 are asleep. The average speed looks okay.
Scenario B: The last runner is a superstar, but the first 9 are asleep. The average speed looks the same as Scenario A.

What happens in the paper:
The standard check takes an "average" of how well the credit signal aligns across the whole network. But in these methods, the signal often only works at one end of the network (either the very beginning or the very end) while the middle layers get nothing. The "average" looks positive, hiding the fact that the deep layers (the middle of the team) are completely lost and not learning anything.

The Solution: A New "Health Check" Protocol

The authors propose a new three-step checklist to make sure the network is actually learning, not just faking it:

Check the "Stability" (Scale Stability): Are the numbers inside the network exploding to huge sizes? If the numbers get too big, it's a sign the system is unstable and the "perfect memo" is about to get crushed (leading to Failure #1).
Check the "Reference" (Reference Validity): Is the "perfect memo" actually big enough to be measured? If it's too small (below the computer's noise floor), the direction check is meaningless.
Check the "Teamwork" (Depth Utility): This is the most important one. The authors say: "Freeze the middle workers." Train the network but keep the deep layers frozen at their random starting state. If the network performs just as well with frozen deep layers as it does with trained ones, the deep layers aren't doing any work. The "rough guess" method failed to teach them.

The Results

When they applied this new checklist to Feedback Alignment methods:

The standard checks said: "Everything looks good! Positive angles, decent scores!"
The new checklist said: "Stop! The deep layers aren't learning. The reference is too small to measure, or the middle workers are asleep."

In fact, in some cases, the "rough guess" methods performed worse than just leaving the deep layers frozen and random. They were actively hurting the network, but the old checks didn't see it.

The Takeaway

You can't just look at the final score or a single "direction" number to see if a new learning method works. You have to dig deeper. You need to make sure the signals aren't too small to measure, and you need to prove that the deep layers are actually contributing to the solution, not just along for the ride. Without these extra checks, researchers might be building future AI on top of methods that don't actually work.

Technical Summary: What Accuracy and Gradient Cosine Miss

Problem Statement

Despite the success of deep learning, training deep networks via biologically plausible and hardware-efficient mechanisms remains an open challenge. Feedback Alignment (FA) methods address the constraints of backpropagation (BP)—specifically weight symmetry and sequential updates—by replacing backward weights with fixed random matrices. While FA and its variant, Direct Feedback Alignment (DFA), have shown promise in training modern architectures like transformers, their evaluation relies on a standard reporting pair: task accuracy and aggregate cosine similarity between the method's credit signal and the BP gradient.

This paper argues that this standard reporting pair is insufficient and can silently pass methods that have not actually trained the deep layers of a network. The authors identify two independent failure modes where the standard metrics provide misleadingly positive signals despite the network failing to learn useful credit assignments in deep layers.

Methodology and Diagnostic Protocol

To diagnose these failures, the authors conducted a controlled audit comparing three methods (BP, FA, and DFA) on residual architectures (ResMLP, ViT-Mini, StudentNet) with identical training recipes, differing only in credit propagation rules. They observed that standard metrics failed to detect when deep blocks were effectively unused or detrimental.

Based on these observations, the authors propose a three-check diagnostic evaluation protocol to replace or supplement the standard reporting pair:

Scale Stability (Diagnostic 1): Measures the maximum per-block residual growth ( $\rho = \max_l \|h_{l+1}\|/\|h_l\|$ $ρ = max_{l} ∥ h_{l + 1} ∥/∥ h_{l} ∥$ ) at the end of training.
- Threshold: Flag if $\rho > 50$ .
- Purpose: Detects "residual-stream explosion," a precursor to reference gradient collapse.
Reference Validity (Diagnostic 2): Measures the norm of the deepest BP reference gradient ( $\|g_L\|$ $∥ g_{L} ∥$ ).
- Threshold: Flag if $\|g_L\| < 10 \cdot \epsilon$ (where $\epsilon$ is the cosine implementation's denominator floor, typically $10^{-8}$ for fp32).
- Purpose: Detects Measurement Degeneracy. In terminal-LayerNorm architectures, residual explosion compresses the BP gradient to the numerical floor. When this occurs, the cosine similarity is computed against floating-point noise rather than a meaningful direction, rendering the metric uninterpretable.
Depth Utility (Diagnostic 3): Computes the gap between the trained model's test accuracy and an architecture-matched frozen-blocks baseline (where residual blocks are held at random initialization, and only the embedding, LayerNorm, and head are trained).
- Threshold: Flag if the gap is below 2 percentage points.
- Purpose: Detects whether the trained depth genuinely contributes to prediction. This check is necessary because even valid cosine measurements do not guarantee that updates in the credit direction compose into useful depth training.

The protocol also mandates per-layer cosine reporting rather than aggregate reporting to avoid masking layerwise heterogeneity.

Key Findings and Failure Modes

The paper identifies two causally independent failure modes that the standard metrics miss:

Mode 1: Measurement Degeneracy (Reference Collapse).
- Mechanism: In terminal-LayerNorm residual architectures, the unbounded growth of the residual stream (driven by the local loss form) causes the BP reference gradient to shrink inversely. Eventually, the gradient norm falls below the numerical floor of the cosine implementation.
- Result: The reported cosine similarity is computed against a normalized floating-point residual (noise) rather than the true BP direction. This mode is prevalent in architectures with terminal LayerNorm (e.g., ResMLP, ViT-Mini) but absent in those without (e.g., StudentNet).
Mode 2: Aggregation Collapse (Layerwise Heterogeneity).
- Mechanism: The aggregate cosine averages over per-layer cosines that are concentrated at opposite ends of the network. For example, FA often aligns only at the deepest layers, while DFA aligns only at the shallowest (embedding) layers.
- Result: The aggregate cosine appears positive, suggesting alignment, while the deep layers (or shallow layers, depending on the method) contribute zero aligned credit.
Cosine Does Not Predict Depth Utility.
- Even when Mode 1 is alleviated (via scale penalties) and the cosine measurement is valid, high cosine alignment does not predict whether the deep layers improve accuracy. The authors demonstrated this using "State Bridge" and "Credit Bridge" probes, which showed that methods with lower cosine alignment could achieve higher accuracy than those with higher alignment.

Results

The authors validated the protocol across multiple architectures and settings:

Standard Metrics Fail: In representative settings (e.g., ResMLP on CIFAR-10), FA and DFA reported non-trivial accuracy and positive aggregate cosine. However, the standard pair gave no signal that these methods failed to exceed a frozen-blocks baseline.
Protocol Success: The proposed protocol correctly identified all failures.
- DFA consistently triggered Mode 1 (scale explosion and reference collapse) and Depth Utility failure across architectures with terminal LayerNorm.
- FA often passed Mode 1 checks (reference remained valid) but failed the Depth Utility check, indicating that while the reference was measurable, the credit signal did not effectively train the deep layers.
Causal Independence: A penalty intervention (adding a scale penalty to the local loss) alleviated Mode 1 (restoring reference validity) without immediately fixing Mode 2 (deep-layer alignment remained poor). This confirmed that the two failure modes are distinct and require separate diagnostics.
Generalizability: The failure patterns held across CIFAR-10 and CIFAR-100, and across ResMLP and ViT-Mini architectures.

Significance and Claims

The paper claims that the standard evaluation protocol for Feedback Alignment is fundamentally flawed for deep residual architectures. By relying on accuracy and aggregate cosine, researchers risk building upon methods where the deep layers are non-functional or even detrimental to performance.

The significance of this work lies in:

Identifying Silent Failures: It exposes two independent mechanisms (measurement degeneracy and aggregation collapse) that render standard metrics uninformative.
Providing a Diagnostic Protocol: It offers a concrete, actionable set of checks (Scale Stability, Reference Validity, Depth Utility) that practitioners can use to verify if an FA method genuinely trains deep layers before scaling or deploying.
Clarifying the Role of Cosine: It establishes that cosine alignment to the BP gradient, even when measured validly, is not a sufficient proxy for the functional utility of depth training.

The authors emphasize that passing the protocol does not certify a method as a universal replacement for backpropagation, but failing it indicates that the method has not successfully trained the deep layers of the specific architecture tested. The work aims to prevent the field from scaling non-functional credit assignment mechanisms.