🤖 machine learning

Measuring Dead Directions: Decomposing and Classifying Singular Structure off Canonical Alignment

This paper introduces a deterministic, alignment-free method to measure and classify singular structures in trained neural networks by extracting direction-specific learning orders from the directional-Fisher rate at a single checkpoint, thereby recovering global learning coefficients and mapping singular fluctuations without requiring canonical alignment or descent preconditions.

Original authors: Tejas Pradeep Shirodkar

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tejas Pradeep Shirodkar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, complex machine (a neural network) that has been trained to do a job. Over time, as it learns, some of its internal parts stop working or become redundant. In the world of deep learning, these are called "dead directions."

This paper introduces a new way to measure and understand these dead parts without having to restart the machine or run it through a long, slow process. Here is the breakdown using simple analogies:

1. The Problem: Finding the "Ghost" Parts

Think of a trained neural network like a giant, tangled ball of yarn. Some strands are tight and doing work (live parts), while others are loose, slack, or completely useless (dead parts).

The Old Way: To find these useless strands, scientists used to have to pull the yarn apart slowly (descent) or align it perfectly with a ruler (canonical alignment) to see where the slack was. If the yarn was twisted or the ruler was crooked, they couldn't find the dead parts.
The New Way: This paper offers a "flashlight" method. You can shine a light on the machine exactly as it sits right now (a frozen checkpoint), even if it's twisted or messy, and instantly identify the dead parts.

2. The Tool: The "Directional Fisher" Scan

The authors use a tool called the Directional Fisher. Imagine you have a specific direction in the machine you want to test.

The Test: You gently push the machine slightly in that direction and see how much the "error" (the difficulty of the job) changes.
The Result:
- If the machine is alive in that direction, the error changes immediately (like pushing a stiff spring).
- If the machine is dead in that direction, the error stays flat for a while before it starts to rise.
- The "Order" ( $k$ ): The paper measures how long that flat spot lasts. This duration is called the "order."
  - A short flat spot means the part is just slightly dead.
  - A long, deep flat spot means the part is completely dead (a "singularity").
  - This "order" tells us exactly how much complexity the machine has lost in that specific spot.

3. The Twist: It Works Even When Twisted

Usually, to measure this, the dead part had to be perfectly aligned with the machine's internal grid (like a straight line on graph paper). But real machines often twist these dead parts so they are diagonal or rotated.

The Analogy: Imagine trying to measure the length of a shadow. If the sun is at a weird angle, the shadow is crooked. Old methods said, "We can't measure this unless the sun is directly overhead."
The Innovation: This new method says, "We don't care about the angle." It can construct the correct "shadow" (the dead direction) mathematically from the machine's internal data and measure it perfectly, even if it's rotated 45 degrees.

4. Sorting the Dead Parts: "Real Death" vs. "Fake Death"

The paper also teaches us to tell the difference between two types of "dead" directions:

Genuine Death (Node-Death): A part that actually stopped working because the machine learned it didn't need it. This is a permanent loss of complexity. The machine "forgot" this feature.
Gauge Symmetry (Fake Death): A part that looks dead but is actually just a rule of the machine's design. It's like a door that looks locked but is actually just a decorative panel that was never meant to open. It doesn't count as a loss of learning; it's just a feature of the architecture.
The Solution: The method looks at the "depth" of the flatness. Genuine death has a specific mathematical signature; fake death (gauge) sits at a different level. The paper separates them automatically.

5. Why This Matters (According to the Paper)

Speed: It's incredibly fast. You don't need to retrain the model or run complex simulations. You just take a snapshot and run a quick calculation.
Clarity: Instead of getting one vague number for the whole machine's complexity, this method breaks it down. It tells you: "Here are 5 dead parts of type A, and 3 dead parts of type B."
Accuracy: It works on different types of machines (Transformers, Convolutional networks) and different layers, proving that the "order" of death is determined by the machine's design (like the type of activation function used) rather than just random chance.

Summary

This paper gives us a deterministic, instant X-ray for neural networks. It allows us to look at a trained AI, find exactly which parts have become useless, measure how useless they are, and distinguish between parts that truly died from learning and parts that were just designed to be decorative—all without needing to align the machine perfectly or run it through a slow training loop.

Technical Summary: Measuring Dead Directions off Canonical Alignment

Problem Statement
Singular learning theory characterizes trained neural networks via the Watanabe triple: the learning coefficient ( $\lambda$ ), multiplicity ( $m$ ), and singular fluctuation ( $\nu$ ). These invariants govern Bayesian free energy and generalization error. However, measuring them on real networks is costly and often relies on stochastic posterior sampling (e.g., SGLD), which yields a single global scalar without localizing the structure to specific network coordinates. Furthermore, previous deterministic methods for reading the "order" ( $k$ ) of singularities required two restrictive preconditions: canonical alignment (the dead direction must align with coordinate axes at every layer) and descent (the reading must occur along a training trajectory approaching the singularity). Real trained networks often violate these conditions, leaving dead directions rotated off the axes or forming diffuse structures that standard per-coordinate scans miss.

Methodology
The paper proposes a descent-free, alignment-free measurement pipeline that operates at a single frozen checkpoint. The method consists of two stages: Detection and Reading.

Detection (Architecture-Adaptive):
- The detector locates the "dead direction" ( $u$ ) by identifying the near-kernel of the layer's natural second-moment object.
- For generic layers, it utilizes the K-FAC (Kronecker-factored approximate curvature) factor pair ( $A \otimes G$ ), exploiting the A-G duality: if the input covariance ( $A$ ) fails to isolate the direction, the output-gradient covariance ( $G$ ) succeeds, and vice versa.
- For convolutional layers, it uses a spatial-patch covariance; for LayerNorm, it uses the closed-form kernel of the normalization scale.
- Crucially, the method constructs the joint mode from these factors rather than searching the Fisher spectrum, avoiding the "gauge floor" (flat symmetry directions) and sampling null spaces that lie below the order-carrying mode.
Reading (Descent-Free Scan):
- Once a direction $u$ is nominated, the method performs a synthetic scan: $\theta(t) = \theta_0 + tu$ , where $\theta_0$ is the frozen checkpoint.
- It measures the directional Fisher $F(\theta(t))$ along this scan.
- The order $k$ is recovered from the growth rate of the Fisher: $u^\top F(\theta(t)) u = \Theta(t^{2(k-1)})$ .
- A purity-matched window selector isolates the asymptotic regime ( $t \to 0$ ) by maximizing the fit quality ( $r^2 > 0.95$ ), ensuring the recovered exponent is not contaminated by lower-order components or gauge noise.
- The method classifies directions based on magnitude and slope: a finite slope indicates a genuine singularity (e.g., node-death, depth singularity), while a flat magnitude at the "floor" indicates a gauge symmetry (e.g., LayerNorm kernel, rotation).

Key Contributions

Alignment-Free Order Recovery: The method recovers the per-direction order $k$ (and thus the local learning coefficient $\lambda_{dir} = 1/(2k)$ ) without requiring the dead direction to align with coordinate axes. It successfully reads rotated node-deaths in deep networks where standard scans fail.
Taxonomy of Dead Structure: The paper classifies dead directions into genuine singularities (finite order determined by activation analytic order or network depth) and flat gauges (infinite order, contributing to multiplicity but not complexity). It introduces a magnitude-based criterion to distinguish curved gauge orbits (which mimic finite orders in slope) from true singularities.
Deterministic Decomposition: Unlike posterior samplers that return a single global $\lambda$ , this method decomposes the complexity into per-direction orders and the dead-subspace dimension (the count of directions at the dominant order).
Optimizer-Dependent Geometry: The study demonstrates that while the order is fixed by the architecture (activation function or depth), the basis and sharpness of the dead structure are determined by the optimizer. Orthogonalizing optimizers produce clean, readable structures, while standard optimizers may leave structures too diffuse or rotated to be read by canonical methods.
Mapping the Watanabe Triple:
- Order ( $k$ ): Determines the universal singular fluctuation $\nu(k)$ .
- Multiplicity ( $m$ ): Recovered from the dead-subspace dimension under a normal-crossing assumption.
- Fluctuation ( $\nu$ ): The paper shows that while the order fixes the theoretical $\nu(k)$ , a trained network's realized $\nu$ is suppressed below this universal value because the "live" structure absorbs the data fluctuation intended for the dead direction.

Results

Validation on Constructed Cells: The method recovers predicted orders exactly ( $k=2$ for GELU/ReLU node-deaths, $k=3$ for squared-ReLU, $k=L$ for depth- $L$ linear networks) with $r^2 \approx 1.000$ .
Real Networks:
- From-Scratch ViT: Successfully recovers the activation-predicted order ( $k \approx 2$ or $3$) for rotated node-deaths in MLP blocks, whereas per-coordinate scans yield deviant values.
- Fine-Tuned DINOv2: Identifies no node-deaths but classifies the dead structure as architectural gauges (LayerNorm-kernel, attention QK rotation), confirming the method's ability to distinguish between learned singularities and symmetries.
- Optimizer Comparison: Vanilla Muon often leaves dead structures too diffuse to read, while scaled-polar orthogonalizers produce clean, axis-aligned or rotated structures that the method reads accurately.
Global Coefficient: On analytic models where the singular structure is enumerable, the assembled per-direction orders match the closed-form global $\lambda$ to machine precision.

Significance and Claims
The paper claims to turn order recovery into a deterministic, architecture-general reading that removes the canonical-alignment and descent preconditions of prior work. It provides a "cheap" (single forward/backward pass) alternative to expensive posterior sampling for decomposing the learning coefficient.

The authors emphasize modesty regarding the scope:

The read is conditional on a detector successfully nominating the direction; it does not find structure in random directions.
It characterizes the structure left by the optimizer; if an optimizer leaves a diffuse solution (e.g., vanilla Muon on deep transformers), there is no clean order to recover.
While it recovers the order and local coefficient exactly, the realized singular fluctuation in trained networks is suppressed by live-structure absorption, meaning the universal $\nu(k)$ is an upper bound, not the exact realized value.
The method currently handles "enumerable" structures; for complex determinantal varieties (e.g., wide matrix deep linear networks), the global assembly may still require posterior sampling or more advanced algebraic resolution.

In summary, the paper establishes a practical framework for "reading" the singular geometry of trained networks at a frozen checkpoint, offering a detailed, per-direction map of complexity that complements global sampling methods.