Technical Summary: Dead-Direction Signatures (DDS)

Problem Statement

Singular Learning Theory (SLT) characterizes the complexity of overparameterized deep networks through the local geometry of their loss singularities. The primary invariant in this framework is the Real Log Canonical Threshold (RLCT, denoted $\lambda$), which measures the effective complexity of a model near a singular minimum. In practice, the Local Learning Coefficient (LLC) has emerged as the canonical estimator for $\lambda$. However, LLC estimation is computationally prohibitive, requiring $10^4$ to $10^6$ forward-backward passes per checkpoint via Stochastic Gradient Langevin Dynamics (SGLD) sampling to integrate over the local posterior. This cost renders LLC impractical for large-scale models (e.g., LLMs) or high-frequency monitoring during training.

The paper addresses the need for a "cheap" alternative to read singular structure without the overhead of posterior integration. Specifically, it seeks a method to detect and quantify "dead directions"—parameter directions along which the loss landscape is flat (singular)—using closed-form spectral readings of network activations and gradients.

Methodology: Dead-Direction Signatures (DDS)

The authors introduce Dead-Direction Signatures (DDS), a family of closed-form spectral observables derived from a theoretical framework (Shirodkar, 2026) that links the decay rate of the Fisher information along dead directions to the RLCT.

Theoretical Foundation

The framework posits that at a singular minimum, the Fisher information matrix $F(\theta)$ loses rank along specific "dead directions" $u$. If the Kullback-Leibler (KL) divergence decays as $K(\theta_0 + tu) \sim t^{2k}$ (where $k$ is the KL order), the directional Fisher information decays as $\Theta(t^{2(k-1)})$. The selection rule connects this decay exponent to the directional contribution of the RLCT ($1/2k$).

The framework establishes a structural correlation between the activation spectrum and the Fisher-Gram spectrum at any singular minimum. For an $L$-layer network with KFAC-decomposable Fisher information $F_\ell \approx A_\ell \otimes G_\ell$:

1. Fisher-side: The smallest positive eigenvalue of the per-sample-gradient Fisher-Gram, $\lambda^+_{\min}(G_\ell)$, decays as $\Theta(t^{2(L-\ell)})$.
2. Activation-side: The squared smallest singular value of the activation matrix, $\sigma^2_{\min}(X_\ell)$, decays as $\Theta(t^{2\ell})$.
3. Correlation: These two quantities are co-monotonic, implying a Spearman rank correlation approaching $+1$ in the asymptotic limit.

The Three DDS Observables

DDS replaces the SGLD posterior chain with three cheap, closed-form readings at a chosen layer $\ell$:

1. Rate Observable ($\lambda^+_{\min}(G_\ell)$): The smallest strictly positive eigenvalue of the Fisher-Gram. It detects the presence of dead directions and localizes them to specific layers (e.g., bottlenecks).
2. Volume Observable ($\log \det^+(G_\ell)$): The log-volume of the active Fisher spectrum (sum of logs of positive eigenvalues). This observable is sensitive to the number of dead directions (rank-deficit $r$).
3. Dual Observable ($\sigma_{\min}(X_\ell)$): The smallest singular value of the activation matrix. It provides an activation-side proxy for the Fisher-side rate, computable via a single forward pass and SVD.

Key Contributions

1. Detection and Rank-Deficit Counting:

   • Detection: $\lambda^+_{\min}(G_\ell)$ collapses by orders of magnitude (e.g., $\sim 246\times$) the instant a dead direction appears, allowing detection from a single backward pass.
   • Counting: The paper introduces a rank-multiplicative volume identity. While single-eigenvalue monitors are "rank-blind," the slope of $\log \det^+(G_\ell)$ scales linearly with the rank-deficit $r$. Specifically, the slope ratio relative to a rank-1 baseline equals $r$ (e.g., slopes of 2.0, 3.1, 4.0 for $r=2, 3, 4$). This allows DDS to count dead directions, a capability single-eigenvalue monitors lack.

2. Validation Against Closed-Form RLCT:

   • On the Aoyagi (2005) reduced-rank regression anchor (where analytical $\lambda$ is known), DDS observables rank-track the analytical $\lambda$ with high correlation ($|\rho| \in [0.95, 0.98]$).
   • The authors demonstrate that while cross-cell rank-tracking is a "sanity gate" (even naive capacity proxies pass it), the rank-multiplicative volume identity serves as a discriminating test that naive proxies cannot reproduce.

3. Extension to Non-Linear Transformers:

   • On a Nanda modular-addition transformer width sweep, DDS observables separate model widths ($d_{\text{model}}$) across 18 orders of magnitude with sign coherence ($|\rho| \in [0.62, 0.96]$).
   • In contrast, calibrated LLC at the same protocol budget is "rank-flat" (unable to distinguish widths), and even a 9x increase in SGLD budget fails to resolve the widths.

4. Cost Efficiency:

   • DDS observables are orders of magnitude cheaper than LLC. $\sigma_{\min}(X_\ell)$ requires one forward pass and an SVD ($O(Nh^2)$). $\lambda^+_{\min}(G_\ell)$ and $\log \det^+(G_\ell)$ require a backward pass and eigendecomposition ($O(h^3)$).
   • On small models, DDS is $\sim 300\times$ cheaper than calibrated LLC; on LLM widths, the structural correlation allows recovering Fisher-side ordering from the cheap activation-side $\sigma_{\min}$, bypassing the expensive $O(h^3)$ Fisher eigendecomposition.

Results

• Closed-Form Anchor: On 14 cells of the Aoyagi 2005 anchor, calibrated LLC recovers analytical $\lambda$ at 99% mean. DDS observables rank-track $\lambda$ with $|\rho| \ge 0.95$.
• Volume Identity: Across 7 noisy-bridge configurations, the $\log \det^+(G)$ slope ratio tracks the rank-deficit $r$ with means of 2.0, 3.1, and 4.0 for $r=2, 3, 4$ (predicted 2, 3, 4).
• Transformer Width Sweep: DDS observables show monotonic descent with increasing $d_{\text{model}}$ (spanning 5–15 orders of magnitude), whereas LLC remains flat.
• Structural Correlation: The static identity $\lambda^+_{\min}(G_\ell) \propto \sigma^2_{\min}(X_\ell)$ holds across diverse testbeds (canonical bridges, Barak sparse parity, Nanda modular addition) with correlations $\rho \ge +0.75$, even in regimes where trajectory rate-fits fail (e.g., AdamW+CE).
• Grokking Dynamics: In Barak sparse parity, DDS observables detect the geometric descent into the singular minimum (dropping $\sim 10$ orders of magnitude) during the "cleanup" phase, while LLC plateaus.

Significance and Scope

The paper claims that DDS provides the first directional, parameter-local handle on singular complexity that is computationally feasible at scale.

• Complementarity: DDS is presented as complementary to LLC. While LLC integrates over the posterior to provide Bayesian summaries (e.g., WAIC, singular fluctuations, developmental plateaus), DDS offers a "directional, layer-local" reading of the singular structure in closed form.
• Limitations: The authors are modest about the scope. DDS is validated on parametric singular models with closed-form RLCT and one non-linear transformer width sweep. The trajectory rate-predictions (e.g., specific power-law exponents) require "canonical alignment" (typically SGD on G-invariant metrics) and may fail under Adam-class optimizers due to non-equivariance. However, the static structural correlation holds even under AdamW+CE.
• Future Work: The paper identifies extending these reads to broader transformer architectures, attention-only chains, and other algorithmic tasks as open empirical work. It also suggests that the same dead-direction reads could eventually extend to the rest of the Watanabe triple (multiplicity $m$, fluctuation $\nu$) and WAIC without SGLD.

In summary, DDS trades the integrated posterior reading of LLC for cheap, closed-form spectral readings, enabling the detection and counting of dead directions at a fraction of the computational cost, thereby making singular complexity analysis tractable for larger models and higher-frequency monitoring.