Are You Measuring the Failure Subspace, or Its Ceiling?

Technical Summary: "Are You Measuring the Failure Subspace, or Its Ceiling?"

Problem Statement
A prevailing empirical claim in Test-Time Adaptation (TTA) and Low-Rank Adaptation (LRA) literature is that the subspace relevant for adapting a model—defined by the span of per-sample gradients, Fisher directions, or "failure directions"—is surprisingly low-dimensional. Methods such as GOLD [7] and various LRA techniques rely on the observation that a small number of eigenvectors capture the vast majority of the energy in these gradient-like vectors ("demands").

However, this paper argues that for the specific demand definitions commonly used (cross-entropy gradients, entropy gradients, and the Average Gradient Outer Product or AGOP), this "low-rank" observation is not an empirical discovery regarding the data or the failure mode, but an algebraic identity. These demands are mathematically confined to the row space of the final classifier weight matrix $W$. Consequently, their covariance rank cannot exceed the number of classes $C$ (or $C-1$ for zero-sum demands). The paper posits that reported low-rank structures may simply be a rediscovery of this algebraic ceiling rather than evidence of a compact, intrinsic failure dimension.

Methodology
To distinguish between an algebraic ceiling and a genuine low-dimensional structure, the authors employ a controlled experimental design:

1. Algebraic Proofs: The paper establishes three propositions proving the rank ceilings for common demands:
   • Cross-Entropy (CE) & Entropy Gradients: Confined to a hyperplane of dimension at most $C-1$ due to the zero-sum property in logit space.
   • AGOP: Confined to the row space of $W$, with a rank ceiling of at most $C$.
2. Ceiling-Free Demand: To measure what lies beyond these ceilings, the authors construct a "free" demand ($d_{act}$) based on activation-space shares ($A \odot -\nabla_A L$). This demand is not an image of $W^\top$ and can theoretically occupy the full ambient feature dimension $D$.
3. Controlled Sweep: Experiments are conducted on nested subsets of CIFAR-100, varying the number of classes $C$ from 2 to 100 while holding the architecture, feature width ($D$), per-class sample count, and optimization protocol constant.
4. Width Sensitivity Test: The authors double the final representation width ($D$ from 256 to 512) at a fixed $C=100$ to test if the measured rank is a property of the task or the measurement apparatus.
5. Decomposition: Free demand vectors are projected onto the row space of $W$ ($P_{row(W)}d$) and its orthogonal complement to analyze the energy distribution between the "ceiling" component and the "tail."

Key Results

• Capped Demands are Pinned to Their Ceiling: For CE, entropy, and AGOP demands, the effective rank ($r_{90}$) tracks the algebraic ceiling closely. At $C=100$, these demands occupy 55–65% of their maximum possible rank ($C$ or $C-1$). As $C$ increases, the fraction of the ceiling occupied actually decreases, indicating that the apparent low-rank structure at large $C$ is a result of the ceiling "emptying out," not the emergence of new structure.
• The Free Demand Exceeds the Ceiling and is Width-Sensitive: The ceiling-free demand exhibits an effective rank that exceeds $C$ (e.g., 41.7 at $C=20$) and grows with $C$ at a rate roughly double that of the capped demands. Crucially, when the feature width $D$ is doubled, the effective rank of the free demand increases significantly (from 111 to 191), whereas the capped demands remain relatively stable. This suggests the rank of the free demand is a function of the ambient space size, not just the task difficulty.
• Decomposition Reveals a Broad Tail: Projecting the free demand onto $row(W)$ reveals that ~74% of its energy resides within the classifier-induced subspace, reproducing the exact spectral cliff at index $C$ seen in capped demands. The remaining ~26% of the energy forms a "broad, weakly structured, knee-free tail" that spans hundreds of dimensions. This tail's rank tracks the size of the ambient space rather than a compact intrinsic dimension.
• Re-evaluation of GOLD: The paper re-analyzes the GOLD method, noting that its claim of "strong low-rank concentration" (top 64–128 eigenvectors capturing >99% energy on CIFAR-100-C) is tautological given the algebraic ceiling of $C=100$. The reported numbers reflect the occupancy of the classifier-induced subspace, not a discovery of a hidden low-dimensional manifold in the full feature space.

Significance and Claims
The paper concludes that the statement "the adaptation subspace is surprisingly low-rank" conflates two distinct objects:

1. An Algebraic Fact: Logit-derived demands are confined to $row(W)$ and cannot exceed rank $C$. This is a theorem, not a finding.
2. A Hypothetical Data Property: An intrinsic, compact failure dimension. The authors find no evidence of such a dimension in their experiments; instead, removing the algebraic ceiling reveals a broad, width-sensitive tail.

Proposed Shift in Reporting:
The authors argue that reporting "effective rank" alone is insufficient and potentially misleading for ceiling-bound demands. They propose that future work in this area should report:

• Ceiling-Normalized Effective Rank ($\rho_{ceil}$): The ratio of effective rank to the algebraic ceiling, separating empirical spectral decay from algebraic constraints.
• Row-Space Energy Share ($\rho$): The fraction of demand energy lying within $row(W)$. This metric is width-stable and directly quantifies the maximum energy accessible to any method restricted to the classifier's row space (which includes most current TTA/LRA methods).

The paper does not claim that low-rank methods are ineffective, but rather that their success should not be attributed to a "hidden low-dimensional structure" in the data, but rather to the fact that they operate within the algebraic constraints of the classifier. The utility of the "tail" (the energy outside $row(W)$) remains an open question.