Technical Summary: Neural Low-Degree Filtering (Neural LoFi)

Problem Statement

The theoretical understanding of how deep neural networks learn useful internal representations from data remains a central open problem. While empirical evidence suggests that deep networks progressively build structured representations (e.g., from edge detectors to semantic motifs), existing theoretical frameworks often fail to explain the mechanism of feature selection and organization across layers.

• The Lazy Regime (NTK): Describes training as kernel regression in a fixed feature space, bypassing feature learning entirely.
• Mean-Field/Infinite-Width Limits: Capture parameter evolution but often require complex dynamical systems or specific parametrizations to analyze feature learning.
• The Gap: There is a lack of a simple, predictive mechanism explaining which features are discovered, in what order, at what sample complexity, and why depth aids in learning compositional functions.

Methodology: Neural Low-Degree Filtering (Neural LoFi)

The authors propose Neural LoFi, a stylized, mathematically tractable limit of gradient-based training that transforms hierarchical feature learning into an explicit iterative spectral procedure.

1. Theoretical Derivation

Neural LoFi is derived from the early dynamics of gradient descent under a layer-wise vanishing initialization scheme ($a_L \ll W_{L-1} \ll \dots \ll W_1$). In this regime, the dynamics of neurons in a layer decouple and are governed by a linear approximation involving two terms:

1. Linear Drift: A constant drift along the mean direction $\hat{u}_\ell = \frac{1}{n}\sum y_\mu z_{\ell-1}(x_\mu)$.
2. Spectral Filtering: A linear projection along a label-weighted second-order moment operator $\hat{C}^{(\ell)} = \frac{1}{n}\sum y_\mu z_{\ell-1}(x_\mu)z_{\ell-1}(x_\mu)^\top$.

The dynamics approximate a power iteration that exponentially amplifies directions corresponding to the largest eigenvalues of $\hat{C}^{(\ell)}$. This leads to the hypothesis that early training acts as a spectral filter selecting the top eigenvectors of this label-weighted operator.

2. The Algorithm

Neural LoFi replaces implicit backpropagation with a layer-wise, backpropagation-free procedure (Algorithm 1):

1. Filter Step: Given the current representation $z_{\ell-1}$, compute the label-weighted moment operator $\hat{C}^{(\ell)}$. Project the representation onto the top $k_\ell$ eigenvectors of $\hat{C}^{(\ell)}$ (and the linear component $\hat{u}_\ell$).
2. Lift Step: Apply a nonlinear random feature map (random weights $R_\ell$ and activation $\sigma$) to the projected features to generate the next representation $z_\ell$.
3. Readout: Fit a final linear or logistic readout on the final layer.

3. Variational Characterization

The method is characterized as solving a variational problem at each layer: selecting features that maximize the low-degree correlation with the label while remaining simple (low RKHS norm) in the geometry induced by the previous layer.

• Linear features: Maximize $|\mathbb{E}[y \psi(x)]|$.
• Second-order features: Maximize $|\mathbb{E}[y \phi(x)^2]|$ subject to orthogonality to previous features.

Key Contributions

1. A Tractable Surrogate for Feature Learning

Neural LoFi provides a mathematically explicit framework for studying multi-layer feature learning beyond the lazy regime. It decouples layers, allowing for a layer-by-layer analysis of representation selection without the complexity of full backpropagation dynamics.

2. The Relevance–Complexity Trade-off

The paper establishes that feature learning is governed by a trade-off where the next layer selects features that have large low-degree correlation with the label but are simple in the current representation's geometry. This is formalized through the RKHS norm constraint.

3. A Quantitative Criterion for Feature Emergence

The authors derive a criterion for when a new feature (concept) emerges during training. A feature becomes learnable when its population correlation $\rho$ exceeds the empirical noise floor $\tau$, which is controlled by the residual effective dimension of the current kernel:
$$\rho^{(k)}_\ell \gg \tau^k_\ell(n) \approx \sqrt{\frac{D^{\text{eff}}_{\ell,k}}{n}}$$
This predicts the sample complexity scale at which specific task-relevant directions become detectable, analogous to phase transitions in spiked matrix models (BBP/EA transitions).

4. Principle of Low-Degree Compositionality

The paper proposes that depth is beneficial not merely for representational capacity, but for low-degree compositionality. A deep network is efficient if the target function can be decomposed such that each intermediate representation exposes a low-degree signal (e.g., linear or quadratic) relative to the current layer. Depth allows the network to transform high-degree structures in the input into low-degree structures in intermediate representations, making them statistically detectable.

5. Adaptive Kernel Construction

Unlike fixed kernel methods (e.g., NTK) that use a single geometry, Neural LoFi constructs a sequence of task-adaptive kernels ($K_0 \to K_1 \to \dots \to K_L$). Each transition is supervised, building the next kernel from low-complexity features in the previous layer that are most correlated with the label.

Results and Empirical Validation

1. Solvable Hierarchical Models

In a mathematically solvable teacher-student model with a planted compositional hierarchy, Neural LoFi successfully recovers hidden representations.

• Emergence: The model exhibits a sharp drop in prediction error and a spike in feature overlap exactly at the predicted sample complexity scale ($n \propto d^{k+\epsilon}$).
• Spectral Outliers: The leading eigenvalues of the moment operator separate from the spectral bulk at the predicted thresholds, confirming the emergence of informative features.

2. Real-Data Experiments (CIFAR-10)

Experiments on fully connected (FCN) and convolutional (CNN) architectures for binary classification (Animals vs. Vehicles) show:

• Performance: Neural LoFi matches or exceeds early-stage gradient descent (GD) performance, particularly in low-data regimes, despite being a one-pass spectral procedure.
• Feature Discovery: LoFi recovers meaningful structured filters (e.g., edge detectors, color sensitivity) without backpropagation.
• Alignment: The features learned by LoFi align with the initial feature discovery phase of SGD. As training progresses, GD moves beyond the LoFi approximation, but LoFi captures the initial spectral structure.
• Emergence Prediction: The effective-dimension criterion accurately predicts the sample size at which individual eigenvectors (concepts) become learnable, matching empirical transitions in feature overlap.

3. Kernel Interpretation

The paper demonstrates that Neural LoFi converges to a deterministic infinite-width limit where the learned features correspond to eigenfunctions of a task-adaptive kernel, validating the kernel-space interpretation of the algorithm.

Significance and Claims

The authors position Neural LoFi not as a replacement for backpropagation, but as a tractable surrogate and a theoretical lens for understanding deep learning.

• Mechanistic Insight: It offers a concrete mechanism for how depth constructs new features from old ones through low-degree compositionality.
• Bridging Theory and Practice: It connects abstract spectral theory (Hessian-like operators, effective dimension) with practical phenomena like feature emergence and the success of deep networks.
• Backpropagation-Free Learning: The algorithm itself serves as a simple, layer-wise, backpropagation-free primitive for feature discovery, sitting between fixed random-feature models and full end-to-end training.
• Pruning and Overparameterization: The framework suggests that large widths are necessary for the discovery of task-correlated directions, while spectral pruning (selecting top eigenvectors) controls the effective dimension for prediction, reconciling overparameterization with pruning.

The paper concludes that a significant portion of deep feature learning is captured by supervised low-degree spectral structure, providing a unified view of feature emergence, sample complexity, and the computational advantage of depth.