Fourier fractal dimension to predict the generalization of deep neural networks

Technical Summary: Fourier Fractal Dimension to Predict the Generalization of Deep Neural Networks

Problem Statement
The paper addresses the fundamental challenge in machine learning of predicting the generalization performance of deep neural networks without relying on hold-out validation data. Current approaches often require removing annotated data for validation, which can introduce bias or reduce the effective training set size. Furthermore, existing theoretical bounds (e.g., from statistical learning theory) are often vacuous for highly overparameterized networks. While recent literature has explored complexity measures based on weight norms, margins, and PAC-Bayesian frameworks, there remains a need for metrics that consistently correlate with the generalization gap across varying architectures and datasets without external data.

Methodology
The authors propose a novel generalization measure based on the Fourier fractal dimension of the network's weight variations. The methodology is grounded in the following theoretical and practical steps:

1. Lévy-Driven Dynamics: The paper models the optimization process of Stochastic Gradient Descent (SGD) not as a standard Gaussian process, but as a discretization of a stochastic differential equation (SDE) driven by heavy-tailed, non-Gaussian noise. Specifically, the weight evolution $W_t$ is modeled using $\alpha$-stable Lévy motion.
2. Characteristic Function Analysis: By analyzing the characteristic function of the Lévy-driven SDE in the frequency domain, the authors derive a relationship between the spectral exponent $\alpha$ and the fractal dimension. The characteristic function $\phi(u)$ is treated as the Fourier transform of the random process.
3. Fractal Dimension Estimation: The method involves computing the log-variation of weights ($\log(W_{t+1}/W_t)$) across training epochs. The power spectral density of these variations follows a power-law relationship $P(\omega) \propto |\omega|^{-\beta}$. By fitting the slope of the log-log plot of the magnitude of the Fourier transform against frequency, the authors extract the parameter $\alpha$, which serves as the proxy for the Fourier fractal dimension.
4. Fourier-Based Optimizer: To demonstrate the utility of this metric, the authors introduce a customized optimizer. This algorithm enforces a reduction in the magnitude of the Fourier transform of the parameters at each epoch. This acts as an active regularizer, aiming to reduce the overall Fourier dimension during training, thereby theoretically improving generalization.

Key Contributions

• Novel Generalization Measure: The paper introduces a metric derived from the Fourier fractal dimension of weight variations, which captures the geometric complexity of the learning process induced by stochastic dynamics.
• Theoretical Connection: It establishes a link between the generalization gap and the scale-invariant properties of parameter updates modeled by Lévy-driven SDEs, leveraging the characteristic function to estimate fractal dimensions.
• Regularization Algorithm: The proposal of a Fourier-based optimizer that actively regulates the fractal dimension during training, offering a principled approach to stabilizing optimization without external validation sets.
• Empirical Validation: Extensive experiments on CIFAR-10, SVHN, and MNIST datasets demonstrate the efficacy of the proposed measure.

Results
The empirical evaluation compares the proposed Fourier generalization measure against a wide array of existing metrics, including norm-based, margin-based, and PAC-Bayesian measures (referenced from prior work [5]).

• Correlation with Generalization Gap: The proposed method achieved the highest Kendall rank correlation coefficients ($\tau$) across all three datasets:
  • CIFAR-10: $\tau = 0.680$
  • SVHN: $\tau = 0.672$
  • MNIST: $\tau = 0.551$
• Comparison: These results significantly outperform competitive baselines such as PAC-Bayes flatness (e.g., 0.613 on CIFAR-10) and inverse margin (0.473 on CIFAR-10).
• Layer Analysis: Heatmaps of Kendall coefficients across different layer combinations indicate that deeper layers tend to exhibit stronger correlations with the generalization gap, suggesting that higher-level semantic representations are critical for model generalizability.
• Optimizer Performance: The Fourier-based optimizer was shown to maintain competitive error rates while smoothing the loss landscape, resulting in a more stable convergence profile compared to standard SGD.

Significance and Claims
The paper claims that frequency-domain fractal analysis offers a powerful predictor for model generalizability that operates purely on internal training dynamics. By modeling SGD as a Lévy-driven process, the authors argue that the resulting fractal dimension provides a robust geometric characterization of the learning trajectory. The work highlights the potential of this approach not only for predicting generalization but also as a foundation for developing more stable optimization algorithms that actively regulate the complexity of the parameter space. The authors emphasize that this method eliminates the need for validation data, providing a self-contained metric for understanding and improving the learning process.