🤖 machine learning

Local Hessian Spectral Filtering for Robust Intrinsic Dimension Estimation

This paper introduces Local Hessian Spectral Dimension (LHSD), a scalable method that robustly estimates Local Intrinsic Dimension in high-dimensional spaces by applying spectral filtering to the log-density Hessian to distinguish tangent directions from noise, thereby enabling effective detection of memorization in large-scale diffusion models.

Original authors: Genki Osada

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Genki Osada

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are standing in a vast, foggy room filled with millions of invisible strings stretching in every direction. Some of these strings are part of a hidden, intricate sculpture (the "manifold") that the data is actually sitting on. Other strings are just random noise floating in the empty space around the sculpture.

Your goal is to figure out how many dimensions the sculpture actually has. Is it a flat 2D sheet? A 3D cube? Or a complex 100-dimensional shape?

This is the problem of Local Intrinsic Dimension (LID) estimation. For a long time, computers tried to solve this by looking at how close data points were to their neighbors (like counting how many people are standing near you in a crowd). But in high-dimensional spaces (like high-resolution images), this method fails because the "crowd" becomes so spread out that everyone looks equally far away.

More recently, scientists started using Diffusion Models (AI that learns to remove noise from images) to solve this. They realized that if you look at how the AI reacts to noise, you can figure out the shape of the data. However, the paper argues that these new methods have a fatal flaw: they get overwhelmed by the "noise strings."

The Problem: The "Noise" Drowns Out the "Signal"

Think of the data as a smooth, flat sheet of paper (the tangent space) floating in a huge, 3D room.

The Signal: If you push the paper along its surface, it moves easily. This represents the "tangent" directions (the actual shape).
The Noise: If you try to push the paper off the surface into the empty room, it hits a massive, invisible wall of resistance. This represents the "normal" directions.

Existing methods tried to measure the "stiffness" of the data by summing up the resistance in every direction. But in high-dimensional spaces, there are thousands of "off-surface" directions and only a few "on-surface" directions. The massive resistance from the off-surface directions completely drowns out the subtle movement of the on-surface directions. It's like trying to hear a whisper (the shape) while standing next to a jet engine (the noise). The result is a broken measurement.

The Solution: LHSD (Local Hessian Spectral Dimension)

The authors propose a new method called LHSD. Instead of listening to the jet engine and the whisper together, LHSD uses a clever trick to filter out the jet engine.

Here is how it works, using a musical analogy:

The Hessian (The Sound Wave): The math behind the AI model creates a "sound wave" of resistance. This wave has many frequencies (eigenvalues).
- The Low Frequencies are the whisper (the smooth surface of the data).
- The High Frequencies are the jet engine (the sharp walls of the noise).
Spectral Filtering (The Equalizer): LHSD applies a digital "equalizer" to this sound wave. It specifically cuts off the high frequencies (the noise) and lets the low frequencies pass through.
Counting the Notes: Once the noise is silenced, the method simply counts how many low-frequency notes are left. That count is the true dimension of the data.

Why It's a Big Deal

The paper highlights three main superpowers of LHSD:

It's Robust: Even in massive, high-dimensional spaces (like images with thousands of pixels), it doesn't get confused by the noise. It successfully ignores the "jet engines" and only counts the "whispers."
It's Fast and Scalable: Calculating the full sound wave for a high-dimensional image usually takes forever (like trying to analyze every single atom in a room). LHSD uses a mathematical shortcut called Stochastic Lanczos Quadrature (SLQ). Think of this as taking a few smart samples of the room to guess the whole shape, rather than measuring every inch. This makes the speed grow linearly with the size of the data, allowing it to handle huge datasets quickly.
It's Verifiable: Unlike other methods that are "black boxes," LHSD gives you a visual dashboard. You can see the "sound wave" and the "filter" on a graph. If the filter is cutting off the right part of the wave, you know the answer is correct. If not, you can adjust the settings until it looks right.

Real-World Tests

The authors tested this on:

Synthetic Shapes: They created fake data shaped like moons, funnels, and cubes. LHSD correctly identified the dimensions of each part, even when they were mixed together in high-dimensional space.
Image Memorization: They used LHSD to detect when an AI model is "cheating" by memorizing training images instead of learning to create new ones.
- The Analogy: If an AI memorizes a photo, it's like a perfect photocopy. It has very little "freedom" or variation (low dimension). If it creates a new, complex image, it has high freedom (high dimension).
- The Result: LHSD successfully spotted the "photocopies" (memorized images) because they had anomalously low dimensions, distinguishing them from normal, complex images.

Summary

In short, previous methods tried to measure the shape of data by listening to everything at once, which failed in complex, high-dimensional environments. LHSD is like a smart noise-canceling headphone that silences the overwhelming background noise, allowing us to clearly hear and count the true dimensions of the data structure. It is faster, more accurate, and easier to trust than the methods that came before it.

Technical Summary: Local Hessian Spectral Filtering for Robust Intrinsic Dimension Estimation

Problem Statement

The estimation of Local Intrinsic Dimension (LID) is critical for understanding the geometric complexity of high-dimensional data manifolds, with applications ranging from anomaly detection to analyzing memorization in generative models. While the manifold hypothesis posits that data concentrates on lower-dimensional structures within high-dimensional ambient spaces, accurately estimating LID in high dimensions (e.g., $D > 1000$ ) remains a significant challenge.

Traditional $k$ -nearest neighbor ( $k$ NN) based methods suffer from the curse of dimensionality, where distance concentration dilutes neighborhood information, leading to severe accuracy degradation. More recently, diffusion model-based approaches (e.g., LIDL, FLIPD) have emerged, leveraging the score function (gradient of log-density) to avoid neighborhood searches. However, these existing diffusion-based methods fail in high-dimensional, high co-dimension settings typical of image data. They indiscriminately sum values (such as score norms or divergences) across all dimensions. In high co-dimensions, the vast normal subspace generates massive curvature signals that overwhelm the tangent signal, resulting in a poor signal-to-noise ratio and significant estimation errors.

Methodology: Local Hessian Spectral Dimension (LHSD)

The authors propose Local Hessian Spectral Dimension (LHSD), a method that robustly estimates LID by explicitly separating tangent directions (signal) from normal directions (noise) using spectral filtering on the log-density Hessian.

Theoretical Foundation

Under the manifold hypothesis, for a point $x$ near a $d$ -dimensional manifold in $\mathbb{R}^D$ , the perturbed distribution $p_t(x)$ (induced by Gaussian noise with scale $\sigma(t)$ ) exhibits a polarized Hessian spectrum:

Tangent Directions: Correspond to the tangent space $T_x\mathcal{M}$ . The eigenvalues $\lambda_i$ are small, scaling as $O(1)$ .
Normal Directions: Correspond to the normal bundle. The eigenvalues $\lambda_j$ are large, scaling as $O(1/\sigma(t)^2)$ .

This separation creates a spectral gap. The intrinsic dimension $d$ corresponds to the multiplicity of near-zero eigenvalues (the null space of the Hessian).

Algorithmic Approach

LHSD estimates LID by counting the tangent directions via a smooth spectral filter $f(\lambda)$ applied to the Hessian eigenvalues:
$\text{LID}(x) \approx \sum_{i=1}^D f(\lambda_i) = \text{tr}(f(H(x)))$
where $H(x) = -\nabla_x s_\theta(x, t)$ is the Hessian of the negative log-density, derived from the score model $s_\theta$ .

Key Components:

Spectral Filtering: The authors employ a Hill-type filter:
$f(\lambda; \sigma(t)) = \frac{1}{1 + (|\lambda|/\kappa(t))^p}$
This filter maps tangent eigenvalues (small $\lambda$ ) to $\approx 1$ and normal eigenvalues (large $\lambda$ ) to $\approx 0$ . The cutoff $\kappa(t) = c/\sigma(t)^2$ is time-varying to normalize the curvature scaling inherent in the normal directions.
Scalable Computation (SLQ): Constructing the full $D \times D$ Hessian is computationally prohibitive ( $O(D^3)$ ). LHSD utilizes Stochastic Lanczos Quadrature (SLQ) to estimate the trace $\text{tr}(f(H))$ without explicit matrix construction. By computing Hessian-Vector Products (HVP) via automatic differentiation and using the Lanczos algorithm to generate a tridiagonal matrix, the method achieves linear computational complexity $O(D)$ .
Verifiable Parameter Selection: A diagnostic metric, Transition Mass $M(t)$ , is introduced to identify a "safe zone" for the noise scale $t$ . This ensures the filter cutoff $\kappa(t)$ resides within the spectral gap between tangent and normal clusters, allowing for visual verification of parameter correctness.

Key Contributions

Robust LID Estimator: LHSD selectively counts tangent dimensions via Hessian spectral filtering, suppressing the influence of normal eigenvalues. This ensures robustness in high-dimensional and high co-dimension regimes where previous methods fail.
Verifiable Hyperparameter Selection: The method provides a diagnostic indicator (Transition Mass) and a visual mechanism (overlaying filter profiles on eigenvalue distributions) to validate parameter choices ( $c, p, t$ ), addressing the "black box" nature of previous estimators.
Scalable Algorithm: By integrating spectral filtering with SLQ, LHSD bypasses explicit Hessian construction, achieving $O(D)$ linear complexity. This enables fast inference even for dimensions exceeding 3,000.
Extensive Empirical Validation: The method is validated on synthetic manifolds, real-world image datasets, and specific benchmarks for diffusion model memorization.

Experimental Results

The paper evaluates LHSD against $k$ NN-based methods (ESS, LPCA) and diffusion-based methods (LIDL, FLIPD, Normal Bundle) across various settings:

High-Dimensional Manifolds: In experiments with ambient dimensions up to $D=3072$ and varying intrinsic dimensions, LHSD consistently achieved the lowest Mean Absolute Error (MAE). For instance, on the $F10+80+200 \subset \mathbb{R}^{3072}$ dataset, FLIPD yielded an MAE of 1240.89, whereas LHSD achieved 5.19.
IDR Benchmarks: On the Intrinsic Dimension Recovery (IDR) benchmarks (embedding 3D manifolds into Fashion-MNIST space, $D=784$ ), existing diffusion methods suffered dramatic performance degradation. LHSD maintained stable behavior with minimal error increase, even with a minimal Lanczos rank ( $m=2$ ).
Computational Efficiency: LHSD demonstrated a speedup of over $250\times$ compared to FLIPD. While FLIPD requires $D$ backpropagations (quadratic complexity), LHSD requires only a small number of HVPs (linear complexity).
Memorization Detection: In an experiment detecting memorized samples in a DDPM trained on CIFAR-10, LHSD successfully distinguished memorized samples (which exhibit anomalously low LID) from non-memorized samples with similar visual complexity. The method achieved an AUROC of 99.87%, demonstrating its utility in identifying overfitting in generative models.

Significance and Claims

The paper claims that LHSD represents a fundamental advancement in LID estimation by addressing the specific failure mode of existing diffusion-based methods: the inability to separate signal from noise in high co-dimensions.

Robustness: The primary significance lies in the method's ability to remain accurate in regimes where the ambient dimension far exceeds the intrinsic dimension, a common scenario in image data.
Interpretability: Unlike previous methods, LHSD offers a mechanism for practitioners to visually diagnose and verify that the underlying manifold structure is being correctly captured, providing a level of interpretability absent in prior approaches.
Scalability: The linear scaling with dimension makes it the only proposed method capable of efficiently handling high-dimensional data (e.g., $D > 3000$ ) without the prohibitive costs associated with SVD or full Hessian construction.

The authors acknowledge a limitation: the method relies on the quality of the trained score model. While they assess training fidelity, establishing rigorous diagnostics for the spectral consistency of score models on complex real-world data remains an open problem.