📊 statistics

Measuring and Decomposing Mode Separation via the Canonical Diffusion

This paper introduces a novel framework for quantifying and decomposing mode separation in high-dimensional distributions by leveraging a canonical reversible diffusion process to extract barrier-sensitive metrics (SSA) and metastability-ordered projections (DA) from sample data and score functions, thereby overcoming the limitations of traditional tools like entropy and PCA.

Original authors: Shaul Tolkovsky, Ori Meidler, Or Zuk

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shaul Tolkovsky, Ori Meidler, Or Zuk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a jar of marbles. Sometimes, the marbles are all mixed together in one big, fluffy cloud. Other times, they are clumped into distinct, separate piles with empty space between them.

In the world of data science, this "clumping" is called mode separation. It's a fundamental question: How sharply does a distribution of data break apart into distinct groups?

The problem is that our usual tools for measuring data are bad at spotting these "empty spaces."

Entropy (a measure of spread) is like a ruler that only measures how wide the jar is. It can't tell if the marbles are one wide cloud or two tight piles.
PCA (a common method to find patterns) is like looking for the direction where the marbles are most spread out. If you have two tight piles side-by-side, PCA might just tell you "they are wide," missing the fact that there's a gap right in the middle.
Mutual Information is a great tool, but it requires you to already know exactly which pile each marble belongs to. In real life, we usually don't have that label.

This paper introduces a new way to measure these gaps using a concept called Canonical Diffusion. Here is the simple breakdown of how it works and what it found.

The Core Idea: The "Drunkard's Walk"

Imagine you drop a single marble into your jar and watch it bounce around randomly (like a drunk person stumbling). This is a "diffusion process."

If the jar has one big cloud of marbles, the marble will wander everywhere quickly. It mixes fast.
If the jar has two separate piles with a deep valley between them, the marble will get stuck in one pile for a long time before it randomly stumbles across the gap to the other pile. It mixes slowly.

The authors created a specific, unique "drunkard's walk" for any dataset. They then measured two things about how this marble moves over time:

SSA (The "Stuckness" Score): This is a single number. It measures how long the marble tends to stay in one area before wandering off.
- Analogy: Think of it like a "memory" score. If the marble remembers where it started for a long time, the score is high. This means there are strong barriers (gaps) keeping the data separated.
- Result: Unlike entropy, this score goes up when the data is fragmented into distinct groups, even if the groups are small.
DA (The "Direction" Finder): This tells you which way to look to see the separation.
- Analogy: If you have two piles of marbles separated by a canyon, PCA might tell you to look left-to-right because the whole jar is wide. DA tells you to look straight across the canyon because that's where the "stuckness" happens. It finds the specific direction where the data is most likely to be split in two.

How They Did It (The Magic Trick)

Usually, to simulate this "drunkard's walk," you need to know the exact mathematical formula for the data distribution, which is often impossible to get in high dimensions (like images with millions of pixels).

The authors used a trick involving AI image generators (specifically "score-based" models like SDXL). These models are trained to "denoise" images. The authors realized that the math used to remove noise is mathematically identical to the "drunkard's walk" they needed. They used the AI's "denoising brain" as a map to guide their marble, allowing them to measure separation in massive, complex datasets without needing the raw formula.

What They Found

They tested this on three different scenarios:

Fake Data (Gaussian Mixtures):
They created fake data with known gaps. The new method (SSA) perfectly tracked how "separated" the data was, matching the gold-standard "Mutual Information" metric, while entropy failed to see the difference between a wide cloud and two tight piles.
AI-Generated Images (SDXL):
They asked an AI to generate pictures of "a person," "a cat," or "a crane" at different settings.
- The Finding: When the AI was set to a "medium" setting, the images were diverse but clearly separated into distinct types (e.g., different breeds of cats or different poses of people). The SSA score peaked here.
- The Contrast: When the AI was set to "high" settings, the images became very similar to each other (low entropy), but the SSA score dropped, correctly identifying that the "gaps" between different types of images had disappeared.
- The Direction: For the prompt "a portrait of an elderly person," the new method (DA) found a direction that separated men from women. The old method (PCA) mixed gender with style and realism, failing to isolate the specific difference.
Molecules (Alanine Dipeptide):
This is a tiny molecule that folds into different shapes. Scientists know there are two main "slow" ways it moves (like twisting a hinge).
- The Finding: Using only static snapshots of the molecule (no video of it moving), the new method (DA) successfully identified the exact two directions (hinges) that control the molecule's movement. It matched the results of methods that usually require expensive, long-running simulations.

Summary

The paper presents a new "ruler" (SSA) and a new "compass" (DA) for data.

SSA tells you how fragmented your data is, ignoring how wide it is.
DA tells you where the fragmentation is happening.

It works by simulating a random walk through the data, using AI denoisers as a shortcut to navigate high-dimensional spaces. It successfully distinguished between "spread out" data and "broken apart" data in synthetic tests, AI image generation, and molecular physics, outperforming traditional tools like entropy and PCA in identifying the true structure of complex data.

Technical Summary: Measuring and Decomposing Mode Separation via the Canonical Diffusion

1. Problem Statement

The paper addresses the difficulty of quantifying mode separation—the geometric property of how sharply a probability density fragments into barrier-separated clusters—in high-dimensional spaces. Existing tools fail to distinguish between dispersion (how widely mass is spread) and fragmentation (how mass is split by barriers):

Differential entropy increases with spread regardless of whether the distribution is unimodal or fragmented.
PCA orders directions by variance, failing to distinguish between a wide single mode and two narrow, separated modes.
Mutual information requires access to the mixture decomposition (latent labels), which is typically unavailable in practice.

The authors seek an intrinsic, continuous quantity sensitive to barriers and a method to identify the linear directions along which these barriers are most pronounced, using only samples and a score function.

2. Methodology

The Canonical Diffusion

The core of the framework is a unique reversible Itô diffusion attached to a target density $f$ . This process is defined by two constraints:

Stationarity: $f$ is the stationary distribution.
Constant Scalar Diffusion: The diffusion matrix is $A_0 = \sigma_f^2 I_d$ , where $\sigma_f^2 = \text{tr}(\text{Cov}_f(X))/d$ is the scalar variance of $f$ .

Under these conditions, the drift is uniquely determined as $b(x) = \frac{1}{2}\sigma_f^2 \nabla \log f(x)$ . The generator of this diffusion has a discrete spectrum $0 = \mu_1 < \mu_2 \le \mu_3 \dots$ , where small eigenvalues correspond to slow relaxation modes crossing high energy barriers.

Two Readouts from Autocovariance

The framework extracts two quantities from the stationary autocovariance matrix $C(\tau) = \mathbb{E}_f[Y_\tau Y_0^\top]$ of the centered process $Y_t = X_t - \bar{x}_f$ :

SSA (Sum of Squared Autocorrelations):
A scalar measure of multimodality defined as the time integral of the squared trace-normalized autocorrelation:
$S(f) = \int_0^\infty \rho(\tau)^2 d\tau, \quad \text{where } \rho(\tau) = \frac{\text{tr}C(\tau)}{\text{tr}C(0)}$
Spectrally, $S(f) = \sum_{k,\ell \ge 2} \frac{w_k w_\ell}{\mu_k + \mu_\ell}$ . It is monotone in the generator eigenvalues and saturates with the number of modes, effectively measuring the "barrier height" rather than just the count.
DA (Dominant Autocorrelation directions):
A directional decomposition obtained via the eigendecomposition of $C(\tau)$ at a specific lag $\tau$ . The eigenvectors $v_j(\tau)$ are ordered by eigenvalue magnitude. As $\tau \to \infty$ , the leading eigenvectors converge to the directions connecting mode centers (specifically, the projection coefficients of the slow eigenfunctions), ordered by metastability rather than variance.

Theoretical Guarantees and Noise Floor

Null Spectrum (Theorem 2): Under an isotropic Gaussian null, the empirical autocovariance spectrum converges to the free additive convolution of two scaled Marchenko–Pastur laws. This yields a closed-form analytic upper edge $\lambda_+(\tau)$ , which serves as a sharp noise floor to distinguish signal eigenvalues from finite-sample noise.
Lag Selection: The noise floor $\lambda_+(\tau)$ allows for the principled selection of a finite lag $\tau^*$ where slow modes are still detectable above the noise, balancing spectral convergence (requiring large $\tau$ ) against signal-to-noise ratio (degrading at large $\tau$ ).

Implementation with Score-Based Models

The method does not require the density $f$ explicitly, only the score $\nabla \log f$ . For high-dimensional data (e.g., images), the authors utilize pretrained score-based generative models (like SDXL) via Tweedie's identity, which provides the score $\nabla \log f_t(x) \approx -\epsilon_\theta(x, t)/\sigma_t$ . This allows the canonical diffusion to be simulated in latent spaces or high-dimensional feature spaces without explicit density estimation.

3. Key Contributions

Canonical Diffusion Construction: A unique, similarity-invariant stochastic process defined solely by the score and first two moments of a density, designed specifically to reveal barrier structures.
SSA and DA Metrics:
- SSA: A scalar measure that tracks mutual information in synthetic mixtures and captures fragmentation missed by entropy.
- DA: A linear decomposition that recovers slow, metastable directions (e.g., separating modes) even when they are not the directions of maximum variance (unlike PCA).
Analytic Noise Theory: The derivation of the limit spectrum for the symmetrized lag-1 autocovariance under a Gaussian null, providing a closed-form threshold $\lambda_+(\tau)$ analogous to the Marchenko–Pastur law for PCA.
Score-Based Integration: A practical pipeline for applying these concepts to high-dimensional data using pretrained denoisers, bypassing the curse of dimensionality in kernel-based methods.

4. Experimental Results

Synthetic Gaussian Mixtures

SSA vs. Mutual Information (MI): In 10D Gaussian mixtures, SSA tracks the rank ordering of MI across various parameter sweeps (separation, variance, weight, mode count). Unlike MI, which saturates as barriers deepen, SSA grows exponentially with barrier height.
SSA vs. Entropy: In a 2D comparison, SSA correctly identified a bimodal distribution (parallel ellipses) as more fragmented than a unimodal isotropic Gaussian with higher entropy, whereas entropy and PCA failed to distinguish the fragmentation.
DA vs. PCA: On the parallel ellipses, DA recovered the direction connecting the two modes (the inter-mode barrier), while PCA selected the direction of maximum within-mode variance.

SDXL Text-to-Image Generations

Guidance Scale Analysis: Applied to SDXL outputs across 11 prompts and 4 guidance scales ( $w$ $w$ ).
- Entropy declined monotonically as guidance increased (distribution became more concentrated).
- SSA peaked at intermediate guidance ( $w \approx 7.5$ ), capturing the "sweet spot" where the distribution fragments into distinct, coherent types (e.g., different styles of a person) before collapsing into near-identical repeats at high guidance.
Directional Decomposition: For the prompt "a portrait of an elderly person," DA1 separated the distribution along a gender axis (Spearman $|\rho|=0.90$ ), whereas PC1 mixed gender with realism and style. For "a photo of a crane," both DA1 and PC1 captured the split between the bird and construction equipment senses, demonstrating that DA aligns with PCA only when the fragmentation axis coincides with the variance axis.

Molecular Dynamics (Alanine Dipeptide)

Recovery of Slow Coordinates: Using static samples from a 22-atom alanine dipeptide system, DA recovered the known slow backbone dihedral coordinates ( $\phi, \psi$ ) without access to time-series trajectory data.
Comparison to TICA: DA achieved the same ordering of slow coordinates as Time-lagged Independent Component Analysis (TICA), a standard trajectory-based method, but using only static samples and a score oracle derived from a kernel density estimate.

5. Significance and Claims

The paper claims to provide a fundamental geometric tool for measuring mode separation that is structurally distinct from dispersion. Its significance lies in:

Bridging Static and Dynamic Analysis: It allows the extraction of metastable structure (typically requiring long trajectories) from static samples by leveraging the spectral properties of a canonical diffusion.
High-Dimensional Applicability: By relying on score-based models, it scales to dimensions where kernel methods (like Diffusion Maps) fail, enabling the analysis of complex generative model outputs and molecular landscapes.
Theoretical Rigor: The provision of an analytic noise floor ( $\lambda_+(\tau)$ ) offers a principled, data-driven method for selecting lags and determining the number of metastable directions, addressing a key limitation in spectral clustering and slow-mode analysis.

The authors remain modest, noting that the framework is linear (cannot capture nonlinear mode geometry) and depends on the quality of the input score. They position the work as a diagnostic tool for existing generative models and a method for extracting slow collective variables from static data, rather than a new generative capability itself.