🤖 machine learning

How fine a change can moments see? A scale law for detecting distribution shift, with a kernel calibration rule

This paper establishes a theoretical scale law linking the fineness of distribution shifts to the required polynomial degree for detection, demonstrating that a bandwidth-calibrated kernel test outperforms both moment-based statistics and topological methods in identifying high-dimensional embedding changes.

Original authors: Adel Kaleche

Published 2026-08-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Adel Kaleche

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard watching a massive, swirling crowd of people. Your job isn't to count heads; it's to spot when the crowd's shape suddenly changes. Maybe a group of people who were standing in a tight circle suddenly break apart to form a ring with a hole in the middle, or a long line of people decides to loop back on itself to form a figure-eight. In the world of artificial intelligence, these "people" are data points called embeddings—mathematical representations of things like sentences, images, or sounds. When the AI's understanding of the world shifts (a "distribution shift"), these points move around.

For a long time, scientists have tried to catch these shifts by looking at simple statistics, like the average position of the crowd (the mean) or how spread out they are (the variance). But what if the crowd changes its shape in a way that keeps the average and the spread exactly the same? That's where topology comes in. Think of topology as the study of "holes" and "loops." A coffee mug and a donut are topologically the same because they both have one hole; a ball has zero holes. Topological data analysis (TDA) tries to count these holes to see if the data has changed. The big question is: Is looking for these "holes" a better way to catch trouble than just checking the math of the crowd's spread? And if so, how do we tune our tools to see them?

This paper, written by independent researcher Adel Kaleche, dives deep into that question. The author sets up a high-stakes game of cat-and-mouse. On one side, there's a "defender" trying to spot changes in data streams. On the other, a clever "adversary" trying to sneak a change past the defender without tripping any alarms. The paper introduces a new "Scale Law"—a rule of thumb that acts like a physics law for how hard it is to see a change.

The core discovery is a bit of a reality check for the topological approach. The paper proves that detecting a specific feature (like a hole or a loop) depends entirely on how fine or small that feature is, not on how many features there are. Imagine trying to spot a tiny pebble in a pile of sand. If the pebble is huge, you can see it easily. If it's microscopic, you need a very powerful microscope. The paper shows that to spot a tiny feature of size $\epsilon$ , you need a mathematical "lens" (a test) with a specific level of power. If the feature is very small, the math required to see it becomes incredibly expensive and complex.

The author tests this law against the "hole-counting" method (persistent homology) and finds that for the kinds of changes usually seen in AI data, the topological method is often overkill. In fact, the paper reveals a surprising trick: the best way to spot a change isn't always a complex hole-counting algorithm. Instead, the "Scale Law" predicts that a simpler tool—a kernel test (specifically an MMD test using a Gaussian kernel)—is the most efficient detective, but only if you set its "zoom level" (bandwidth) correctly. The paper measures this and finds that the perfect zoom level is almost exactly the size of the change itself (a ratio of about 1.12).

Here is the twist: The paper explicitly argues against the idea that topological summaries are the magic bullet for all data shifts. Through a series of rigorous tests, the author shows that:

Simple math often wins: For "coarse" changes (big, obvious shifts), simple statistics like kurtosis (which measures how "peaked" or "flat" a distribution is) work just as well as complex topological methods.
The "Hole" is a trap: The paper provides a counterexample where a ring of data (which has a hole) looks mathematically identical to a solid disk (which has no hole) when you check the mean, variance, and even the fourth-order moments. This proves that you cannot simply say "fourth-order math sees all holes." Sometimes, the hole is invisible to standard math, but the paper argues that in real-world AI attacks, the changes usually follow a pattern where simple math does work.
Cost matters: The topological method is incredibly expensive. The paper calculates that using the topological summary (specifically the "first landscape") costs about 116 times more computing power than using kurtosis, yet it often performs worse. Even the better topological summary ("total persistence") only catches up to the cheap math methods, never beating them significantly, while still costing a fortune.
The Adversary wins against everything but the tuned kernel: When the "adversary" is smart enough to trick the mean, the variance, the density, and even the kurtosis, the topological methods fail completely. The only thing that still detects the change is the kernel test, but only if the researcher sets the bandwidth (zoom) to match the size of the change.

The paper is very careful about what it claims. It does not say topological methods are useless forever. It says that for the specific task of monitoring AI data streams for shifts, they are currently dominated by cost and performance by a simpler, well-tuned kernel test. The "Scale Law" tells us why: detecting fine details is hard, and the topological method tries to see everything at once, which is inefficient. The paper concludes that if you want to catch a change, don't just throw a complex topological net at it; instead, use the Scale Law to figure out how big the change is, and tune your simpler detector to that exact size. It's a lesson in knowing your enemy's size and choosing the right tool, rather than using the most expensive tool in the shed.

Technical Summary: A Scale Law for Detecting Distribution Shift

Problem Statement
The paper addresses the challenge of detecting distribution shifts in streams of high-dimensional embeddings. While standard approaches frame this as a choice of statistic (e.g., covariance drift, density estimation, or topological summaries like persistent homology), the author argues that the efficacy of any moment-based detector is fundamentally constrained by the spatial scale ( $\varepsilon$ ) of the change and the mass fraction ( $f$ ) it carries, rather than the number of features or the dimensionality of the space. The work specifically investigates whether topological methods (Persistent Homology) offer a distinct advantage over moment-based statistics in detecting these shifts, particularly under adversarial conditions.

Methodology and Theoretical Framework
The core of the paper is the derivation and empirical validation of a scale law governing the complexity required to detect distributional changes.

The Scale Law (Lower Bound):
- The author proves that certifying a feature of spatial scale $\varepsilon$ carrying a mass fraction $f$ requires polynomial tests of degree $N^*$ satisfying:
  $N^* \ge \frac{\log(1/f)}{2\varepsilon}$
- This bound is derived via the Chebyshev extremal problem. It implies that the cost of detection is governed by the fineness of the feature, not the count of features.
- For a topology of complexity $b$ (e.g., $b$ loops), a lower bound of $N^* \ge 4b - 1$ is established using Gauss quadrature arguments.
The Upper Bound and Calibration Rule:
- While the lower bound is strict for polynomial tests, the paper identifies that Gaussian test functions (the RKHS witness of an RBF kernel) can achieve a rate of $N^* \propto \log(1/f)/\varepsilon$ .
- This leads to a practical calibration rule: The bandwidth ( $\sigma$ ) of a Maximum Mean Discrepancy (MMD) test should be set to match the feature scale ( $\sigma \approx \varepsilon$ ).
Counterexamples and Limitations:
- The law is one-sided: it provides a lower bound on the necessary order but never certifies sufficiency.
- A specific counterexample is constructed: an annulus with non-zero thickness carries non-trivial first homology ( $H_1 \neq 0$ ) yet shares its mean, covariance, and all moments up to order four with a filled disk. This refutes the idea that "fourth-order information governs canonical topological change" in all cases.
Empirical Evaluation:
- Datasets: Real embedding streams (MiniLM, bge-1024) from 20 Newsgroups and AG News corpora.
- Adversary: A gradient-based adversary optimized to evade specific defender statistics ( $\mu, \Sigma$ , k-NN, kurtosis) while preserving the target topology.
- Baselines: Covariance drift, Mahalanobis distance, k-NN density, Mardia kurtosis, and various topological summaries (First Persistence Landscape, Total Persistence, etc.).
- Protocol: Sliding windows ( $W=200$ ), PCA reduction to intrinsic dimension, and evaluation at fixed False Positive Rates (FPR) and Average Run Lengths (ARL).

Key Results

Validation of the Scale Law:
- Experiments confirm that the optimal MMD bandwidth $\sigma^*$ is approximately equal to the feature scale $\varepsilon$ (median ratio $\sigma^*/\varepsilon = 1.12$ , IQR $[1.01, 1.52]$ ).
- A data-driven estimate of $\varepsilon$ allows the MMD test to reach an AUC $\ge 0.95$ .
- Against an adaptive adversary optimized against $\{\mu, \Sigma, \text{k-NN}, \text{kurtosis}\}$ , all those statistics are evaded. Only the bandwidth-matched kernel test (MMD-RBF) successfully detects the shift.
Performance of Persistent Homology (PH):
- Summary Choice is Critical: The performance of PH is dominated by the choice of summary statistic.
  - First Persistence Landscape: Performs poorly (AUC $\approx 0.75$ , recall 0.00 at FPR=1%). Its performance degrades with ambient dimension and is sensitive to reduction dimension.
  - Total Persistence: Performs significantly better (AUC $\approx 0.99$ , recall 0.75 at FPR=1%), matching the performance of kurtosis in non-adversarial settings.
- Redundancy: The "Total Persistence" summary behaves structurally like a low-order moment (a readout of the radial profile). In the regime of coarse, single-scale changes (the focus of the study), it is redundant with kurtosis but incurs a massive computational cost (116 $\times$ kurtosis).
- Adversarial Failure: Against the adaptive adversary, both Total Persistence and kurtosis fail simultaneously. The only surviving detector is the bandwidth-matched kernel test.
Cost vs. Power:
- There is no "power gap" where PH detects what moments miss in this specific task; rather, there is a cost gap.
- PH (even with the best summary) costs roughly 116 times more than kurtosis and fails under the same adversarial conditions.
- The "degradation with ambient dimension" observed in some metrics is a property of the landscape summary, not of persistent homology itself.

Significance and Claims
The paper concludes with a modest but precise set of claims:

The Scale Law: It establishes a theoretical lower bound on the moment order required to detect features of a given scale, proving that detection cost is a function of feature fineness, not feature count.
Calibration Rule: It provides a concrete, empirically validated rule for setting the bandwidth of kernel two-sample tests ( $\sigma \approx \varepsilon$ ), which is critical for their success against adaptive adversaries.
Verdict on Topology: The paper does not claim topological summaries are useless. However, it concludes that for coarse, single-scale monitoring (the regime of the experiments), topological summaries are dominated by kernel tests on cost.
- The summary that works (Total Persistence) essentially acts like a moment, aligning with the law's prediction that low-order moments suffice for coarse features.
- The paper suggests that the genuine value of persistent homology lies in detecting fine or multi-scale structures where the equivalent moment order would be prohibitively high, a regime not present in the tested embedding shifts.

In summary, the work argues that for distribution shift detection in embeddings, the choice of statistic is constrained by the scale of the shift. The optimal practical approach is a kernel test with bandwidth matched to that scale, while topological methods offer no advantage in cost or power for the specific class of coarse deformations tested.

Technical Summary: A Scale Law for Detecting Distribution Shift

More like this