📊 statistics

Heteroscedasticity of Denoising Score Matching with Generalised Smooth Noise

This paper reveals that Denoising Score Matching (DSM) suffers from inherent heteroscedasticity due to noise levels and data geometry, and proposes a theoretically derived weighting function to stabilize training variance while providing a justification for existing heuristics in diffusion models.

Original authors: Juyan Zhang, Rhys Newbury, Xinyang Zhang, Tin Tran, Dana Kulic, Michael Burke

Published 2026-08-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Juyan Zhang, Rhys Newbury, Xinyang Zhang, Tin Tran, Dana Kulic, Michael Burke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers learn to create art, music, or even new molecules by studying a giant, messy library of existing examples. To do this, they use a clever trick called "Score Matching." Think of the data (like a picture of a cat) as a landscape with hills and valleys. The "score" is simply a compass that always points uphill toward the most likely places to find a cat. If the computer can learn to hold this compass perfectly, it can wander through the landscape and eventually find a brand-new, realistic cat to draw.

But here's the catch: the computer can't see the whole map at once. It's like trying to learn the shape of a mountain while standing in a thick fog. So, instead of looking at the real mountain, the computer practices on a version of the mountain that has been covered in static noise, like a TV screen full of snow. It tries to guess how to clean the noise off. This practice method is called "Denoising Score Matching" (DSM). For a long time, scientists assumed this practice was a perfect, free substitute for the real thing. They thought, "If the average direction of the compass is right, we're good to go." But this paper asks a nagging question: Is the practice field actually hiding a secret trap that makes learning unstable?

The authors of this paper, a team from Monash University and Amazon, have discovered that the practice field is indeed a bit of a trickster. They found that Denoising Score Matching is inherently "heteroscedastic." That's a fancy word for saying the amount of "noise" or uncertainty in the computer's learning signal changes wildly depending on where it is in the data landscape.

To use a playful analogy, imagine you are trying to learn to throw darts at a moving target. In a perfect world, every throw would be equally hard or easy. But in this "Denoising" game, some throws are like tossing a dart in a calm room, while others are like trying to throw while standing on a shaking boat in a storm. The paper proves that the "shaking boat" parts happen naturally in specific regions of the data, like the edges between different clusters of information. Because the computer doesn't know which throws are on the boat and which are on solid ground, it treats them all the same. This causes the learning process to wobble and become inefficient, like a student trying to study for a test while someone keeps turning the lights on and off.

The researchers didn't just find the problem; they built a theoretical "stabilizer" to fix it. They derived a special mathematical formula, called a "Godambe weighting," that acts like a smart filter. This filter tells the computer, "Hey, that throw was on a shaking boat; don't trust it as much. But that other throw was on solid ground; pay extra attention to that." By adjusting the importance of each piece of information based on how shaky it is, the computer can learn much more smoothly.

However, there is a twist. The perfect filter requires knowing the exact shape of the shaking boat, which is often impossible to calculate for complex, high-dimensional data (like real-world images). So, the authors also proposed a "good enough" approximation. They showed that a simple, existing trick used by many modern AI models—weighting the learning by the square of the noise level—actually emerges naturally from their math. This explains why that simple trick works so well in practice, even though it's not the perfect solution.

Ultimately, the paper reveals a fundamental trade-off. You can have a mathematically perfect, statistically efficient learning method, but it might be too unstable to train in the real world. Or, you can use a slightly less perfect, "approximate" method that keeps the training stable and gets the job done. The authors prove that the popular methods we use today are essentially making a smart compromise: they sacrifice a tiny bit of statistical perfection to avoid the chaos of the "shaking boat," ensuring the AI can actually learn to create those amazing images and sounds we see today.

Technical Summary: Heteroscedasticity of Denoising Score Matching with Generalised Smooth Noise

Problem Statement
Denoising Score Matching (DSM) has become the standard tractable proxy for Score Matching (SM) in modern generative modeling, particularly for energy-based and diffusion models. While foundational work by Vincent (2011) established that the expected gradients of DSM and SM objectives are equivalent, leading to the assumption that DSM is a "free" substitute, this paper argues that this equivalence is incomplete. The authors demonstrate that DSM is not merely a noisy approximation of SM but introduces a fundamental statistical difference: inherent heteroscedasticity. In DSM, the variance of the regression targets (the denoising scores) fluctuates unpredictably based on both the noise level and the local geometry of the underlying data distribution. This input-dependent variance creates an irreducible noise floor that destabilizes training dynamics and leads to statistically inefficient estimators if left unaddressed.

Methodology
The paper develops a rigorous statistical framework to analyze and correct for this heteroscedasticity across arbitrary order score functions and general smooth noise distributions (extending beyond the standard Gaussian assumptions).

Theoretical Derivation of Heteroscedasticity:
- The authors derive an iterative recursion for arbitrary-order score functions (Theorem 3.3) applicable to general smooth noise distributions.
- Using a Pythagorean decomposition of the $L_2$ loss, they prove that the DSM objective contains an irreducible variance term ( $tr \, E[V_{x_0|xt}(h_k)]$ ) that does not exist in the true SM objective (Theorem 3.5).
- They show that this term depends dynamically on the input $x_t$ and noise level $t$ , acting as an "unavoidable, underlying noise floor" that destabilizes parameter gradients and model parameters (Lemma 3.8).
Godambe-Optimal Weighting:
- To counteract this heteroscedasticity, the authors derive the variance-optimal (Godambe) weighting function (Lemma 3.10). This ideal weight matrix $W$ is the inverse of the conditional variance of the denoising target ( $W = V_{x_0|xt}[h_k]^{-1}$ ).
- Applying this weight theoretically renders the objective homoscedastic and recovers the Best Linear Unbiased Estimator (BLUE) in ideal settings.
- However, the authors note that this optimal solution depends on intractable quantities (conditional higher-order moments or local curvature) in high-dimensional settings, making exact correction impractical.
Practical Approximation:
- Recognizing the computational intractability of the optimal weights, the authors propose a first-order Taylor expansion approximation (Lemma 3.12).
- This approximation relies on the Fisher information of the noise distribution rather than the data-dependent curvature.
- Crucially, they identify a fundamental tradeoff: while the ideal weighting minimizes the asymptotic variance of the estimator, it can increase the variance of stochastic gradients during training. Conversely, the approximation reduces gradient variance (improving optimization stability) at the cost of statistical efficiency (Lemma 3.14).

Key Contributions

Intrinsic Heteroscedasticity of DSM: The paper proves that DSM induces input-dependent variance determined by noise levels and data geometry, establishing a fundamental statistical gap between DSM and SM (Theorem 3.5).
Optimal Weighting and Limits: The authors derive the Godambe-optimal weighting to restore homoscedasticity but demonstrate its reliance on intractable quantities, highlighting the practical limits of exact correction (Lemma 3.10).
Theoretical Grounding for Heuristics: The framework generalizes to arbitrary score orders and noise distributions. It provides a theoretical justification for the widely used $\sigma_t^2$ weighting heuristic in isotropic Gaussian diffusion, showing it emerges as a specific instance of the proposed approximation (Lemma 3.12).
Statistical vs. Optimization Tradeoff: The work clarifies that statistically optimal weighting can increase gradient variance, while heuristic approximations improve optimization stability but yield suboptimal estimators (Lemma 3.8).

Results
The authors validate their theory through experiments on synthetic and standard datasets:

Second-Order Score Estimation: On a Gaussian Mixture Model (GMM), unweighted DSM exhibited severe instability and poor convergence. Both the ideal and Taylor-approximated weighting schemes stabilized training. However, the ideal weighting completely prevented late-stage divergence observed in the approximation at low noise levels.
Non-Gaussian Noise: Experiments on a 1D Mixture of Cauchy dataset corrupted by various noise families (Cauchy, Generalized Normal, Laplace, Student-t) showed that training without weights resulted in severe performance degradation (high MAPE). Applying the derived first-order optimal weights substantially reduced the Mean Absolute Percentage Error (MAPE) across all noise distributions.
Gradient Variance: Empirical analysis of model parameter gradients revealed that while ideal weights minimize parameter variance asymptotically, the approximated (heuristic) weights achieve significantly lower gradient variance during training, particularly as noise levels increase. This confirms the theoretical tradeoff between statistical efficiency and optimization stability.
High-Dimensional Datasets: On standard generative datasets (Spiral, Moons, MNIST, CIFAR-10), incorporating the first-order approximation or Hutchinson-based curvature estimates provided minor but consistent improvements in distribution modeling (lower MAPE) compared to unweighted baselines.

Significance and Claims
The paper claims to fundamentally shift the understanding of DSM from a "free" substitute for SM to a distinct estimation problem with its own noise structure. By formalizing DSM as a heteroscedastic estimation problem, the work:

Explains the empirical success of existing heuristics (like $\sigma_t^2$ weighting) while clarifying their limitations in capturing complex local geometries.
Provides a principled framework for deriving weighting functions for non-Gaussian and higher-order diffusion models, filling a gap where heuristic strategies were previously absent.
Highlights that improving DSM requires balancing statistical efficiency against optimization stability, suggesting that future work must navigate this tradeoff rather than optimizing for one in isolation.

The authors remain modest, acknowledging that their approximation relies on Fisher information and ignores residual heteroscedasticity from complex target geometries. They note that translating these stabilized objectives to efficient generation in non-Gaussian settings will require specialized samplers to handle potential discontinuities in Stochastic Differential Equations (SDEs).

More like this