📊 statistics

From Score Approximation to Distribution Approximation in Score-Based Diffusion Models

This paper establishes a rigorous quantitative connection between score function approximation and distribution generation in score-based diffusion models by proving that accurate neural network approximation of the score function guarantees a small Kullback-Leibler divergence between the generated and target distributions, with the error explicitly bounded by the score approximation error, noise schedule, and terminal prior mismatch.

Original authors: Lan V. Truong

Published 2026-07-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lan V. Truong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to paint a masterpiece, but you can't show it the final picture. Instead, you give it a blurry, noisy version of the art and ask it to guess how to "un-blur" it, step by step, until the image is clear again. This is the heart of score-based diffusion models, a cutting-edge technology in artificial intelligence that has revolutionized how computers generate images, music, and even molecular structures. The secret sauce here is something called the score function. Think of the score function as a magical compass that always points the robot in the direction of "more likely" or "less noisy" data. If the robot is standing in a foggy field, the score function tells it, "Take a step this way, and the fog will get thinner."

For a long time, scientists knew that neural networks (the robot's brain) were incredibly good at learning to hold this compass. Famous math theorems proved that if you give a neural network enough neurons, it can mimic almost any function, including this score compass, with perfect precision. However, a big question hung over the field: Just because the robot learns the compass perfectly, does that guarantee it will actually paint the masterpiece? In other words, does a perfect approximation of the direction guarantee a perfect approximation of the final picture? Until now, this link was a bit of a mystery, leaving researchers unsure if the robot's internal compass was truly translating to the quality of the art it produced.

This paper, titled "From Score Approximation to Distribution Approximation in Score-Based Diffusion Models," steps in to solve that mystery with a rigorous mathematical proof. The authors, Lan V. Truong, demonstrate that if a neural network approximates the true score function (the compass) accurately enough, then the final distribution of images or data generated by the model will be mathematically close to the real target data. They don't just guess this; they prove it using a clever combination of three mathematical tools: a theorem about how well neural networks can mimic functions, a theorem about how to compare different paths a particle might take (Girsanov's theorem), and a rule about how information gets lost or preserved when you look at a system from different angles.

The paper establishes a clear, quantitative rule: the error in the final generated picture is directly tied to the error in the compass, plus a small, unavoidable "mismatch" penalty. Imagine the diffusion process as a journey. The robot starts at the end of the journey (a pile of random noise) and walks backward to the beginning (the clear image). The paper proves that if the robot's compass is slightly off during the walk, the final destination will be slightly off, but the distance it gets lost is strictly limited by how bad the compass was. Crucially, the authors show that the only other thing that can make the final picture imperfect is if the "starting point" of the robot's journey (the random noise distribution) doesn't perfectly match the "ending point" of the forward process. If that mismatch is small, and the compass is sharp, the generated data will be incredibly close to the real thing.

This work is a foundational piece of the puzzle. It doesn't claim to invent a new way to generate images or claim that all current models are perfect. Instead, it provides the theoretical safety net that explains why these models work so well. It confirms that the expressive power of neural networks—our ability to teach them to be perfect compasses—directly translates into the expressive power of the diffusion models themselves. By bridging the gap between "learning a function" and "generating a distribution," the paper gives researchers a solid mathematical guarantee: if you can train your neural network to find the right direction, you are mathematically guaranteed to get a good result, provided you manage the initial noise conditions correctly.

Technical Summary: From Score Approximation to Distribution Approximation in Score-Based Diffusion Models

Problem Statement
Score-based diffusion models have achieved empirical success in generative modeling, yet their approximation-theoretic foundations remain incomplete. While classical universal approximation theorems guarantee that neural networks can approximate the score function ( $\nabla_x \log p_t(x)$ ) arbitrarily well, it has remained unclear whether such function-level guarantees translate into guarantees for the approximation of the probability distributions generated by the reverse diffusion process. Specifically, there is a lack of rigorous quantitative connection between the $L^2$ error of the learned score function and the Kullback–Leibler (KL) divergence between the target data distribution and the distribution generated by the reverse process.

Methodology
The paper establishes a rigorous quantitative link between score approximation and distribution approximation through a three-step analytical framework combining approximation theory, stochastic analysis, and information theory:

Path Measure Analysis via Girsanov's Theorem: The authors analyze the relationship between the true reverse-time stochastic differential equation (SDE) and the learned reverse-time SDE (where the true score is replaced by a neural network approximation $s_\theta$ ). By applying Girsanov's theorem on path space, they derive an explicit identity for the KL divergence between the path measures $P$ (true) and $Q_\theta$ (learned). This identity decomposes the path-space divergence into two components: the KL divergence between the terminal distribution of the forward process ( $p_T$ ) and the prior used to initialize the reverse process ( $p_{prior}$ ), and an integral term representing the time-integrated $L^2$ error of the score approximation weighted by the diffusion coefficient $g(t)$ .
Projection to Data Distribution via Data Processing Inequality: To connect path-space divergence to the final generated data distribution, the authors apply the data processing inequality for relative entropy. They prove that the KL divergence between the marginal distributions at time $t=0$ (the target $\mu$ and the generated $\mu_\theta$ ) is bounded above by the KL divergence between the corresponding path measures. This step utilizes the Radon–Nikodym derivative and Jensen's inequality to show that projecting the path measures onto the initial state cannot increase the divergence.
Integration with Universal Approximation: The authors combine the derived bounds with Hornik's universal approximation theorem. They demonstrate that for any $\epsilon > 0$ , there exists a neural network $s_\theta$ (satisfying specific regularity conditions like global Lipschitz continuity) that approximates the true score function within an $L^2(\mu)$ error of $\epsilon$ .

Key Results
The central result is Theorem 1, which provides an explicit upper bound on the distribution approximation error. The theorem states that if a neural network $s_\theta$ approximates the true score function $s$ such that $\|s_\theta - s\|_{L^2(\mu)}^2 < \epsilon$ , then the KL divergence between the target data distribution $\mu$ and the generated distribution $\mu_\theta$ satisfies:

$KL(\mu \| \mu_\theta) < \delta + \frac{1}{2}g_{max}^2 \epsilon$

Where:

$\delta = KL(p_T \| p_{prior})$ quantifies the irreducible mismatch between the forward process's terminal distribution and the reverse process's prior.
$g_{max} = \max_{t \in [0,T]} |g(t)|$ is the maximum diffusion noise schedule coefficient.
$\epsilon$ is the squared $L^2$ approximation error of the score function.

This result holds under assumptions of global Lipschitz continuity for the drift and score functions (satisfied by common neural network architectures with specific activations) and the Novikov condition for the exponential martingale.

Significance and Claims
The paper claims to provide a rigorous theoretical foundation linking the expressive power of neural networks to the generative capability of score-based diffusion models. Specifically:

It establishes that arbitrarily accurate approximation of the score function in the $L^2$ sense implies arbitrarily small KL divergence in the generated distribution, up to the irreducible prior mismatch.
The work complements recent statistical learning analyses (e.g., Chen et al.) that focus on finite-sample settings and structural data assumptions. Instead, this paper offers a general approximation-theoretic perspective in Euclidean space, independent of specific data distribution structures.
The derived bound offers a simple, explicit justification for score-based learning: improving the accuracy of the score network directly improves the quality of the generated distribution in the KL sense.

The authors note that while the analysis focuses on continuous-time SDEs, the framework suggests potential extensions to discrete-time models (DDPMs), probability flow ODEs, and non-Euclidean data spaces, though these are left for future work.

Technical Summary: From Score Approximation to Distribution Approximation in Score-Based Diffusion Models

More like this