📊 statistics

On quantitative Laplace-type convergence results for some exponential probability measures, with two applications

This paper establishes quantitative Laplace-type convergence bounds for exponential probability measures with norm-like potentials under a generalized Jacobian condition using geometric measure theory tools, and applies these results to maximum entropy models and the low-temperature convergence of Stochastic Gradient Langevin Dynamics for non-convex minimization.

Original authors: Valentin De Bortoli, Agnès Desolneux

Published 2026-04-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Valentin De Bortoli, Agnès Desolneux

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find the absolute lowest point in a vast, foggy landscape. This landscape represents a complex problem, like training a neural network or understanding the structure of an image. The "height" of the land at any point is determined by a function called a potential (let's call it $U$ ). Your goal is to find the "valleys" where this height is zero.

In the world of mathematics and machine learning, there is a common tool called Laplace's method. Think of this as a "temperature control" for your search.

High Temperature ( $\epsilon$ is large): The fog is thick. You can wander anywhere, and the probability of being in any spot is spread out. You aren't focused on the lowest point yet.
Low Temperature ( $\epsilon$ approaches 0): The fog clears. The "heat" dies down, and the probability mass (the chance of finding yourself somewhere) collapses entirely onto the very bottom of the valleys.

The Problem: The "Flat" Valleys

Traditionally, mathematicians have a rule for how fast this collapse happens. They say: "If the bottom of the valley is a sharp, smooth bowl (like a perfect parabola), we can calculate exactly how the probability concentrates." This requires the "Hessian" (a measure of the bowl's curvature) to be invertible—basically, the bowl must have a distinct, non-flat bottom.

But here is the catch: In many modern applications (like deep learning or image processing), the valleys aren't always sharp bowls. Sometimes, the bottom of the valley is a flat plateau or a curved ridge. Imagine a valley that looks like a long, flat riverbed rather than a single point. In these cases, the old rules break down because the "curvature" is zero or undefined. The standard math tools get stuck.

The Solution: A New Map and a New Ruler

The authors of this paper, Valentin De Bortoli and Agnès Desolneux, propose a new way to handle these "flat" or "ridge-like" valleys.

The Shape of the Valley: They focus on a specific type of landscape where the height is determined by the "length" of a vector (like a norm). Imagine the landscape is shaped by how far you are from a target line or surface.
The New Tool (Geometric Measure Theory): Instead of looking at the curvature of the bowl, they use a tool called the Coarea Formula.
- Analogy: Imagine you want to measure the volume of a loaf of bread. The old way was to slice it into thin, flat layers (curvature). The new way is to slice it along the grain of the bread (the level sets). They slice the landscape into layers of equal height and measure the "surface area" of each slice.
- They use a concept called the Generalized Jacobian, which acts like a custom ruler that adjusts for the shape of the valley floor, even if it's flat or weirdly shaped.

What They Found (The "Quantitative" Results)

The paper doesn't just say "it converges." It gives a speed limit.

They proved that as the temperature ( $\epsilon$ ) drops, the probability distribution gets closer to the final "perfect" distribution (concentrated on the valley floor) at a specific rate.
They measured this distance using the Wasserstein distance.
- Analogy: Imagine you have a pile of sand (the current distribution) and you want to move it to match a target shape (the final distribution). The Wasserstein distance is the minimum amount of "work" (energy) needed to move the sand grains to their new spots.
The Result: They showed that the work needed decreases predictably as the temperature drops. Specifically, the error shrinks roughly proportional to $\epsilon^{1/k}$ (where $k$ depends on the shape of the valley).

Real-World Applications Mentioned in the Paper

The authors apply this new math to three specific scenarios:

Maximum Entropy Models (Microcanonical vs. Macrocanonical):
- The Setup: In physics and image processing, there are two ways to define a "perfect" distribution. One is strict (the "Microcanonical"): You must be exactly on the zero-error line. The other is relaxed (the "Macrocanonical"): You are allowed to be slightly off, as long as the average error is small.
- The Discovery: The authors show that if you just let the relaxed version get colder and colder, it doesn't automatically become the strict version. It becomes a "twisted" version. However, if you adjust your "ruler" (the Generalized Jacobian) correctly, you can use the relaxed version to perfectly sample the strict version.
- Experiment: They tested this on simple shapes (like finding the zeros of a polynomial or an ellipse) and showed that their method correctly identifies the uniform distribution along the curve, whereas the standard method gets the density wrong.
Variational Autoencoders (VAEs):
- The Setup: VAEs are a type of AI used to generate images. They have a "latent space" (a hidden code) that generates the image.
- The Discovery: The authors show that the "posterior" (the AI's belief about the hidden code given an image) concentrates around the correct values as the noise decreases. They provide a formula for how fast this belief sharpens, which helps in understanding how stable these AI models are.
Stochastic Gradient Langevin Dynamics (SGLD):
- The Setup: This is a popular algorithm used to train AI models on non-convex problems (landscapes with many hills and valleys). It adds random noise to help the algorithm jump out of small "local" valleys to find the "global" best one.
- The Discovery: The authors analyzed what happens when this algorithm runs at very low temperatures. They found that the algorithm's final state concentrates on the best solutions, but with a catch: it depends on a "Thermodynamic Barrier."
- The Barrier Analogy: Imagine a deep valley (the global minimum) separated from a shallow valley (a local minimum) by a hill. If the hill is too high, the algorithm might get stuck in the shallow valley even at low temperatures. The authors introduced a new way to measure this "hill height" (thermodynamic barrier) to predict if the algorithm will succeed in finding the true global minimum as the dataset gets larger.

Summary

In simple terms, this paper fixes a broken tool used to find the "best" solutions in complex, flat landscapes. By using a new geometric slicing method (Coarea formula) instead of the old curvature method, they provided a precise speed limit for how fast AI and statistical models converge to their optimal states, even when those states aren't simple, sharp points. They proved this works for specific types of "flat" valleys and demonstrated its utility in image generation and AI training.

1. Problem Statement

The paper addresses the asymptotic behavior of a sequence of Gibbs probability measures $(\pi_\varepsilon)_{\varepsilon > 0}$ as the "temperature" parameter $\varepsilon \to 0$ . These measures are defined by densities proportional to $\exp[-U(x)/\varepsilon]$ with respect to the Lebesgue measure.

The Challenge: Classical Laplace approximation results rely on the invertibility of the Hessian of the potential $U$ at its minimizers. However, in many modern machine learning applications (e.g., Variational Autoencoders, microcanonical distributions), the potential takes the form $U(x) = \|F(x)\|^k$ . In these cases, particularly when the dimension of the input $d$ is greater than or equal to the dimension of the output $p$ ( $d \ge p$ ), the Hessian of $U$ at the minimizers (where $F(x)=0$ ) is often singular (non-invertible).
The Goal: The authors aim to establish quantitative, non-asymptotic bounds on the convergence of $\pi_\varepsilon$ to a limiting measure $\pi_0$ in the Wasserstein distance of order 1 ( $W_1$ ), without requiring the Hessian of $U$ to be invertible. Instead, they require invertibility conditions on the generalized Jacobian of the underlying function $F$ .

2. Methodology

The authors employ tools from Geometric Measure Theory, specifically the Coarea Formula, to handle the degeneracy of the potential.

Norm-like Potentials: The study focuses on potentials of the form $U(x) = \|F(x)\|^k$ , where $F: \mathbb{R}^d \to \mathbb{R}^p$ is a smooth function and $k \in \mathbb{N}$ .
Generalized Jacobian: Instead of the Hessian of $U$ , the analysis relies on the generalized Jacobian $J_F(x)$ , defined as:
$J_F(x) = \begin{cases} \det(DF(x)DF(x)^\top)^{1/2} & \text{if } d \ge p \\ \det(DF(x)^\top DF(x))^{1/2} & \text{if } d \le p \end{cases}$
The key assumption is that $J_F(x) \neq 0$ for all $x$ in the zero set $F^{-1}(0)$ .
Coarea Formula Application:
- For the case $d \ge p$ , the authors use the Coarea Formula to transform the integral over $\mathbb{R}^d$ into an integral over the level sets of $F$ . This allows them to express the measure $\pi_\varepsilon$ in terms of integrals over the manifolds $F^{-1}(t)$ .
- They establish the Lipschitz regularity of the mapping $t \mapsto \int_{F^{-1}(t)} \phi(x) J_F(x)^{-1} d\mathcal{H}^{d-p}(x)$ , where $\mathcal{H}$ is the Hausdorff measure. This regularity is crucial for deriving the convergence rate.
- For the case $d \le p$ , they adapt classical Morse theory and Laplace method arguments, as the Hessian invertibility condition is naturally satisfied in this regime under their assumptions.
Thermodynamic Barrier: In the context of non-convex optimization, they introduce the concept of a thermodynamic barrier ( $c^*$ ), which quantifies the energy gap between local minima and global minima. This is distinct from the "kinetic barrier" used in simulated annealing.

3. Key Contributions

A. Quantitative Convergence Theorems

The paper establishes that under the assumption that $F$ is smooth and $J_F(x) \neq 0$ on the zero set:

Convergence Rate: The Wasserstein distance between the Gibbs measure $\pi_\varepsilon$ and the limiting measure $\pi_0$ converges at a rate of $O(\varepsilon^{1/k})$ .
$W_1(\pi_\varepsilon, \pi_0) \le C \varepsilon^{1/k}$
Limiting Measure: The limit $\pi_0$ is supported on the set of minimizers $F^{-1}(0)$ and has a density proportional to $J_F(x)^{-1}$ with respect to the Hausdorff measure on that set.
Generality: These results hold even when the Hessian of $U$ is singular, covering cases where $d \ge p$ (common in over-parameterized neural networks).

B. Application 1: Maximum Entropy Distributions (Microcanonical vs. Macrocanonical)

Context: The authors analyze the relationship between Macrocanonical distributions (which satisfy constraints in expectation, e.g., $\mathbb{E}[\|F(x)\|^2] = \varepsilon$ ) and Microcanonical distributions (which satisfy constraints almost surely, i.e., $F(x)=0$ ).
Finding: They prove that the limit of a standard sequence of macrocanonical distributions as $\varepsilon \to 0$ is not the uniform microcanonical distribution. Instead, it converges to a "twisted" distribution weighted by the reference measure and the Jacobian.
Solution: They propose a correction algorithm (using a specific weighting $\Psi(x) = J_F(x)$ ) to recover the true uniform microcanonical distribution. This is demonstrated experimentally on polynomial zeros and elliptical constraints.

C. Application 2: Stochastic Gradient Langevin Dynamics (SGLD)

Context: SGLD is used to sample from non-convex potentials in machine learning. Existing literature provides bounds on the expected risk $E[U(X_k)]$ .
Finding: The authors characterize the limiting distribution of SGLD iterates at low temperatures. They show that the distribution concentrates on the minimizers of the population loss.
Thermodynamic Barrier: They introduce the thermodynamic barrier to derive quantitative bounds. They show that if the barrier is small (local minima are close to global minima), the convergence to the global minimum set is slower. Specifically, they provide non-asymptotic bounds on the distance between SGLD iterates and the limiting measure concentrated on the global minimizers.

4. Key Results Summary

Component	Result
Convergence Rate	$W_1(\pi_\varepsilon, \pi_0) = O(\varepsilon^{1/k})$ .
Condition	Invertibility of the generalized Jacobian $J_F$ on $F^{-1}(0)$ , not the Hessian of $U$ .
Limiting Measure	Concentrated on $F^{-1}(0)$ with density $\propto J_F(x)^{-1}$ .
Macrocanonical Limit	Standard macrocanonical limits are "twisted"; a specific correction yields the uniform microcanonical distribution.
SGLD Behavior	Quantitative bounds established between SGLD iterates and the limiting measure, dependent on the thermodynamic barrier $c^*$ .

5. Significance

Bridging Theory and Practice: The results provide rigorous theoretical justification for the behavior of algorithms in over-parameterized settings (where $d \ge p$ ) where standard Hessian-based Laplace approximations fail.
New Tools for ML: By utilizing Geometric Measure Theory (Coarea formula), the paper opens a new pathway for analyzing probability measures on manifolds and singular sets, which are prevalent in deep learning (e.g., VAEs, generative models).
Algorithmic Improvement: The identification of the "twisted" nature of macrocanonical limits suggests that standard sampling methods for constrained optimization may need Jacobian corrections to achieve uniform sampling on the constraint manifold.
Non-Convex Optimization: The introduction of the thermodynamic barrier provides a refined tool for analyzing the convergence of SGLD in non-convex landscapes, distinguishing between the difficulty of escaping local minima (kinetic) and the structural properties of the energy landscape (thermodynamic).

In conclusion, this work extends the classical Laplace method to a broader class of singular potentials, providing precise quantitative convergence rates and offering new insights into the behavior of maximum entropy models and stochastic optimization algorithms in machine learning.