📊 statistics

One Inverse Step is a Convex Program: Bayes-Limit Calibration of Diffusion Inversion

This paper establishes that a single implicit DDIM inversion step constitutes a strongly convex program at the Bayes limit, providing a rigorous, hypothesis-free framework to certify model errors and geometric limitations in pretrained diffusion models through the analysis of stationarity conditions and curvature bounds.

Original authors: Gordei Verbii

Published 2026-08-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Gordei Verbii

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, a powerful class of tools known as diffusion models has learned to generate stunningly realistic images, from faces to landscapes, by slowly reversing a process of adding noise. These models work by learning a map that guides a chaotic cloud of static back into a clear picture. Scientists have long suspected that these models are not just memorizing pixels but are actually learning the hidden geometric shape of the data they were trained on, a concept often called a manifold. Think of this shape as a thin, crumpled sheet of paper floating inside a vast, high-dimensional room; the data points are scattered along this sheet, and the model is supposed to understand its folds and curves. Researchers have tried to peek at this hidden geometry by running the model backward for a single step, hoping to measure the curvature of the sheet. However, until now, it was unclear whether these measurements were actually revealing the shape of the data or simply reflecting the quirks of the model's own training.

A new study by independent researcher Gordei Verbii takes a fresh look at this single-step reversal, treating it not as a vague probe but as a precise mathematical instrument with a known calibration. The researcher discovered that this single step is actually a problem of finding the lowest point in a smooth, bowl-shaped landscape. Because this landscape is perfectly shaped, the solution is guaranteed to be unique and stable, provided the model is perfect. This finding allows the researcher to separate the behavior of the model from the behavior of the data. By calculating exactly what a perfect model should do, the study establishes a strict set of rules that any working model must follow. When the researcher tested these rules against real, trained models used for images like those in the CIFAR-10 and CelebA-HQ datasets, the models failed the test. They did not just make small errors; they violated fundamental geometric laws that a perfect model could never break. Specifically, while the mathematically exact score never leaves the allowed bound, every trained score probed in the study crossed it, with violations ranging from 1.26 to 4.66 times the allowed value.

The core of the discovery lies in the realization that the model's attempt to reverse the noise is governed by a fixed schedule, a pre-determined path of how noise is removed. The study proves that for a perfect model, this path creates a landscape so well-behaved that the solution to the reversal problem is always a single, unique point. If a trained model produces more than one solution, it is a definitive sign that the model has made an error in its learning, regardless of what the data looks like. Furthermore, the study identifies a specific limit on how much the model can bend or twist during this process. A perfect model must stay within a certain boundary, but the trained models tested in the study consistently pushed past this limit. This violation is not a matter of the data being too complex or the noise being too high; it is a direct certificate that the model's internal math is broken.

The researchers also investigated why these models fail to reveal the hidden geometry they are supposed to encode. They found that the models are not failing simply because they are too weak, but because the probe itself is structurally unmeasurable on trained models. To see the true shape of the data, the model needs to look deep enough into the noise to feel the curvature of the underlying sheet. However, the study shows that the depth required to see this curvature (the "convergence shell") is several times deeper than the depth where the model has any training data. This creates a conflict: the "Fermi window" where the geometric law is valid and the "training support" where the model has learned to operate do not overlap. It is like trying to measure the curve of a mountain by standing only on the very tip of its peak; no matter how hard you look, you cannot see the slope because you are not standing far enough down the side, and the model's training simply does not extend to the necessary depth.

This limitation explains why previous attempts to measure the intrinsic dimension of data using these models have produced confusing or contradictory results. The study shows that the tools used to measure this dimension are actually measuring the model's own training limitations rather than the data itself. When the researchers tested the models on synthetic data where the true shape was known, the trained models consistently reported the wrong dimension, often guessing the size of the entire room instead of the size of the sheet. In contrast, when the researchers used a mathematically perfect, theoretical model that knew the exact shape, the measurements were accurate to within a fraction of a percent, recovering the true codimension on every closed class. This stark difference proves that the failure lies with the trained neural networks, not with the method of measurement.

The study also clarifies a specific behavior often seen in these models: oscillation. When a model tries to reverse the noise, it sometimes overshoots and bounces back and forth instead of settling down. The research shows that this bouncing is not a sign of complex data structure but a simple mechanical failure, similar to a car with brakes that are too weak to stop it from rolling back and forth. The study provides a precise formula for when this bouncing should happen and confirms that it happens exactly where the model's internal math becomes unstable. By understanding this, the researchers can distinguish between a model that is struggling with the data and a model that is simply broken.

Ultimately, this work reframes how we should view these powerful AI tools. Instead of assuming they have secretly learned the deep geometry of the world, the study suggests we should treat them as instruments that need careful calibration. The paper provides a checklist of what a perfect model should look like, and it shows that current models fall short of this ideal. The findings do not mean that diffusion models are useless; they are still incredibly effective at generating images. However, it does mean that we cannot trust them to tell us the true geometric shape of the data they were trained on. The hidden geometry is there, but the models are currently too shallow to see it, and their attempts to measure it are often just reflections of their own training limitations. This insight offers a clearer path forward: to truly understand the geometry of data, we must first build models that are mathematically robust enough to reach the necessary depth, or we must accept that our current tools are only seeing the surface.

Technical Summary: Diffusion Inverse Step as a Strongly Convex Program

Problem Statement
Diffusion models are widely used to estimate the score function (the gradient of the log-density) of noise-convolved data, theoretically encoding local manifold geometry. A common experimental probe involves performing a single deterministic DDIM inversion step to solve for the pre-image of a noisy sample. While literature often treats the resulting Jacobian or convergence behavior as an estimator of intrinsic dimension or manifold properties, this paper argues that the probe is frequently misinterpreted. The central question is: What exactly does a single inverse step measure, and under what conditions are its readouts valid? The paper posits that without a closed-form calibration against the Bayes-optimal denoiser, deviations in trained models cannot be attributed to geometric properties of the data manifold versus errors in the model itself.

Methodology
The author reframes the single-step DDIM inversion not merely as a fixed-point iteration but as the stationarity condition of an explicit potential function. By assuming the network is the Bayes-optimal denoiser (the "oracle"), they derive exact closed-form expressions for the potential, its Hessian, and the resulting contraction properties.

Potential Formulation: The residual of the inversion map $F(x) = x - G(x)$ is shown to be the gradient of a potential $\Psi_t(x) = \frac{1}{2}\|x - A_t y\|^2 + B_t \sigma_{t+1} \log p_{t+1}(x)$ .
Convexity Analysis: The Hessian of this potential, $\nabla^2 \Psi_t$ , is proven to be strongly convex at the Bayes limit with a modulus $\mu = e^{-h_t}$ , where $h_t$ is the log-SNR gap of the step. This modulus depends only on the noise schedule, not on the data distribution, manifold dimension, or curvature.
Convergence Domain: The paper analyzes the Picard iteration (standard DDIM inversion) as unit-step gradient descent on this potential. It derives exact boundaries for the convergence domain in terms of the manifold's focal radius (inverse of maximum curvature), separating the "oscillation shell" (where the iteration oscillates) from the "divergence shell" (where it diverges).
Calibration and Falsification: The author constructs a "calibration suite" using analytically integrable manifolds (circles, spheres) where the Bayes-optimal score is known. They compare exact scores against trained DDPM models on CIFAR-10, CelebA-HQ-256, and synthetic point clouds. Crucially, they introduce an "oracle control" where the exact Bayes denoiser is run through the same experimental harness as the trained models to distinguish between a broken estimator (the readout method) and a broken model (the trained score).

Key Contributions

Inverse Step as a Convex Program: The paper proves that one implicit inversion step is the gradient of a strongly convex potential with a modulus fixed solely by the schedule ( $e^{-h_t}$ ). Consequently, the solution is unique at the Bayes limit. A second solution exists only if the trained score violates a specific posterior-covariance bound, serving as a hypothesis-free certificate of model error.
Schedule-Dependent Contraction: The contraction constant for the inversion map is derived as $\rho_g \leq \rho^*_g = 1 - e^{-h_t}$ . For standard DDPM schedules, this constant is strictly less than 0.326, implying unconditional contraction at the Bayes limit. The measured contraction rate in trained models is shown to be $\rho_g = \rho^*_g S_t$ , where $S_t$ is a stiffness ratio reflecting model error.
Convergence Domain Geometry: The boundaries of the convergence domain (oscillation and divergence) are proven to be fixed fractions of the manifold's focal radius ( $w_{osc} = 1/2$ , $w_{div} = 1/(1+\rho^*_g)$ ). These boundaries are independent of the data law and dimension.
Hypothesis-Free Falsifications: Several readouts are identified that require no manifold assumptions (e.g., reach, unimodality). Specifically, the paper establishes an unconditional ceiling $\sigma_t \lambda_{\max}(\text{sym } J) \leq 1$ derived solely from the positive semi-definiteness of the posterior covariance. Violations of this ceiling in trained models certify model error directly.
Calibration of Readouts: The paper provides a calibration table for exact scores on integrable manifolds, showing that readouts (dimension, scaling exponents, stiffness) match closed-form predictions to machine precision. It corrects previous literature regarding the Bayes residual at the noiseless point and tangential scaling exponents.

Results

Exact Scores: On analytically integrable manifolds, exact scores reproduce all theoretical predictions (stiffness, contraction, convergence boundaries) to within machine precision. The convergence shells are observed exactly where the theory predicts them.
Trained Models (Images):
- Violations of Bounds: All trained models on CIFAR-10 and CelebA-HQ-256 violate the unconditional ceiling $\sigma_t \lambda_{\max}(\text{sym } J) \leq 1$ by factors of 1.26 to 4.66.
- Missing Geometry: Trained scores fail to exhibit the predicted convergence shells (oscillation/divergence boundaries). The paper attributes this not to the absence of geometry, but to a structural limitation: the "Fermi window" (the region where the Taylor expansion of the manifold geometry is valid) and the model's training support do not overlap sufficiently to reach the predicted shell radii. The trained Hessian-Lipschitz constant is found to be only 2–12% of the curvature required to bend the multiplier field, rendering the shells unmeasurable.
- Dimension Estimators: Standard dimension estimators (e.g., Fokker–Planck readouts) on sampled points are shown to measure the model's optimality gap rather than the data manifold's intrinsic dimension. At the noiseless point, they recover the intrinsic dimension but with a curvature-dependent bias.
Trained Models (Point Clouds): Trained models on synthetic point clouds fail to distinguish between manifolds of different intrinsic dimensions (e.g., $d=1$ vs. $d=2$ ) in flattened representations, whereas the exact oracle correctly recovers the codimension. This failure is attributed to the model, not the readout protocol.
Descent Estimators: The paper demonstrates that descent-based estimators of the Łojasiewicz exponent return $1/2$ tautologically under divergent steps due to the quadratic nature of the energy landscape in the transverse direction, regardless of the true exponent.

Significance and Claims
The paper claims to shift the paradigm of using diffusion inversion from an "estimator" of unknown geometric properties to an "instrument" with a known calibration curve.

Attribution of Error: By establishing exact Bayes values, the paper provides a method to attribute deviations in trained models to specific structural failures (e.g., violation of the posterior covariance bound, insufficient Hessian-Lipschitz constants) rather than ambiguous geometric hypotheses.
Limitations of Current Probes: The author concludes that the geometry the probe was designed to find (the convergence domain boundaries) is structurally unmeasurable by current trained scores on natural images because the models' second derivatives are too weak to bend the iteration trajectory into the predicted shells within the valid training support.
Modest Scope: The paper explicitly states that its focal-radius law and convergence domain results hold for the Bayes limit and the oracle. It does not claim that trained models possess these geometric structures, but rather that the instrument cannot resolve them due to the specific mismatch between the model's training support and the required curvature scale. The paper refrains from proposing new applications, focusing instead on the rigorous calibration and falsification of existing inversion-based probes.

More like this