📊 statistics

MAD: Manifold Attracted Diffusion

This paper introduces Manifold Attracted Diffusion (MAD), a method that leverages the manifold hypothesis to modify the inference procedure of score-based diffusion models, enabling the efficient generation of noiseless samples from noisy training data by suppressing small off-manifold variations while preserving significant on-manifold structures.

Original authors: Dennis Elbrächter, Giovanni S. Alberti, Matteo Santacesaria

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dennis Elbrächter, Giovanni S. Alberti, Matteo Santacesaria

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to draw perfect pictures of cats. You give the robot a huge stack of reference photos, but there's a catch: every single photo in the stack is covered in a thick layer of static, scratches, and random speckles of dust.

If you ask a standard AI to learn from these dirty photos, it will learn to draw cats with the static and scratches. It thinks the dust is part of the cat.

The paper you shared introduces a new method called MAD (Manifold Attracted Diffusion). Think of MAD not as a new teacher, but as a special filter the robot uses after it has finished learning, right before it draws the final picture.

Here is how it works, using simple analogies:

1. The "Manifold" (The Invisible Track)

The paper relies on an idea called the Manifold Hypothesis. Imagine that all "real" cat pictures exist on a very specific, invisible train track floating in a massive, 3D space.

On the track: These are the meaningful parts of the image (the shape of the ear, the curve of the tail, the color of the fur). Moving along the track changes the cat from a kitten to an adult, or a Siamese to a Persian.
Off the track: This is the empty space around the track. If you move in these directions, you aren't changing the cat; you are just adding random noise, dust, or static.

The problem is that the robot's training data is so noisy that the "track" is hard to see. The robot gets confused and thinks the dust is part of the track.

2. The "Extended Score" (The Magnetic Compass)

Standard AI models use something called a "score" to figure out which way to move to make the picture clearer. It's like a compass that points toward the "most likely" next step. But if the data is noisy, this compass gets jittery and points in the wrong direction (toward the dust).

The authors invented a new tool called the Extended Score.

The Analogy: Imagine the standard compass is a sensitive needle that gets blown around by a gentle breeze (noise). The Extended Score is like a heavy, magnetized compass.
How it works: It has a special property: if the wind (noise) is very weak, the magnet pulls the needle straight to the center of the track and ignores the breeze. But if the wind is actually a strong, meaningful change (like turning a cat's head), the magnet lets the needle move with it.

In technical terms, the paper says this tool treats tiny variations (noise) as if they don't exist, effectively shrinking them to zero. But it leaves the big, important variations (the actual image features) alone.

3. The Process: "Attracting" the Image

When the robot generates a picture, it starts with a cloud of random static.

Standard Method: The robot slowly cleans the static, but because it learned from dirty photos, it leaves the "dust" patterns in the final image.
MAD Method: The robot uses the Extended Score compass. As it cleans the image, this compass acts like a powerful magnet pulling the image pixels toward the "invisible track" of real data.
- Any pixel that is just random noise gets pulled hard back to the track (removed).
- Any pixel that is part of the real cat shape is allowed to stay where it belongs.

The result is a "soft thresholding" effect. It's like a sieve that lets the big, heavy rocks (real features) fall through but shakes out all the tiny, light dust (noise).

What Did They Prove?

The authors tested this idea in three ways:

Toy Problems: They drew simple shapes on a computer and showed that the method could pull the shapes out of a mess of noise, while standard methods just reproduced the mess.
Real Photos: They took models trained on noisy versions of famous datasets (like faces from FFHQ or animals from ImageNet). When they used the standard method, the faces looked grainy. When they used MAD, the faces looked clean, and the backgrounds became solid colors (because the random background noise was "attracted" away).
Real Science Data: They tested it on Cryo-Electron Microscopy images (pictures of tiny biological particles). These images are incredibly noisy. The standard method produced blurry, noisy blobs. MAD managed to pull out clear shapes that matched what scientists know the particles look like, even though the AI had never seen a clean version of these particles before.

The Bottom Line

The paper claims that MAD allows you to take an AI model trained on dirty, noisy data and use a simple mathematical tweak during the drawing process to generate clean images.

You don't need to retrain the AI from scratch. You just change the "compass" it uses at the very end. It's a way of telling the AI: "Ignore the tiny wiggles; they are just noise. Only listen to the big movements."

Important Note: The paper explicitly states this is for generating new, clean images from a noisy dataset. It is not a tool for fixing a single, specific photo you already have (like cleaning up one old family portrait); it is for creating new examples of what that object should look like without the noise.

Technical Summary: Manifold Attracted Diffusion (MAD)

Problem Statement
Score-based diffusion models have established themselves as state-of-the-art methods for image synthesis, often outperforming GANs and VAEs in sample quality and diversity. However, a significant limitation arises when training data is corrupted by noise from measurement errors, compression artifacts, or data collection processes. Standard diffusion models trained directly on such noisy datasets learn to reproduce the corruption, generating samples that retain the noise characteristics of the training data. While existing approaches to noisy training often require modifying the training process itself (e.g., adapting loss functions or requiring a mix of clean and noisy data), there is a lack of efficient methods that can generate clean samples from a noisy distribution using established training frameworks and pre-trained networks without retraining.

Methodology
The proposed method, Manifold Attracted Diffusion (MAD), addresses this by modifying the inference procedure of score-based models rather than the training process. The approach is grounded in the manifold hypothesis, which posits that meaningful data resides on a low-dimensional manifold within a high-dimensional ambient space. The authors interpret noise as low-magnitude variations in directions orthogonal (off-manifold) to this structure, whereas meaningful data variations occur along the manifold (on-manifold).

The core technical innovation is the introduction of the extended score.

Definition: For a probability measure $p$ and a parameter $\gamma > 0$ , the extended score $H_\gamma p$ is defined as $(1 + \gamma)S(p * g_\gamma) + \gamma \frac{d}{d\gamma}S(p * g_\gamma)$ , where $S$ is the standard score operator ( $\nabla \log p$ ) and $g_\gamma$ is a Gaussian kernel.
Properties: Unlike the standard score, which is undefined for Dirac delta distributions (representing data on a lower-dimensional manifold with zero variance in certain directions), the extended score is well-defined. Crucially, for a Dirac delta, the extended score behaves similarly to the score of a Gaussian with unit variance. This allows the inference process to treat low-variance (off-manifold) directions as if they have a non-zero variance, effectively applying a "soft thresholding" effect.
Inference Algorithm: The authors propose an iterative inference algorithm that utilizes an approximation of the extended score. Given a pre-trained denoiser network $D_\theta$ (which approximates the standard score), the extended score is approximated by evaluating the network at a slightly perturbed noise level and computing a finite difference to approximate the derivative with respect to $\gamma$ .
Implementation: The algorithm introduces hyperparameters ( $a, b, p$ ) that control the strength of the "manifold attraction." By adjusting these, the inference procedure suppresses small variations in off-manifold directions (noise) while preserving larger variations in on-manifold directions (signal). The method is compatible with established frameworks, such as the probability flow ODE formulation by Karras et al. (2022), and can utilize pre-trained networks.

Key Contributions

Formal Definition and Analysis: The paper provides a rigorous mathematical definition of the extended score, analyzing its behavior on both standard distributions and Dirac delta measures, demonstrating its ability to generalize the score operator to degenerate distributions.
Efficient Inference Algorithm: The authors present an inference algorithm that reduces noise in generated samples without requiring a special training procedure. This makes the method compatible with existing training pipelines and pre-trained models.
Empirical Validation: The efficacy of the approach is demonstrated across toy problems, synthetic data, and real-world datasets, including FFHQ, AFHQ, ImageNet, and cryo-electron microscopy (Cryo-EM) data.

Results

Toy and Synthetic Data: In 2D illustrative examples, the extended score successfully collapses trajectories of low-variance Gaussians to their means (mimicking Dirac deltas) while preserving the spread of high-variance components. On synthetic datasets with added Gaussian noise and pixel removal, MAD significantly outperforms standard score inference, generating samples that recover the original clean shapes.
Real-World Image Data: Experiments on FFHQ, AFHQv2, and ImageNet show that increasing the manifold attraction parameter ( $b$ ) results in cleaner backgrounds and sharper primary features, effectively thresholding out high-frequency noise and secondary variations.
Quantitative Metrics: On CIFAR-10 and FFHQ datasets corrupted with Gaussian, shot, and pixel-removal noise, MAD achieves significantly lower Fréchet Inception Distance (FID) scores compared to standard inference. Notably, standard inference FIDs closely match the noisy training data FIDs, confirming that standard models reproduce corruption, whereas MAD successfully denoises.
Cryo-EM Application: Applied to the EMPIAR-11618 dataset (noisy single-particle Cryo-EM images), MAD generated samples that strongly resembled the refined shapes obtained by traditional methods, despite the network being trained exclusively on noisy data.

Significance and Claims
The paper claims that MAD offers a practical solution for "blind generative denoising," where the specific degradation model is unknown and only noisy data is available. The method's significance lies in its ability to leverage the geometric structure of data manifolds to filter noise during the generation phase, rather than relying on complex training-time adaptations.

The authors are modest regarding the scope of their claims:

They do not claim to solve inverse problems (like denoising a single given image) but rather to learn from a noisy dataset to produce new, clean samples from the underlying distribution.
They acknowledge that reaching state-of-the-art performance in specific domains (like Cryo-EM) would require incorporating more domain-specific prior knowledge, which is outside the scope of this work.
They note that while the method is effective, the computational cost is at most double that of standard inference (due to the extra network evaluation), though empirical observations suggest using the extended score only in the latter half of inference steps may suffice.

The work positions the extended score as a geometrically grounded mechanism to improve generation quality, potentially applicable even to models trained on clean data, and opens avenues for future research in adaptive parameter selection and integration with conditional diffusion models.