🤖 machine learning

Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles

This paper introduces a self-supervised framework utilizing the Invariant Contamination Ratio (ICR) to jointly evaluate diffusion models, revealing that optimal representation invariance occurs at intermediate noise levels and that rising residual energy serves as an early indicator of memorization during training.

Original authors: Xiao Li, Yixuan Jia, Zekai Zhang, Xiang Li, Lianghe Shi, Jinxin Zhou, Zhihui Zhu, Liyue Shen, Qing Qu

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiao Li, Yixuan Jia, Zekai Zhang, Xiang Li, Lianghe Shi, Jinxin Zhou, Zhihui Zhu, Liyue Shen, Qing Qu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Diffusion Model as a master chef learning to cook a specific type of dish (like a perfect steak) by starting with a bowl of random noise and slowly removing the "noise" until the steak appears. Usually, we judge this chef by looking at the final steak: does it look tasty? (This is called generative quality).

However, this paper asks a different question: Is the chef actually learning the concept of a steak, or are they just memorizing the exact pictures of the steaks they were shown during training?

The authors introduce a new way to peek inside the chef's brain (the model's internal "representation space") without needing to taste the final dish. They call this new tool ICR (Invariant Contamination Ratio).

Here is the breakdown of their discovery using simple analogies:

1. The Two Ingredients: The "Core" and the "Static"

When the model looks at an image, it breaks it down into two parts:

The Invariant Core (The "Essence"): This is the stable part of the image that stays the same even if you rotate it, crop it, or add a little static noise. It's the "steak-ness" of the steak.
The Residual (The "Static"): This is the messy part that changes when you tweak the image. It includes the specific lighting, the exact pixel noise, or the unique quirks of a single photo.

The Problem: A good model should have a strong "Essence" and very little "Static." If the "Static" gets too loud, it drowns out the "Essence."

2. The New Ruler: ICR (Invariant Contamination Ratio)

The authors created a score called ICR to measure how much "Static" is contaminating the "Essence."

Low ICR: The "Essence" is loud and clear; the "Static" is quiet. (Good!)
High ICR: The "Static" is drowning out the "Essence." (Bad!)

Think of it like listening to a song on the radio. If the music is clear and the static hiss is low, the signal is good. If the hiss is loud, you can't hear the song. ICR measures that hiss.

3. Discovery #1: The "Sweet Spot" in the Noise

Diffusion models work by adding different levels of noise to images. The authors found that the model doesn't learn the "Essence" equally well at all noise levels.

Too little noise: The model is too focused on tiny, specific details (like a specific scratch on a steak).
Too much noise: The image is so blurry the model can't see anything.
The Sweet Spot: There is a "Goldilocks" zone in the middle where the noise level is just right. Here, the ICR is at its lowest, meaning the model has the clearest view of the "Essence." Interestingly, this is also the exact spot where the model performs best on other tasks (like identifying what the image is).

4. Discovery #2: Catching the Chef "Memorizing"

This is the most exciting part. Usually, to know if a model is "memorizing" (cheating by remembering the training data instead of learning the rules), you have to wait until the end and check if it generates exact copies of the training images. This is slow and expensive.

The authors found that ICR acts as an early warning system:

In a data-rich kitchen (lots of training images): As the chef gets better, the ICR score steadily goes down. The "Essence" gets clearer, and the "Static" gets quieter.
In a data-poor kitchen (few training images): The ICR score goes down at first (the chef is learning), but then it starts to go back up.
- The "U-Shape": The score drops, hits a minimum, and then rises.
- The Warning: When the score starts rising again, it means the chef has stopped learning the general rules and has started memorizing the specific details of the few images they were shown.

Why this matters: You can see this "U-shape" happen during training, before the model even starts generating bad copies. It tells you exactly when to stop training to avoid the model "overfitting" (memorizing).

Summary

The paper argues that we don't need to wait for the final "steak" to know if the chef is doing a good job. By listening to the internal "static" (using ICR), we can:

Find the perfect noise level to extract the best features.
Detect exactly when the model stops learning and starts memorizing, allowing us to stop training at the perfect moment.

It's like having a microphone inside the chef's head that tells you, "Hey, you're starting to just repeat the recipe instead of cooking!" before the dish even hits the plate.

Technical Summary: Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles

1. Problem Statement

Diffusion models have established themselves as state-of-the-art generative systems and, more recently, as powerful self-supervised representation learners. However, the theoretical and practical connection between their generative capabilities and their representation learning properties remains under-explored. Unlike traditional self-supervised learning (SSL) methods that explicitly optimize for invariance to data augmentations, diffusion models are trained via a denoising objective to recover clean signals from Gaussian-corrupted inputs.

This distinction raises fundamental questions: To what extent do diffusion representations implicitly capture the geometric properties of "good" representations (such as invariance and expansion) sought in SSL? Furthermore, how do these properties evolve across different noise levels and throughout the training dynamics, particularly in data-limited regimes where models transition from generalization to memorization? Current evaluation metrics, such as the Fréchet Inception Distance (FID), are often insufficient for detecting early memorization or for providing intrinsic, label-free signals during training.

2. Methodology

2.1 Invariance-Residual Decomposition

The authors propose a framework rooted in SSL principles to analyze diffusion features. For a given training image $\mathbf{x}_0$ , they consider a random perturbation $a$ encompassing both standard semantic-preserving augmentations and the additive Gaussian noise inherent to the diffusion process. Let $\mathbf{h}(a(\mathbf{x}_0))$ be the representation extracted from a fixed layer of the diffusion model.

The authors decompose this representation into two components:

Invariant Component ( $\mathbf{s}$ ): The conditional mean of the representation across perturbations, $\mathbf{s}(\mathbf{x}_0) = \mathbb{E}_a[\mathbf{h}(a(\mathbf{x}_0)) \mid \mathbf{x}_0]$ . This captures stable, semantic attributes resilient to corruption.
Residual Component ( $\boldsymbol{\xi}$ ): The deviation from the mean, $\boldsymbol{\xi}(a, \mathbf{x}_0) = \mathbf{h}(a(\mathbf{x}_0)) - \mathbf{s}(\mathbf{x}_0)$ . This captures view-specific, augmentation-sensitive variations.

This decomposition leads to a total covariance decomposition: $\boldsymbol{\Sigma}_h = \boldsymbol{\Sigma}_s + \boldsymbol{\Sigma}_\xi$ , where $\boldsymbol{\Sigma}_s$ represents the spread of invariant identities and $\boldsymbol{\Sigma}_\xi$ represents the aggregate residual variation.

2.2 The Invariant Contamination Ratio (ICR)

To quantify the "health" of the representation space, the authors introduce the Invariant Contamination Ratio (ICR). They formulate a generalized eigenvalue problem:
$\boldsymbol{\Sigma}_s \mathbf{v}_i = \lambda_i \boldsymbol{\Sigma}_\xi \mathbf{v}_i$
where $\lambda_i$ represents the signal-to-noise ratio (SNR) along the $i$ -th Fisher direction $\mathbf{v}_i$ . A high $\lambda_i$ indicates a direction dominated by stable invariant structure, while a low $\lambda_i$ indicates dominance by residual noise.

The ICR is defined as:
$\text{ICR} = \frac{1}{1 + \frac{1}{d}\sum_{i=1}^d \lambda_i}$
where $d$ is the feature dimension.

Low ICR: Indicates that invariant signal dominates residual variation (high SNR), suggesting a robust representation.
High ICR: Indicates that residual variation (contamination) occupies a significant portion of the representation space.

Crucially, ICR is label-free and can be computed entirely from training features without external evaluators or held-out test sets.

3. Key Contributions

A Representation-Based Evaluation Metric: The paper introduces the ICR, a single scalar metric derived from an invariance-residual decomposition. It measures the extent to which augmentation and noise-sensitive variation contaminates the stable structure of the feature space.
Identification of a Semantic Window: The authors demonstrate that diffusion noise schedules admit an intermediate "semantic window" where ICR is minimized. This minimum coincides with the noise levels that yield the highest linear classification accuracy, providing a simple, SSL-based rule for selecting optimal noise scales for downstream tasks.
Tracking Generalization vs. Memorization: The paper reveals distinct learning phases in the representation space:
- Data-Rich Regime: ICR decreases monotonically as training progresses, closely tracking improvements in generative quality (FID).
- Data-Limited Regime: ICR exhibits a U-shaped trajectory. It decreases initially as the model learns to generalize, reaches a minimum, and then increases as the model begins to memorize specific training samples. This makes ICR a sensitive early-stopping signal for memorization, detectable before standard generation metrics or nearest-neighbor tests indicate overfitting.

4. Experimental Results

The framework was evaluated on CIFAR-10, CIFAR-100, and ImageNet using various diffusion architectures (EDM, SiT).

Noise Level Analysis: Across all datasets, the ICR curve is U-shaped with respect to noise levels. The minimum ICR aligns precisely with the peak in linear classification accuracy. This confirms that intermediate noise levels offer the best trade-off between capturing fine-grained details and stable semantic structures.
Data-Rich Training: In experiments with full training sets (e.g., 50k CIFAR-10 images, 1.28M ImageNet images), ICR and FID both decreased monotonically over training iterations. This suggests that as the model learns the data manifold, the internal representation becomes increasingly invariant and "pure."
Data-Limited Training: In experiments with limited data (e.g., 4k CIFAR-10 images, 10k ImageNet images), ICR followed a U-shaped curve. The minimum of the ICR curve occurred just before the memorization ratio began to rise. Visualizations of nearest neighbors confirmed that during the ICR minimum, neighbors were semantically meaningful, whereas in the overfitting phase (rising ICR), neighbors became visually unrelated.
Feature Expansion Dynamics: Analysis of the traces of $\boldsymbol{\Sigma}_s$ and $\boldsymbol{\Sigma}_\xi$ showed that in data-rich regimes, feature expansion is driven by the growth of invariant structure. In data-limited regimes, once invariant structure saturates, further expansion is driven by residual variation, leading to the rise in ICR.

5. Significance and Claims

The paper claims that diffusion models can be effectively monitored and evaluated through the geometry of their learned representations using self-supervised principles. The primary significance lies in providing intrinsic, training-time signals that do not rely on:

External pre-trained networks (like DINOv2 or Inception).
Expensive sampling and generation of test images.
Ground-truth labels.

The authors argue that ICR serves as a unified diagnostic tool that bridges the gap between representation learning and generative modeling. It offers a practical method to identify optimal noise scales for feature extraction and to detect the onset of memorization in data-scarce scenarios, addressing a critical gap in current evaluation methodologies where standard metrics like FID may fail to detect early overfitting. The work suggests that the transition from generalization to memorization is fundamentally a shift in the representation space from invariant structure to residual variation.