Identifying Memorization of Diffusion Models through $p$-Laplace Analysis: Estimators, Bounds and Applications

This paper proposes a novel method for identifying memorized training data in diffusion models by leveraging $p$-Laplace operators derived from estimated score functions, providing both theoretical error bounds and empirical validation on text-to-image models where the approach successfully detects memorization even without access to the conditioning text.

The Gist

Imagine you have a giant, magical paint machine (a Diffusion Model) that learns to create beautiful pictures by studying a huge library of existing art. Over time, it gets so good that it can invent entirely new landscapes, faces, and scenes.

However, sometimes this machine gets a little too good. Instead of inventing something new, it accidentally copies a specific painting from its library and spits it out again. This is called Memorization. It's like a student who, instead of writing an essay from their own ideas, just memorizes and recites a paragraph from a textbook. This is bad because it might leak private information or violate copyright.

The problem is: How do we catch the machine when it's cheating?

Usually, the machine's "brain" (its internal math) is a black box. We can't easily see if it's remembering a specific image or just making a new one that looks similar.

The Solution: The "Topographic Map" Analogy

The authors of this paper came up with a clever way to peek inside the machine's brain. They treat the machine's knowledge like a landscape of hills and valleys.

• Normal Data: Imagine a smooth, rolling hill. If you walk around, the ground is gentle and predictable. This represents the machine learning general concepts (like "a cat" or "a sunset").
• Memorized Data: Now, imagine someone took a tiny, sharp spike and stuck it right into the middle of that smooth hill. This spike represents a specific, memorized image. Because the machine saw this exact image so many times (or it was the only one of its kind), the "probability" of that spot is incredibly high, creating a sharp peak.

The authors wanted to find these sharp spikes in the landscape.

The Tool: The "p-Laplace" Compass

To find these spikes, they used a mathematical tool called the p-Laplace operator.

Think of the p-Laplace as a super-sensitive weather vane or a flux detector.

• If you stand on a smooth hill, the wind (the mathematical "gradient") blows gently in all directions, balancing out. The weather vane spins lazily.
• If you stand on a sharp spike (a memorized image), the wind rushes inward from all sides toward the peak with great force. The weather vane spins wildly and points straight down into the hole.

The authors realized that by measuring this "inward rush" (the flux), they could spot the memorized spikes.

The Secret Sauce: Why "p=1" is the Best

The paper tested different settings for this weather vane, labeled by a number called p.

• p=2 (The Standard): This is like a standard compass. It's good, but it gets confused by the size of the wind. If the wind is just slightly stronger or weaker, the reading changes a lot.
• p=1 (The Magic Setting): The authors discovered that setting p=1 is like a compass that only cares about the direction of the wind, not how hard it's blowing.
  • Analogy: Imagine trying to find a mountain peak in a foggy storm. If you only look at how hard the wind hits you, you might get confused by a sudden gust. But if you just look at which way the wind is pushing you, you can clearly see it's pushing you toward the peak, no matter how strong the gusts are.

Because the machine's internal math isn't perfect (it's an approximation), the "wind strength" is often noisy. But the "wind direction" is usually correct. By using p=1, the authors created a filter that ignores the noise and highlights the memorized spikes perfectly.

How They Tested It

1. The Practice Run: They started with a simple, fake world (a 2D map of Gaussian hills). They planted a "fake spike" by repeating one data point 250 times. Their p=1 weather vane immediately spotted the spike, while the other methods got confused.
2. The Real World: They then took a famous, real-world image generator (Stable Diffusion) and tested it on 500 prompts known to cause memorization.
   • The Result: Their method was incredibly accurate. Even when they didn't know the text prompt used to generate the image (the "post-generation" regime), their p=1 compass could still point out, "Hey, this image is a copy!" with 91% accuracy.

Why This Matters

This research gives us a new "lie detector" for AI.

• Privacy: It helps us see if an AI is leaking sensitive training data (like a doctor's photo or a private document).
• Copyright: It helps artists and companies see if an AI is just copying their work instead of creating something new.
• Safety: It ensures that the AI is actually learning and generalizing, rather than just acting like a broken record player.

In short, the authors built a mathematical "spike detector" that ignores the noise and finds the hidden copies in the AI's memory, ensuring these powerful tools remain creative rather than just copycats.

Technical Summary

1. Problem Statement

Generative models, particularly Diffusion Models, are prone to memorization, where the model reproduces training data samples rather than generating novel content. This poses significant risks regarding privacy (leakage of sensitive data) and copyright.

• The Core Issue: Memorized samples often manifest as "bumps" or local maxima in the learned probability density function (PDF), particularly in sparsely populated regions or due to data replication.
• The Challenge: The underlying probability distribution of natural images is unknown. While diffusion models learn the score function ($\nabla \log p(x)$), it is unclear how to leverage this learned score to detect these specific "bumps" or quantify the geometry of the probability landscape to identify memorization.
• Gap: Existing methods often require access to the conditioning prompt (text) or rely on Hessian-based approaches. There is a need for a robust, theoretically grounded method to detect memorization in a post-generation regime (even without the original prompt).

2. Methodology

The authors propose using the p-Laplace operator ($\Delta_p$) to analyze the geometry of the learned probability distribution.

A. Theoretical Foundation

The p-Laplace operator is defined as:
$$\Delta_p u = \nabla \cdot (|\nabla u|^{p-2} \nabla u)$$
where $u = \log p(x)$ is the log-probability.

• Hypothesis: Memorized samples correspond to local maxima in the log-probability. Around a local maximum, gradient vectors point inward, resulting in a negative flux. Therefore, memorized points should exhibit significantly lower (more negative) p-Laplace values compared to non-memorized points.
• Score Function Approximation: Since the true score $s(x) = \nabla \log p(x)$ is unknown, the method utilizes the score function $\hat{s}(x)$ estimated by a trained diffusion model.

B. Estimation Techniques

The paper proposes two numerical approximations for the p-Laplace operator based on the learned score $\hat{s}(x)$:

1. Volume Integral Formulation: Approximates the operator by averaging the divergence over a ball $B_R(x_0)$.
2. Boundary Integral Formulation (Divergence Theorem): Approximates the operator by integrating the flux over the boundary sphere $\partial B_R(x_0)$.
   $$\Delta_p u(x_0) \approx \frac{1}{|B_R|} \int_{\partial B_R} |\hat{s}(y)|^{p-2} \hat{s}(y) \cdot n \, ds$$

C. Error Bounds

The authors derive theoretical error bounds for these estimators. They prove that if the error between the true score $s$ and the learned score $\hat{s}$ is bounded by $\delta$, and the score norms are bounded by $m$ and $M$, the error in the p-Laplace estimation is bounded by a constant $C_p$ dependent on $p$, $m$, $M$, and the geometry of the integration domain.

D. Operational Regime

The analysis is performed in the "small-$\alpha$" post-generation regime. This involves:

• Operating in the latent space (e.g., of Stable Diffusion).
• Using the final denoising steps where noise is minimal.
• This ensures the density is smooth enough for the divergence theorem to apply while remaining geometrically faithful to local structures (attraction basins).

3. Key Contributions

1. Novel Application of p-Laplace: First work to estimate the p-Laplace operator using diffusion model score functions to characterize the learned probability distribution and identify memorization.
2. Theoretical Error Bounds: Proven mathematical bounds on the estimation error of the p-Laplace operator when using learned scores, providing reliability guarantees.
3. Optimal Parameter Identification: Through extensive experiments, the authors demonstrate that $p=1$ (1-Laplace) combined with the boundary integral formulation is the most robust approach.
   • Reasoning: The 1-Laplace relies on normalized gradients. Since diffusion models approximate the direction of the score much better than its magnitude, the 1-Laplace cancels out magnitude errors, leading to superior performance.
4. Prompt-Agnostic Detection: The method successfully identifies memorization in a post-generation regime without requiring access to the conditioning text (prompt), a significant advantage over prior methods.

4. Experimental Results

The study validates the approach across three scales:

• Synthetic GMM (Gaussian Mixture Models):

  • Used a 2D GMM with a known ground truth to validate the numerical approximations.
  • Result: The 1-Laplace boundary formulation showed the lowest error rates and highest fidelity to the true p-Laplace.
  • Memorization Detection: By artificially replicating a training point (creating a "spike"), the method successfully identified the memorized point as a distinct outlier with the lowest percentile p-Laplace value.

• Error Bound Verification:

  • Empirically validated the theoretical error bounds derived in Section 3.5. The empirical errors consistently stayed below the theoretical upper bounds.

• Large-Scale Text-to-Image (Stable Diffusion v1.4):

  • Dataset: 500 known memorized prompts and 500 non-memorized prompts (LLM-generated), generating ~3,000 images.
  • Comparison: Compared against a state-of-the-art method by Wen et al. [69] (which uses score differences).
  • Performance:
    • With Prompt Access: The proposed method achieved an AUC of 0.958, comparable to the competitor (0.957).
    • Without Prompt Access (Post-Generation): The proposed method achieved an AUC of 0.913, significantly outperforming the competitor (0.502), which failed without the prompt.
  • Qualitative: Visualizations showed clear distinction in 1-Laplace values between memorized and non-memorized images.

5. Significance

• Privacy and Safety: Provides a robust tool for auditing generative models to detect unintended data leakage, crucial for copyright compliance and privacy protection.
• Theoretical Insight: Advances the understanding of the geometry of implicitly learned probability functions in diffusion models, linking memorization to specific differential properties (p-Laplace values).
• Practical Utility: The ability to detect memorization without the original prompt makes this method applicable to real-world scenarios where generated images are analyzed after the fact (e.g., content moderation, forensic analysis).
• Methodological Guidance: Establishes that for score-based analysis of high-dimensional distributions, the 1-Laplace boundary formulation is the preferred estimator due to its invariance to magnitude estimation errors.