Cyclic Denoising Reveals Ultrastable Memories in Diffusion Models

Technical Summary: Cyclic Denoising Reveals Ultrastable Memories in Diffusion Models

Problem Statement
Large generative models, particularly diffusion models, are trained on massive web-scale datasets, creating risks where models may retain and reproduce specific training examples, including copyrighted artwork, proprietary media, and sensitive personal data. Existing extraction attacks (e.g., "generate-and-filter") rely heavily on external signals such as known training captions, access to the training set, or auxiliary classifiers to identify memorized content. These methods often require large-scale generation followed by post-hoc filtering or membership inference. There is a need for an extraction method that relies solely on the model's internal sampling dynamics, requires no prior knowledge of training data or prompts, and can operate without weight inspection or gradient access.

Methodology: Cyclic Denoising
The authors introduce cyclic denoising, a protocol inspired by the physics of driven disordered systems (specifically random organization and mechanical annealing). The method treats the diffusion model as a dynamical system rather than just a sampler.

1. Protocol: Starting from an initial state (image or latent), the model undergoes repeated cycles of:

   • Forward Noising: Adding noise to the state up to a controlled intermediate amplitude $\gamma$ (where $\gamma \in [0, 1]$ represents the fraction of the inference trajectory traversed).
   • Reverse Denoising: Fully denoising the state back to a clean image ($\gamma=0$).
   • Iteration: The output of one cycle initializes the next, creating a stroboscopic trajectory observed at the end of each cycle.

2. Control Parameter: The cycling amplitude $\gamma$ acts as a drive amplitude. Low $\gamma$ applies weak perturbations, while high $\gamma$ applies strong perturbations. The process uses standard samplers (DDPM) with fresh Gaussian noise at each step, making it a stochastic cyclic driving process.

3. Detection Mechanism: The authors monitor the cosine similarity between consecutive stroboscopic states ($z_n$ and $z_{n-1}$).

   • Absorbing Episodes: Long plateaus of near-unity similarity indicate the trajectory has locked into a stable attractor basin.
   • Basin Hopping: Sharp drops in similarity indicate transitions between basins.
   • Ultrastability: Attractors that persist for hundreds to thousands of cycles, even after severe corruption (high $\gamma$), are identified as "ultrastable."

Key Contributions

1. Dynamical Probe of Generative Landscapes: The paper establishes cyclic denoising as a physics-inspired tool to traverse the learned generative landscape. Unlike standard sampling which draws independent points, this method turns sampling into a dynamical system controlled by $\gamma$, revealing the stability structure of the model.
2. Characterization of Long-Time Dynamics: The authors identify a rich spectrum of dynamical behaviors:
   • Trivial Absorbing States: At low amplitudes, trajectories collapse into featureless fixed points or limit cycles (oscillating patterns).
   • Basin Hopping: At intermediate amplitudes, the system explores the landscape, hopping between basins.
   • Ultrastable Attractors: At higher amplitudes, the system settles into deep, long-lived attractors that resist change.
3. Yielding-Like Transition: The study demonstrates a transition analogous to yielding in disordered solids. As $\gamma$ increases, the system moves from a "stuck" phase (absorbing states) to an exploratory phase (basin hopping), eventually isolating the deepest attractors.
4. Prompt-Free Extraction Attack: The authors demonstrate that many of these deep, ultrastable attractors correspond to memorized training images (e.g., stock photos, brand watermarks, web-crawl artifacts). This constitutes an extraction attack that requires only sampler-level control, no training data access, no captions, and no gradients.
5. Stability as a Memorization Diagnostic: In prompt-conditioned settings, the authors show that removing the prompt and continuing to cycle reveals the intrinsic stability of the recovered image. Genuinely memorized images remain locked (ultrastable) even without conditioning, whereas concept-based basins decorrelate rapidly.

Results

• Stable Diffusion v1.4 (Latent Space): The method successfully recovered memorized images, including specific stock photography templates (e.g., rooms with yellow chairs, white sofas) and brand logos. Reverse image search confirmed these were duplicated web-crawl artifacts present in the training set. The recovered attractors formed a "dynamical fingerprint" of the model, recurring across different random seeds and initial conditions.
• CIFAR-10 DDPM (Pixel Space): The phenomenon was replicated in a smaller, pixel-space model. Cyclic denoising drove the system to memorized CIFAR-10 training images (e.g., a specific car), which were verified against known memorization benchmarks. The dynamics exhibited "two-level systems" behavior, hopping between a memorized image and its horizontally flipped variant (likely due to training augmentations).
• Amplitude Dependence:
  • Low $\gamma$: Trivial fixed points or limit cycles.
  • Intermediate $\gamma$: Simple, repeated artifacts (logos, web templates).
  • High $\gamma$: Rich, complex memorized images (product photos, detailed scenes).
• Prompt Removal Experiments: When cycling with known memorization prompts, the system converged to the memorized image. Upon removing the prompt, these images remained ultrastable (persisting for thousands of cycles), whereas prompts for non-memorized concepts (e.g., "Abbey Road") resulted in fluctuating concept attractors that decorrelated immediately after prompt removal.

Significance and Claims
The paper claims that cyclic denoising provides a new, dynamical route for probing memorization. It posits that memorized examples are not merely rare outputs of standard sampling but are deep, ultrastable attractors in the generative landscape that are rarely visited unless the model is driven by cyclic perturbations.

• Privacy and Auditing: The method serves as a practical tool for memorization auditing, capable of identifying copyrighted or sensitive training content without requiring knowledge of the training set or specific prompts.
• Threat Model: The authors argue that standard sampling may significantly underestimate memorization risks. An adversary with sampler-level access could use cyclic forcing to extract sensitive content that remains hidden during normal operation.
• Physics Connection: The work bridges generative AI and statistical physics, showing that the stability of learned representations in diffusion models mirrors the yielding transitions observed in driven disordered solids.

The authors remain modest regarding limitations, noting that the attack requires sampler-level control (not applicable to standard API endpoints that only return final samples) and that the systematic characterization of the attractor distribution (e.g., the fraction of total memorized data recoverable) requires further quantitative investigation.