Technical Summary: Initialization is Half the Battle: Generating Diverse Images from a Guidance Potential Posterior

1. Problem Statement

Despite the high fidelity of modern generative models, including Diffusion Models and Flow Matching models, they frequently suffer from mode collapse. Even with varying random seeds, generated samples often converge to stereotypic outputs, limiting creative potential and, in extreme cases, leading to the exact memorization of training data.

Existing training-free strategies to enhance diversity predominantly focus on intervening during the generation trajectory (e.g., manipulating guidance scales or repelling trajectories from one another). The authors identify a critical oversight in these approaches: the standard isotropic Gaussian initialization ($x_T \sim \mathcal{N}(0, I)$) is often agnostic to the guidance potential landscape. Geometrically, regions with high conditional guidance potential induce high volume contraction rates in the generative flow. Consequently, distinct stochastic trajectories initialized in these high-potential regions are attracted into a single dominant mode, causing collapse. Conversely, the authors hypothesize that initializing from low-potential basins allows trajectories to naturally diverge, preserving diversity.

2. Methodology: Diversity-Inducing Initialization (DivIn)

The paper proposes DivIn, a training-free, inference-time method that reframes the selection of initial noise not as drawing from a fixed prior, but as sampling from a guidance potential posterior.

2.1. Theoretical Formulation

The authors define a target distribution $p_{\text{diverse}}(x_T | c)$ that re-weights the standard Gaussian prior towards diversity-rich regions:
$$p_{\text{diverse}}(x_T | c) \propto \exp(-\tau \cdot U(x_T, c)) \cdot p(x_T)$$
where:

• $p(x_T)$ is the standard isotropic Gaussian prior.
• $\tau$ is a temperature hyperparameter controlling the trade-off between prior adherence and diversity maximization.
• $U(x_T, c)$ is a guidance potential acting as a proxy for the conditional guidance intensity at initialization.

2.2. The Guidance Potential Proxy

To ensure robustness and plug-and-play compatibility across different inference schedules (e.g., varying step counts) and architectures (Diffusion vs. Flow Matching), the potential $U(x_T, c)$ is defined in the estimated data space using a generalized Tweedie potential:
$$U(x_T, c) = \|\hat{x}_0(x_T, c) - \hat{x}_0(x_T, \emptyset)\|^2$$
Here, $\hat{x}_0$ represents the single-step denoising estimate of the clean data. This metric measures the Euclidean distance between the conditional and unconditional denoising estimates. Minimizing this potential penalizes initial states where the conditional guidance exerts an excessively strong attraction toward a dominant mode, biasing initialization toward flatter regions of the landscape.

2.3. Sampling via Langevin Dynamics

Since direct sampling from $p_{\text{diverse}}$ is intractable, DivIn employs Langevin Dynamics to iteratively transition samples from the initial Gaussian prior toward the target posterior. The update rule for the latent state $x_T$ at iteration $k$ is:
$$x_T^{(k+1)} = x_T^{(k)} - \eta \left( \tau \nabla_{x_T^{(k)}} U(x_T^{(k)}, c) + x_T^{(k)} \right) + \sqrt{2\eta}\xi^{(k)}$$
This mechanism balances three forces:

1. Diversity Force ($-\tau \nabla U$): Drives the latent away from high-potential basins toward flatter regions.
2. Prior Constraint ($-x_T$): Anchors the latent within the valid manifold of the original normal distribution, preventing off-manifold drift.
3. Stochastic Noise ($\sqrt{2\eta}\xi$): Injects noise to escape shallow local minima and encourages exploration.

Unlike deterministic optimization methods (e.g., Sharpness-Aware Initialization, SAIL) which collapse the distribution volume by seeking a single optimal seed, DivIn performs distributional posterior sampling, preserving entropy and enabling genuine diversity.

3. Key Contributions

1. Identification of Initialization as a Critical Factor: The paper reveals that mode collapse is significantly driven by the geometry of the initialization landscape, specifically the attraction of trajectories into high-potential regions defined by the conditional guidance.
2. Guidance Potential Posterior Formulation: The authors formulate the optimal initialization as a diversity-weighted posterior that re-weights the standard prior, utilizing a generalized Tweedie potential that is robust to inference step variations and unifies Diffusion and Flow Matching architectures.
3. DivIn Algorithm: A plug-and-play, training-free algorithm leveraging Langevin dynamics to sample from this posterior. It is orthogonal to existing trajectory-based diversity methods.
4. Theoretical Justification: The paper provides theoretical analysis (Theorem A.3) linking high conditional sharpness (curvature) to rapid entropy loss during the reverse process, demonstrating that sampling from low-potential regions preserves distribution volume.

4. Experimental Results

Extensive experiments were conducted on ImageNet-1K (class-to-image) and a general prompt dataset (text-to-image) using Stable Diffusion v1.4 and Stable Diffusion 3.5 Medium (Flow Matching).

• Diversity Improvement: DivIn consistently outperforms baseline initialization methods (like SAIL) and trajectory-based methods (Particle Guidance, CADS, Interval Guidance) in diversity metrics such as Vendi Score, Recall, and Coverage.
  • On ImageNet (SD v1.4), DivIn achieved a Vendi Score of 4.688 (vs. 4.265 for the base model), outperforming SAIL (4.549) and trajectory methods.
  • On text-to-image (SD v3.5), DivIn increased the Vendi Score from 1.803 to 1.864 for the base model.
• Quality Preservation: Unlike many diversity methods that trade quality for diversity, DivIn maintains or improves image fidelity. It achieved the lowest FID (16.158) among standalone baselines on ImageNet and maintained high CLIP and Aesthetic scores on text-to-image tasks.
• Orthogonality and Pareto Frontier: A key finding is that DivIn is orthogonal to trajectory-based interventions. Combining DivIn with methods like PG, CADS, or IG significantly expands the diversity-quality Pareto frontier, achieving higher diversity at the same fidelity levels than either method alone.
• Efficiency: DivIn incurs minimal computational overhead. With a single Langevin step ($K=1$), it adds only ~3% to the wall-clock time per image (e.g., 0.754s to 0.779s), whereas deterministic optimization methods like SAIL often require more complex rejection sampling or second-order approximations.
• Stability: Unlike SAIL, which is sensitive to early stopping thresholds and can lead to high-frequency artifacts due to off-manifold latents, DivIn remains stable and preserves the Gaussianity of the latent space.

5. Significance and Claims

The paper claims that initialization is "half the battle" in generating diverse images. By shifting the focus from the generation trajectory to the initialization landscape, DivIn addresses a root cause of mode collapse that trajectory-based methods neglect.

The authors emphasize that:

• DivIn is a universal, plug-and-play strategy compatible with both standard diffusion and modern flow matching models.
• It provides a distinct source of diversity not captured by trajectory interventions, allowing for the discovery of diverse modes that standard sampling misses.
• The method effectively balances the trade-off between diversity and quality, pushing the Pareto frontier outward without penalizing semantic alignment or image fidelity.
• It offers a more robust alternative to deterministic seed optimization (like SAIL) by maintaining distributional entropy through stochastic sampling.

The paper concludes that while DivIn currently relies on a temperature hyperparameter and adds minimal inference overhead, it represents a fundamental shift in how generative diversity can be enhanced, offering a powerful orthogonal tool to existing sampling mechanisms.