Technical Summary: Divide-and-Denoise

Problem Statement

The proliferation of pre-trained diffusion models offers significant opportunities for composition, allowing users to combine the expertise of multiple specialized models (e.g., a model trained on dogs and another on cats) to generate complex data that no single model could produce alone. However, naively combining these models presents two primary challenges:

1. Dominance: One model may overpower the others, leading to the neglect of specific concepts.
2. Disagreement: Models may conflict, resulting in incoherent outputs, overlapping objects, or mismatched attributes (e.g., a "dog" and "cat" merging into a single incoherent blob).

Existing approaches, such as defining composite distributions via product densities or user-defined spatial partitions (e.g., MultiDiffusion), often fail to preserve the distinct characteristics of each model when conflicts arise. User-defined partitions are cumbersome, often infeasible for complex domains (like protein structures), and do not adapt to the specific strengths or weaknesses of the models at different stages of generation.

Methodology: Divide-and-Denoise

The authors propose Divide-and-Denoise, a game-theoretic framework that coordinates multiple pre-trained diffusion models during the sampling process. The method operates via two coupled processes that evolve in tandem at every timestep $t$:

1. The Division Process (Fair Allocation)

Instead of fixed partitions, the method infers an optimal "division of labor" online. The latent image is treated as a set of $m$ features (goods) to be allocated among $n$ models (players).

• Utilities: Each model $i$ is assigned a utility $u_{ij}(x, t)$ for each feature $j$ of the latent $x$. The paper defines two types of utilities:
  • Score-based: Derived from the magnitude of the score difference between conditional and unconditional guidance (applicable to any conditional diffusion model).
  • Attention-based: Derived from cross-attention maps (specific to models with cross-attention conditioning).
• Fair Division Game: At each step, the method solves a fair division game to determine an allocation matrix $Q$. The objective is to maximize the expected total utility (efficiency) subject to fairness constraints (e.g., proportionality, envy-freeness).
• Optimization: The allocation $Q_t$ is updated by solving a constrained optimization problem regularized by Kullback-Leibler (KL) divergence from the previous timestep's allocation to ensure temporal smoothness. The solution is derived via a dual problem, yielding a decomposable allocation where each feature is assigned a fractional weight to each model.

2. The Composite Denoising Process

Once the allocation $Q_t$ is determined, the method constructs a composite denoising kernel $p^c_t$ that aligns with this division.

• Alignment: The composite update is formulated to keep the denoising step for features allocated to model $i$ close to model $i$'s proposal, while maximizing the total utility.
• Update Rule: Under simplifying assumptions (factorized distributions and linear utility), the optimal composite mean $\mu^c_t$ is derived as a weighted sum of individual model means plus a guidance term:
  $$\mu^c_t = \sum_{i=1}^n \mu^i_t(x_t) \odot Q^i_t + \frac{\sigma^2_t}{\alpha_t} \nabla_{x_t} U_t(x_t, Q)$$
  Here, $\odot$ denotes feature-wise product, and the second term acts as a guidance update based on the gradient of the total utility.
• Fictitious Player: To handle low-utility regions where models might conflict or lack preference, a "fictitious player" is introduced with a uniform utility. This player's proposal is the average of all models, encouraging collaboration in ambiguous regions without enforcing fairness constraints on it.

Key Contributions

1. Inference-Time Coordination Algorithm: A method to coordinate multiple pre-trained diffusion models with differing expertise without requiring shared weights, architectures, or training data.
2. Coupled Processes: The integration of a dynamic division process (solving a fair division game) with a composite denoising process (aligning models to their assigned regions).
3. Utility Formulations: General definitions for model utilities applicable to any conditional diffusion model (score-based) and a specific instantiation for cross-attention models (attention-based).
4. Empirical Validation: Demonstration that a team of single-concept models, coordinated via this method, outperforms existing composition techniques and even multi-concept models on standard benchmarks.

Experimental Results

The method was evaluated on conditional image generation using Stable Diffusion 2.0 and Diffusion Transformer (DiT) models.

• Benchmarks: Evaluation was conducted using the GenEval benchmark (focusing on object presence and attribute binding), CLIP-Score, VQAScore, and ImageReward.
• Performance:
  • Object Generation: In the "Concepts as Objects" setting (e.g., generating an image with a handbag and a sports ball), Divide-and-Denoise significantly outperformed baselines like Averaging, Composable Diffusion, and MultiDiffusion. For 3-object generation, it achieved 59.5% on the GenEval "%images" metric compared to 14.75% for Multi-Concept Diffusion and 14.00% for MultiDiffusion.
  • Attribute Binding: When generating objects with specific colors (e.g., "orange horse"), the method successfully bound attributes to the correct objects, outperforming baselines that often suffered from attribute leakage or missing objects.
  • Conflict Resolution: In scenarios with conflicting concepts (e.g., "desert" vs. "snowy mountain"), the method produced diverse spatial layouts and higher quality images than baselines, avoiding the blending of incompatible concepts.
  • Fairness: The inclusion of fairness constraints (proportionality) further improved metrics, particularly as the number of models increased, preventing any single model from dominating the generation.

Significance and Claims

The paper claims that Divide-and-Denoise provides a general recipe for controllable and reusable generative modeling. Its significance lies in:

• Resolving Conflicts: It effectively resolves conflicts between models without requiring manual intervention or ground-truth partitions.
• Model Agnosticism: The formulation is not tied to specific model architectures or conditioning mechanisms, working with both latent diffusion models (Stable Diffusion) and transformer-based models (DiT).
• Cooperative Interaction: It highlights that cooperative interaction between pre-trained models, mediated by game-theoretic fairness, can yield results superior to training a single large multi-concept model.
• Scalability: The method scales to multiple models (tested up to 4) and domains, with potential extensions to text-to-graph, text-to-audio, and audio-to-image generation.

The authors note that while the method introduces computational overhead (primarily in the division stage), runtime optimizations (such as restricting the division and guidance steps to early timesteps) can significantly reduce costs while maintaining performance gains.