Technical Summary: When Are Two Scores Better Than One? Investigating Ensembles of Diffusion Models

Problem Statement

While ensembling is a well-established technique in supervised learning for improving model robustness and accuracy by aggregating multiple predictors, its application to unconditional score-based diffusion models remains largely unexplored. Standard strategies to improve diffusion models typically involve scaling up model depth or training duration, which incurs significant computational costs. The authors investigate whether ensembling multiple diffusion models—specifically by aggregating their estimated score functions—provides tangible benefits for generative modeling, particularly in terms of perceptual sample quality.

Methodology

The study employs a "democratic" ensemble framework where $K$ independently trained models contribute equally to the aggregation process. The research covers both deep learning-based approaches (neural networks) and non-deep approaches (decision trees).

1. Ensemble Construction

• Deep Ensembles (DE): The authors train $K$ U-Nets (or ADM architectures) independently on the same dataset using different random seeds. They also explore Monte Carlo (MC) Dropout as a lightweight alternative to training separate networks.
• Decision Tree Ensembles: For tabular data, they utilize Random Forests, training distinct trees at various noise levels to approximate the score function.

2. Aggregation Strategies

The core of the investigation lies in how to combine the $K$ score predictors $\{s^{(k)}_\theta(x, t)\}_{k=1}^K$ at each timestep $t$ during the reverse diffusion process. The paper evaluates several aggregation rules:

• Arithmetic Mean: The standard approach, averaging scores coordinate-wise.
• Geometric Mean & Median: Alternative coordinate-wise aggregations.
• Dominant Feature: Selecting the component with the largest absolute value across all experts for each coordinate.
• Mixture of Experts (MoE): Randomly selecting a single expert model prior to the sampling run.
• Alternating Sampling: Randomly selecting a different expert at each noise level $t_n$.
• Sum of Scores: Summing the scores (interpreted as an unnormalized product of experts).

3. Theoretical Analysis

The authors provide theoretical insights into the relationship between score aggregation and density composition. They analyze whether averaging scores corresponds to sampling from a "Product of Experts" (PoE) distribution. They prove that diffusion (adding noise) and density composition (PoE) do not commute, meaning that averaging scores at intermediate timesteps does not strictly correspond to sampling from the geometric mean of the initial data distributions.

4. Experimental Setup

Experiments were conducted on:

• Image Datasets: CIFAR-10 (32x32) and FFHQ (256x256).
• Tabular Datasets: Iris, Airfoil Self-Noise, and Wine Quality datasets.
• Metrics: Fréchet Inception Distance (FID), Kernel Inception Distance (KID), Denoising Diffusion Score Matching (LDDSM) loss, and Wasserstein distance (for tabular data).

Key Results

1. Limited Improvement in Perceptual Quality

Contrary to the success of ensembling in supervised learning, the authors find that ensembling diffusion models fails to consistently enhance perceptual quality metrics (FID, KID).

• On CIFAR-10 and FFHQ, the arithmetic mean of scores generally yields marginal or no improvement over the average individual model and never surpasses the best single model.
• In some cases (e.g., CIFAR-10 with specific configurations), ensembling performed worse than the weakest individual model.
• This holds true across various aggregation strategies (Geometric mean, Median, Dominant feature), with the exception of the Mixture of Experts (MoE) in specific contexts.

2. Improvement in Training Objective

While perceptual metrics stagnate, the LDDSM loss consistently decreases as the ensemble size $K$ increases. This aligns with theoretical expectations based on Jensen's inequality, where averaging estimators reduces variance in the score matching objective. However, this reduction in training loss does not translate to better sample quality.

3. Specific Success Cases

• Mixture of Experts (MoE): On FFHQ, MoE outperformed even the best individual model. This suggests that leveraging inter-model variability by sampling from a single expert per run can be beneficial, though it does not constitute a true "ensemble" in the sense of combining predictions for a single sample.
• Tabular Data (Dominant Feature): For Random Forests on tabular data, the Dominant Feature aggregation strategy significantly outperformed arithmetic mean and other methods. The authors attribute this to a systematic underestimation of noise magnitude by simple averaging, which the Dominant strategy corrects.

4. The Disconnect Between Loss and Quality

The study highlights a fundamental disconnect: the score matching loss (what the model optimizes) and perceptual quality metrics (how the model is evaluated) evolve differently. Perturbing the score function led to chaotic variations in FID that could not be predicted by changes in the loss, reinforcing that high likelihood or low score-matching error does not guarantee high perceptual quality.

Significance and Claims

The paper's primary contribution is a rigorous empirical and theoretical investigation demonstrating that standard ensembling techniques do not provide the same benefits for diffusion models as they do for supervised models.

• Modest Gains: The authors claim that ensembling yields only marginal, if any, improvements in sample quality, despite consistent reductions in the training objective.
• Theoretical Insight: They clarify that step-wise score averaging does not equate to sampling from a Product of Experts distribution due to the non-commutativity of diffusion and density composition.
• Cost-Benefit Analysis: Given the substantial increase in training and sampling costs, standard ensembling offers limited returns for unconditional diffusion models.
• Exceptions: The paper identifies specific niches where ensembling helps: (1) when models are sufficiently complementary (MoE), and (2) when using specific aggregation rules for tabular data (Dominant Feature) to correct bias.

The authors conclude that while ensembling improves score estimation, the gap between score matching and perceptual quality prevents these improvements from manifesting in better generated images, suggesting that future progress requires strategies beyond simple score aggregation.