When Do Diffusion Models learn to Generate Multiple Objects?

1. Problem Statement

Text-to-image diffusion models have achieved impressive visual fidelity for single-object generation but remain unreliable when generating multi-object compositions. Despite empirical evidence of failures in tasks like counting, attribute binding (e.g., "red cup and blue TV"), and spatial relations (e.g., "train under the desk"), the underlying causes are unclear.

The paper investigates whether these failures stem from:

1. Data Skewness/Imbalance: The model fails to learn individual concepts (e.g., specific counts or colors) because they appear infrequently in training data.
2. Compositional Generalization Failure: The model cannot recombine known concepts into unseen combinations (e.g., if it has seen "red sphere" and "blue cube" separately, can it generate "red sphere and blue cube" together?).
3. Scene Complexity: The difficulty of managing multiple objects simultaneously, regardless of data distribution.

2. Methodology

A. MOSAIC Framework

To isolate causal factors, the authors introduce MOSAIC (Multi-Object Spatial relations, AttrIbution, Counting), a controlled dataset generation framework. Unlike real-world datasets (e.g., LAION-2B) which have intertwined biases, MOSAIC explicitly parameterizes:

• Attribution: Binding colors to specific object types (e.g., sphere vs. cube).
• Spatial Relations: Defining angular positions between objects (e.g., 18° intervals).
• Counting: Generating specific numbers of objects (1 to 10).

The framework allows for systematic variation of:

• Dataset Size: 2k, 10k, 50k, 100k samples.
• Data Distribution: Uniform vs. Skewed (mimicking real-world frequency imbalances).
• Scene Complexity: "Base" (simple) vs. "Complex" (adding distractors/duplicates) vs. "Grid" (constraining objects to radial sectors to reduce spatial variability).
• Compositional Difficulty: Systematically holding out specific concept combinations (diagonals in a concept matrix) during training to test generalization to unseen pairs.

B. Experimental Setup

• Models: Two architectures were trained from scratch: a U-Net (approx. 90M parameters, score-matching objective) and a Diffusion Transformer (DiT) (approx. 90M parameters, flow-matching objective).
• Evaluation: Task-specific discriminative classifiers (CNNs/ResNets) were trained to evaluate generation accuracy.
• Real-world Validation: The findings were extended to:
  • Fine-tuning: Using LoRA on pre-trained Stable Diffusion 3 (SD3) on the SPEC benchmark.
  • Object Co-occurrence: A less controlled variant of MOSAIC with realistic object shapes, occlusions, and varied viewpoints.

3. Key Findings & Results

A. Concept Generalization (RQ1: Data Imbalance vs. Complexity)

• Scene Complexity is the Dominant Factor: Contrary to the hypothesis that data imbalance is the primary cause, scene complexity drives performance degradation.
  • Attribution and Spatial Relations (2 objects) generalize well even with small datasets (2k–10k) and skewed distributions.
  • Counting (up to 10 objects) fails significantly in low-data regimes (10k–50k), regardless of whether the data is uniform or skewed.
• Counting Dynamics: Counting exhibits a unique "early peak and collapse" behavior. In low-data regimes, the model initially learns to count but then degrades as training continues, failing to maintain distinct object instances. This is not due to memorization (which decreases as data scales) but a failure to generalize.
• Inductive Bias Helps: Introducing a Grid Layout (constraining objects to specific sectors) significantly stabilizes counting performance across all dataset sizes, suggesting that explicit spatial priors are necessary for robust counting.

B. Compositional Generalization (RQ2: Unseen Combinations)

• Generalization Collapses with Unseen Combinations: As the number of held-out concept pairs increases during training, the model's ability to generate unseen compositions degrades rapidly.
• Hierarchy of Difficulty: The difficulty of recombining concepts follows a strict order:
  1. Attribution (Most robust)
  2. Counting
  3. Spatial Relations (Most fragile; performance drops sharply even with few unseen combinations).
• Architecture Independence: These trends hold for both U-Net and DiT architectures, indicating a fundamental limitation in current diffusion models rather than a specific architectural flaw.
• Fine-tuning Limitations: Fine-tuning SD3 on realistic data (SPEC) showed that while spatial relations improve, counting accuracy deteriorates, reinforcing that counting remains a fragile capability even in pre-trained models.

4. Key Contributions

1. MOSAIC Dataset: A novel, controlled diagnostic framework that disentangles multi-object reasoning into distinct dimensions (counting, attribution, spatial relations) to enable causal analysis of data effects.
2. Disentangling Data vs. Complexity: The paper provides evidence that scene complexity, not just data imbalance, is the primary bottleneck for multi-object generation, particularly for counting tasks.
3. Compositional Generalization Limits: It demonstrates that diffusion models lack inherent mechanisms for robust compositional generalization; performance degrades linearly as the number of unseen training combinations increases, with spatial relations being the most vulnerable.
4. Inductive Bias Necessity: The results suggest that vanilla diffusion models require stronger inductive biases (e.g., spatial grids or layout conditioning) to handle multi-object generation reliably, especially in low-data regimes.

5. Significance

This work challenges the assumption that simply scaling up data or using larger models will solve multi-object generation failures. It highlights that:

• Current Diffusion Models are fundamentally limited in their ability to recombine concepts without explicit structural guidance.
• Data Design Matters: Simply having "more data" is insufficient if the data does not cover the combinatorial space of concepts or if the scene complexity exceeds the model's capacity to maintain object distinctness.
• Future Directions: The findings motivate the development of new architectures with stronger inductive biases (e.g., explicit layout conditioning, object-centric representations) and better data design strategies to ensure robust multi-object compositional generation.