Technical Summary: ELDiff

Problem Statement

In multi-object text-to-image (T2I) diffusion, ensuring semantic consistency between textual prompts and generated visual content is critical. While token-supervised diffusion models (e.g., TokenCompose) have improved this by learning object-wise consistency between image content and segmentation maps, they face two primary limitations:

1. Segmentation Map Bias: Grounding segmentation maps generated by tools like SAM are not always reliable. Directly enforcing consistency with these maps causes models to overfit to spurious or incorrect regions.
2. Semantic Overlap Conflict: When multiple objects in a prompt correspond to overlapping spatial regions (e.g., a "cat on a chair"), optimizing consistency for each noun token independently creates contradictory gradients. This leads to semantic competition and degraded image quality.

Existing methods often lack mechanisms to handle the uncertainty of segmentation labels or to explicitly quantify and resolve conflicts between overlapping semantic concepts.

Methodology

The authors propose ELDiff, an end-to-end fine-tuning strategy that integrates Evidential Deep Learning (EDL) and Dempster-Shafer Theory (DST) into the T2I diffusion pipeline. The approach introduces two novel loss functions to address the identified problems without requiring additional inference-time manipulations.

1. Pixel Evidence Loss ($L_{PixEvi}$)

To mitigate segmentation map bias, ELDiff treats the cross-attention maps of noun tokens as evidence distributions rather than deterministic predictions.

• Mechanism: The cross-attention map is fed into an evidential network to output evidence values ($e$), which parameterize a Dirichlet distribution.
• Uncertainty Metric: Using Subjective Logic, the model derives a belief mass ($b$) and an uncertainty value ($u$) for each pixel. High uncertainty indicates unreliable labels (e.g., biased segmentation).
• Loss Function: The loss combines a cross-entropy term ($L_{ce}$), an evidential cross-entropy ($L_{ice}$), and a Kullback-Leibler divergence ($L_{KL}$). By optimizing the expected prediction under the Dirichlet distribution, the model learns to remain cautious when uncertainty is high, preventing overfitting to incorrect segmentation masks.

2. Token Conflict Loss ($L_{TokCon}$)

To resolve semantic overlap conflicts, ELDiff aggregates evidence from different noun tokens using Dempster-Shafer Evidence Theory.

• Mechanism: The cross-attention maps of different nouns are treated as distinct pieces of evidence. The model calculates a conflict coefficient ($K$) in overlapping regions using the DST combination rule.
• Loss Function: The total loss consists of:
  • Intra-object consistency ($L_{intra}$): Encourages attention maps to focus on their target regions.
  • Inter-object conflict ($L_{inter}$): Penalizes conflicting activations in overlapping areas based on the calculated conflict coefficient.
• Goal: This allows the model to adaptively adjust the spatial distribution of attention maps, effectively decoupling semantically overlapping concepts.

The final training objective combines the standard diffusion loss ($L_{LDM}$) with the proposed losses:
$$L_{ELDiff} = L_{LDM} + L_{PixEvi} + L_{TokCon}$$

Key Contributions

• Problem Formulation: The paper explicitly identifies and formulates the issues of segmentation map bias and semantic overlap conflict in object-wise consistency constraints for T2I diffusion.
• ELDiff Framework: A new fine-tuning pipeline that seamlessly integrates into existing T2I diffusion models (e.g., Stable Diffusion v1.4, v2.1, v3.5, SDXL, and Qwen-Image).
• Pixel Evidence Loss: A novel loss that restrains overconfidence in unreliable labels by modeling cross-attention maps as Dirichlet distributions and utilizing pixel-level uncertainty estimation.
• Token Conflict Loss: A mechanism to weaken semantic overlap conflicts by treating noun tokens as evidences and using DST to measure and optimize a conflict factor.
• Efficiency: The method improves multi-object composition without incurring additional inference-time costs or requiring extra manipulations during generation.

Experimental Results

Extensive experiments demonstrate that ELDiff outperforms existing training-based and training-free T2I diffusion models across multiple backbones and benchmarks:

• Multi-Category Instance Composition: On Stable Diffusion v1.4, ELDiff achieves a state-of-the-art Object Accuracy (OA) of 0.5563, surpassing TokenCompose (0.5215). It also shows significant improvements in MG2-MG5 metrics.
• Photorealism and Consistency: ELDiff achieves lower FID scores (19.25 on COCO, 55.13 on Flickr) and higher CLIP scores compared to baselines, indicating better image quality and text-image alignment.
• Generalization: The method demonstrates strong generalization when applied to SD v2.1, SD v3.5, and the stronger Qwen-Image backbone, consistently outperforming other fine-tuning strategies (e.g., CoMat, IterComp, TokenCompose) on T2I-CompBench and GenEval benchmarks.
• Efficiency: The method maintains inference latency comparable to the base models (e.g., ~7.5s for 50 DDIM steps), unlike other methods that require longer inference times.
• User Study: In evaluations on SD v2.1 and v3.5, ELDiff received the highest preference rates for image realism, compositionality, and overall quality from human evaluators.

Significance and Claims

The paper claims that ELDiff provides a robust solution for multi-object composition by leveraging uncertainty metrics and conflict detection. By addressing the inherent unreliability of segmentation maps and the contradictions arising from overlapping semantics, ELDiff enhances the fault tolerance of the diffusion process. The authors emphasize that their approach offers a seamless integration into existing training pipelines, improving generation quality for complex scenes without the computational overhead of additional inference steps. While the method shows superior performance in object composition, the authors modestly acknowledge that attribute binding (e.g., specific colors or textures) remains an area for future improvement.