Ar2Can: An Architect and an Artist Leveraging a Canvas for Multi-Human Generation

1. Problem Statement

Current text-to-image diffusion models struggle significantly with multi-human generation. When prompted to generate scenes with multiple distinct individuals (e.g., "five friends having coffee"), existing models frequently fail due to:

• Identity Merging: Faces blend together, losing distinct identities.
• Identity Swapping: Reference identities are assigned to the wrong spatial locations.
• Count Errors: The model generates the wrong number of people.
• Spatial Conflicts: Without explicit constraints, models conflate "where" people should be with "what" they look like, leading to overlapping or duplicated faces.

Existing solutions either require explicit user annotations at inference time (limiting usability), excel only at single-subject personalization, or fail on recent benchmarks like the MultiHuman-Testbench.

2. Methodology: The Ar2Can Framework

Ar2Can introduces a novel two-stage framework that disentangles spatial planning from identity rendering. The system consists of three core components: an Architect, an Artist, and a Canvas.

A. The Architect (Spatial Planning)

The Architect generates a structured spatial layout (bounding boxes and optionally poses) based on the text prompt, determining where each person should appear before rendering begins. The paper proposes two variants:

1. Architect-A (LLM-based): Uses Qwen-2.5 (0.5B) fine-tuned via Supervised Fine-Tuning (SFT). It treats layout generation as an autoregressive task, predicting bounding box coordinates using special tokens (<SoL>, <EoL>, <C>). It excels at count accuracy due to strong language understanding.
2. Architect-B (T2I-based): Uses Flux-Schnell fine-tuned via Group Relative Policy Optimization (GRPO). It leverages the 2D nature of diffusion models to generate plausible spatial distributions and poses. It excels at spatial diversity and action alignment.

B. The Artist (Identity-Preserving Rendering)

The Artist is a single Flux-Kontext model trained to synthesize the final photorealistic image conditioned on the Architect's layout, the text prompt, and reference identity images.

• Training Strategy: Instead of Supervised Fine-Tuning (SFT), the Artist is trained using GRPO with a compositional reward function. This allows the model to optimize non-differentiable objectives like exact person counts and identity matching.
• Compositional Rewards:
  1. Count Accuracy: Rewards exact matching of the generated person count to the target.
  2. Prompt Alignment (HPSv3): Ensures the image matches the text description and aesthetic quality.
  3. Spatially-Grounded Face Matching (Key Innovation):
     • Uses the Hungarian Algorithm to match predicted bounding box centroids (from the Architect) with detected face centroids (from the generated image).
     • Computes ArcFace similarity between reference identities and the matched generated faces.
     • This prevents "copy-paste" artifacts by allowing flexible spatial positioning while enforcing identity consistency.
  4. Pose Reward: Encourages frontal poses to reduce unnatural artifacts.

C. Efficiency Optimizations

• Token Dropping & Sharing: To reduce computational cost, the model only processes tokens within the required spatial regions. For overlapping faces, it assigns identical RoPE positional encodings, forcing the model to learn natural occlusion and depth ordering rather than pasting identities at the same coordinate.
• Curriculum Learning: Training starts with simpler scenes (2-3 people) and gradually introduces more complex groups (up to 7 people) to stabilize learning.

D. Data Strategy

A major contribution is the ability to train without real multi-human images. The authors curate a hybrid dataset by:

1. Generating synthetic multi-person scenes using DisCo.
2. Replacing synthetic faces with real reference faces from curated datasets (multi-view and single-view).
3. Using this hybrid data to reinforce the Artist's predictions via RL.

3. Key Contributions

1. Disentangled Architecture: A two-stage framework separating spatial planning (Architect) from identity rendering (Artist), solving the fundamental limitation of conflating layout and appearance in single-stage models.
2. Two Architect Variants: Offering a trade-off between language-based count accuracy (Architect-A) and 2D spatial diversity (Architect-B).
3. Novel Reward Mechanism: A Hungarian-based spatially-grounded face matching reward that couples identity similarity with spatial proximity, effectively preventing identity merging and swapping.
4. Synthetic Data Efficiency: Demonstrating state-of-the-art performance using primarily synthetic data and real reference faces, eliminating the need for massive proprietary multi-human datasets.
5. Token Sharing Strategy: A mechanism to handle occlusions and reduce inference time by 2x while maintaining high quality.

4. Experimental Results

Evaluated on the MultiHuman-Testbench (MHTB) and Multi-ID-Test:

• Count Accuracy: Ar2Can (Arch-A) achieves 90.2%, outperforming all baselines (e.g., WithAnyone at 89.8%, GPT-Image-1 at 87.9%).
• Identity Preservation: Ar2Can (Arch-B) achieves a Multi-ID score of 68.2, significantly surpassing the previous best open-source method (MH-OmniGen at 54.5) and proprietary models like GPT-Image-1 (28.8).
• Unified Metric: Ar2Can achieves the highest unified scores (72.4 and 72.2), balancing identity, prompt alignment, and quality.
• Latency: With token sharing, Ar2Can achieves a total inference time of 15.5s (Arch-A) on an A100 GPU, offering a superior quality-speed trade-off compared to slower baselines like UMO-OmniGen2 (74s).
• Qualitative: The method consistently preserves identities and follows complex prompts without the merging or hallucination seen in other methods.

5. Significance

Ar2Can represents a paradigm shift in multi-human generation by moving away from end-to-end monolithic generation toward a modular, planning-based approach.

• Solves the "Identity Merging" Problem: By explicitly grounding faces to spatial plans before rendering, it eliminates the conflation of identity and location that plagues current diffusion models.
• Data Efficiency: It proves that high-fidelity multi-human generation does not require access to massive, proprietary datasets of real group photos, making the technology more accessible and privacy-friendly.
• Scalability: The framework scales effectively to larger groups (5+ people) where other methods fail, maintaining consistent identity preservation and spatial coherence.
• Generalization: The approach extends beyond humans to multi-object scenes, demonstrating emergent capabilities in handling occlusion and depth ordering.

In summary, Ar2Can sets a new state-of-the-art for personalized multi-human image generation by combining the reasoning power of LLMs, the spatial priors of diffusion models, and the optimization power of Reinforcement Learning.