💻 computer science

Composing People Together: Iterative Pose-Image Generation for Multi-Person Interaction Scenes

This paper introduces a dual pose-image generation framework that leverages person-centric structural priors and an iterative scene construction scheme to overcome the limitations of current text-to-image models in creating semantically diverse and compositionally accurate multi-person interaction scenes.

Original authors: Wenxuan Peng, Bharath Hariharan, Hadar Averbuch-Elor

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenxuan Peng, Bharath Hariharan, Hadar Averbuch-Elor

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very talented but slightly confused artist how to paint a complex scene with many people interacting, like a dance, a rugby match, or a priest washing another's hands.

Current AI art generators (like SDXL or FLUX) are like that artist: they are great at painting single people or simple landscapes. But when you ask them to paint a group of people doing specific things together, they often get lost. They might forget one person, mix up who is holding whose hand, or make everyone stand in the exact same boring pose. It's like asking a choir to sing a complex harmony, and they all end up singing the same note or forgetting their parts.

This paper, "Composing People Together," introduces a new way to guide the AI so it can get these group interactions right. Here is how they did it, using simple analogies:

1. The "Skeleton and Skin" Trick (Dual Pose-Image Generation)

Usually, when an AI tries to draw a person, it just guesses the shape and the clothes at the same time. It's like trying to sculpt a statue while simultaneously painting it; if the shape is wrong, the paint looks wrong, too.

The authors' solution is to make the AI draw two things at once:

The Skeleton (Pose): A simple stick-figure drawing showing exactly where the arms, legs, and heads are.
The Skin (Image): The final, colorful, realistic photo.

The Analogy: Think of it like building a house. First, you build the sturdy wooden frame (the skeleton). Once the frame is perfect, you hang the drywall and paint the walls (the skin). By forcing the AI to "draw the skeleton" first, it ensures the people are standing in the right positions before it worries about their clothes or faces. This keeps the structure solid so the final picture doesn't collapse.

2. The "Name Tag" System (Cross-Modal Alignment)

Even with a skeleton, the AI can get confused about who is doing what. If you say, "The man in the red shirt lifts the woman," the AI might accidentally give the red shirt to the woman or make the man lift the wrong person.

The authors created a special "Name Tag" system using a technique called Rotary Positional Encoding (RoPE).

The Analogy: Imagine every person in the scene has a unique colored wristband (a "Role ID"). The text describing the man in red, the stick-figure arm of the man in red, and the pixels of the man's red shirt all get the same colored wristband.
This forces the AI to realize: "Oh, this text, this arm, and this shirt all belong to the same person." It prevents the AI from mixing up the roles, ensuring the priest washes the right hands and the rugby player tackles the right opponent.

3. The "One-by-One" Assembly Line (Iterative Generation)

Trying to generate a whole group of people in one giant leap is like trying to solve a 1,000-piece puzzle all at once. It's too hard, and the AI often misses pieces.

Instead, this method builds the scene one person at a time.

The Analogy: Imagine building a train. You don't try to snap all the cars together in one second. You start with the engine (Person 1). Once the engine is there, you add the first car (Person 2) attached to it. Then you add the second car (Person 3) attached to the second car.
The AI looks at the "skeleton" of the people already drawn and adds the next person based on that. This breaks a giant, scary problem into a series of small, easy steps. It ensures that as the scene grows, the new person fits perfectly with the people already there.

4. The New "Test" (DrawWaldoWorlds)

To prove their method works, the authors couldn't just use old tests because those tests didn't really check if people were interacting correctly. They built a new test called DrawWaldoWorlds.

The Analogy: It's like a "Where's Waldo?" game but for AI. They give the AI a very specific description (e.g., "The man in the blue hat is holding the umbrella over the woman in the red coat") and ask the AI to draw it. Then, they check: Did the AI get the hat color right? Is the umbrella actually over the woman? Did it forget the woman entirely?
They tested their method against top competitors (like FLUX and SDXL) and found that their method was much better at getting the details right and creating diverse, non-repetitive scenes.

Summary

In short, the paper says: "To get AI to draw complex group interactions, don't just ask it to 'paint a picture.' Instead, make it build a skeleton first, tag every person with a unique ID so it knows who is who, and add people one by one like building a train."

The result is images where people are actually interacting correctly, rather than just standing next to each other in a confused pile.

Technical Summary: Composing People Together

Problem Statement

Current text-to-image diffusion models struggle to generate semantically diverse and compositionally accurate scenes involving multi-person interactions. While recent models (e.g., FLUX, SDXL) have improved in capturing individual appearance, they frequently fail to correctly realize specific roles, spatial relationships, and mutual affordances between people. Common failure modes include omitting entities, misaligning actions (e.g., a priest washing hands without an assistant pouring water), collapsing into repetitive poses, and failing to ground fine-grained interaction semantics. Existing approaches relying on coarse spatial layouts (bounding boxes) or external structural cues lack the granularity required to model the vast combinatorial space of human poses, expressions, and social contexts.

Methodology

The authors propose a framework that bridges this gap by introducing person-centric structural priors directly into pretrained diffusion transformers. The core of the method consists of three integrated components:

1. Dual Pose–Image Representation

Instead of treating human pose as an external condition, the model jointly predicts a 2D pose visualization image and its corresponding RGB image.

Architecture: Built upon the FLUX Diffusion Transformer (DiT), the method introduces a parallel pose stream initialized from the image-branch parameters.
Training Strategy: The pretrained text and image streams are frozen to preserve visual quality. Only the pose stream (using LoRA) and specific projection layers are trainable.
Joint Reasoning: The model learns to reason about interaction-relevant structure explicitly during generation rather than relying on implicit appearance cues. The pose is represented as a rendered 2D image (OpenPose format) to maximize compatibility with the pretrained VAE.

2. Cross-Modal Alignment via Role-Aware RoPE

To ensure consistent grounding across text, pose, and image modalities, the authors modify the 3D Rotary Positional Encoding (RoPE) used in the transformer.

Mechanism: The temporal coordinate $\tau$ in the standard $(\tau, x, y)$ encoding is repurposed to establish a unified role identity.
Implementation: All tokens associated with a specific person (text descriptions, pose tokens, and image tokens within a bounding box) are assigned a shared $\tau$ index (e.g., $\tau = i$ for the $i$ -th person). Tokens outside bounding boxes use $\tau = 0$ . This binds per-person textual descriptions to their corresponding visual and structural regions without altering the backbone architecture.

3. Iterative Scene Construction

Leveraging the decomposable nature of pose representations, the authors employ an iterative generation scheme to construct complex scenes.

Process: The scene is built incrementally, adding one person at a time. At stage $i$ , the model conditions on the previously generated pose state $P_{i-1}$ (representing people $1$ to $i-1$ ), the global interaction description, and the specific description for the $i$ -th person.
Advantage: This transforms the complex multi-person reasoning problem into a sequence of simpler sub-tasks, effectively decomposing the generation complexity and reducing semantic entanglement.
Training: Ground-truth images are decomposed into intermediate pose states ( $P_1, \dots, P_N$ ). The model is supervised on the pose at every stage, but only on the final RGB image at the last stage.

Key Contributions

Dual Pose–Image Representation: A novel architecture that injects person-centric structural priors into diffusion transformers, enabling iterative multi-person generation by jointly predicting pose and appearance.
DrawWaldoWorlds Benchmark: A new evaluation benchmark derived from the Who's Waldo dataset, featuring three tiers of prompt specificity (A, B, C) to evaluate alignment and diversity in interaction-centric multi-human synthesis.
Performance Improvements: Demonstrated significant gains in both semantic alignment (faithfulness to prompt) and scene diversity compared to state-of-the-art text-to-image, editing, and layout-controlled baselines.

Results

The method was evaluated on DrawWaldoWorlds and the MultiHuman-Testbench.

Semantic Alignment: The proposed method achieved the highest VQA Accuracy and VQA Similarity scores across all prompt tiers. It maintained high alignment even as prompt complexity increased (Tier C), whereas baselines often failed to capture all entities or their interactions.
Diversity: The method outperformed most baselines in diversity metrics (DINO Diff, LPIPS, GRADE), producing varied yet semantically coherent realizations of the same prompt.
User Study: In a blind comparison against FLUX [dev] and FLUX Kontext [dev], human judges preferred the proposed method's outputs in 63.3% and 79.0% of cases, respectively, citing better adherence to the text.
Ablation Studies: Removing any of the three core components (pose supervision, iterative composition, or role-aware RoPE) resulted in measurable performance drops, confirming that all elements are necessary for strong semantic alignment.

Significance

The paper argues that explicit, person-centric structural abstractions serve as a powerful bridge between high-level semantic intent and low-level visual generation. By disentangling interaction structure from appearance and leveraging iterative composition, the approach offers a scalable solution to the combinatorial complexity inherent in multi-entity scenes. The authors suggest that integrating such structured, decomposable representations as first-class modalities within generative models opens new opportunities for more faithful, interpretable, and controllable synthesis, not only for human interactions but potentially for complex scenes involving richer dynamics and diverse interacting entities.