🤖 AI

Social Structure Matters in 3D Human-Human Interaction Generation

This paper proposes the "Solo-to-Social" framework, which addresses the challenges of text-driven 3D human-human interaction generation by employing a planner-executor paradigm where an LLM infers the underlying social structure (phases and roles) and a motion executor grounds this structure into physically plausible, coordinated two-person motions.

Original authors: Zhongju Wang, Beier Wang, Yatao Bian, Pichao Wang, Zhi Wang, Daoyi Dong, Hongdong Li, Huadong Mo, Zhenhong Sun

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhongju Wang, Beier Wang, Yatao Bian, Pichao Wang, Zhi Wang, Daoyi Dong, Hongdong Li, Huadong Mo, Zhenhong Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Two People, One Text

Imagine you ask a computer to draw a picture of two people hugging. If you just ask it to "draw a hug," it might draw two people standing next to each other, or maybe one person hugging a wall.

In the world of 3D animation, making two people move realistically together based on a text description (like "one person approaches, hugs, and then lets go") is incredibly hard. Previous methods treated this like two separate solo dances happening at the same time. But human interaction isn't just two solo dances; it's a conversation of movement.

The authors argue that the secret ingredient missing from these animations is Social Structure.

What is "Social Structure"?

Think of a social interaction like a play in a theater.

Solo Motion: This is like an actor rehearsing their lines alone. They know how to walk, wave, or jump.
Social Structure: This is the script and the stage directions. It tells you:
1. The Phases: The story has acts (e.g., Approach, Contact, Release). You can't hug someone before you walk up to them.
2. The Roles: In every scene, who is doing what? Is Person A the "hugger" and Person B the "hug-ee"? Is Person A the "attacker" and Person B the "defender"?

Without this structure, the computer might make Person A try to hug while Person B is walking away, or make them face the wrong way. The result looks like two people who don't know each other, rather than two people interacting.

The Discovery: The "Thinker" vs. The "Dancer"

The researchers tested a very smart AI (a Large Language Model or LLM) to see if it could handle this whole job alone. They found a clear split in abilities:

The LLM is a Great "Director" (The Thinker):
If you ask the LLM to plan the scene, it is excellent. It can break the story down into phases ("First they walk, then they touch, then they separate") and assign roles ("Person A initiates, Person B receives"). It understands the logic of the interaction perfectly.
The LLM is a Bad "Dancer" (The Executor):
However, when you ask the LLM to actually generate the 3D movement (the coordinates of the bones), it fails. It produces stiff, static, or wobbly movements. It's like a director who knows exactly how a scene should look but has never learned how to move their own body.

The Solution: The "Planner-Executor" Team

Because the LLM is good at thinking but bad at moving, the authors created a two-part team called "Think with LLM, Move with Motion Skill."

Part 1: The Planner (The LLM)

The LLM acts as the Screenwriter and Director.

It takes a simple text prompt (e.g., "Two people high-five").
It breaks it down into a structured plan:
- Phase 1 (Approach): Person A walks toward Person B.
- Phase 2 (Contact): Person A raises hand, Person B raises hand.
- Phase 3 (Release): They pull hands apart.
Crucially, it assigns specific roles to each person so they know who is doing what.

Part 2: The Executor (The Motion Model)

The Executor is a specialized robot trained on thousands of hours of solo human movement (like a professional dancer who has practiced millions of steps).

Instead of teaching this robot to learn from scratch, the authors gave it a specialized upgrade (called LoRA).
This upgrade allows the robot to listen to the Director's plan.
The Magic Trick: The robot doesn't just move; it moves relative to the other person.
- Self-Conditioning: It remembers what it just did to ensure smooth transitions (no jerky jumps between phases).
- Partner-Conditioning: It looks at where the other person is right now and adjusts its own movement to match. If the partner steps back, the robot steps forward to meet them.

The Result: A Coordinated Dance

By combining the Director's script (Social Structure) with the Dancer's muscle memory (Motion Skill), the system creates animations where:

The timing is perfect (they don't miss each other's hands).
The roles make sense (the hugger actually hugs, the hug-ee actually accepts it).
The movement flows naturally, just like real humans.

Summary Analogy

Imagine you want to build a house.

Old Methods: You hired two bricklayers and told them, "Build a house." They each built a wall, but the walls didn't connect, and the roof was floating.
The LLM Alone: You hired an architect who wrote a perfect blueprint but couldn't hold a trowel. The blueprint was great, but the house never got built.
This Paper's Method: You hired the Architect to write a detailed, step-by-step plan with specific roles for each worker. Then, you hired a Master Builder who is an expert at laying bricks. The Architect tells the Builder exactly what to do at every step, and the Builder uses their expert skills to lay the bricks perfectly. The result is a house that stands up and looks exactly as intended.

The paper proves that for two people to interact realistically in 3D, you need to explicitly model the social script (phases and roles) and let a specialized motion model execute it, rather than trying to force a text AI to do the physical dancing itself.

Technical Summary: Social Structure Matters in 3D Human-Human Interaction Generation

Problem Definition

While text-to-motion generation has achieved significant success in synthesizing realistic single-person 3D motions from natural language, extending this capability to text-driven 3D human-human interaction (HHI) remains challenging. Existing approaches often treat HHI as a direct spatial extension of solo motion generation, jointly modeling two actors or composing individual motion priors. However, this overlooks a fundamental property of HHI: interactions are not merely the sum of two plausible individual motions but are organized by an underlying social structure.

This social structure governs two critical dimensions:

Phase Progression: The temporal organization of an interaction into stages such as approach, contact, release, or in-place coordination.
Partner-Aware Coordination: The asymmetric yet coupled responsibilities of actors within each phase (e.g., initiator vs. receiver, attacker vs. defender).

Without explicitly modeling this structure, generated motions often lack phase consistency (e.g., actors approaching at inconsistent times) or partner awareness (e.g., facing the wrong direction or executing incompatible roles), resulting in interactions that fail to appear coherent despite individual poses being plausible.

Methodology

The paper proposes a planner-executor paradigm titled "Think with LLM, Move with Motion Skill," which separates social structure planning from motion execution. This approach is grounded in the observation that Large Language Models (LLMs) possess strong semantic reasoning capabilities but lack the kinematic precision required for continuous 3D motion generation.

1. Social Structure Planning (The Planner)

The authors utilize an LLM (specifically Qwen3.5) to convert implicit global interaction descriptions into explicit, motion-aligned social supervision.

Motion-Aligned Annotation: To prevent the LLM from generating linguistically plausible but motion-inconsistent plans, the system extracts "motion facts" from real HHI datasets (InterHuman and InterX). These facts include inter-person distance evolution, motion initiators, contact semantics, and limb-level cues.
Decomposition: The LLM is conditioned on the global text, detected phase labels (approach, contact, release, in-place), and these motion facts. It then outputs a structured plan consisting of phase-level units and partner-aware role assignments (e.g., specific actions for Person 1 and Person 2 in each phase).
Result: This process transforms coarse text-motion pairs into fine-grained, motion-aligned social annotations ( $S$ ), serving as a bridge between language intent and motion execution.

2. Solo-to-Social (S2S) Motion Execution (The Executor)

To ground the planned social structure into continuous, coordinated 3D motion, the authors introduce the Solo-to-Social (S2S) framework. Instead of training a new model from scratch, S2S adapts a pretrained solo motion backbone (HY-Motion 1.0) using parameter-efficient Low-Rank Adaptation (LoRA).

Phase-wise Self-Conditioning: To ensure smooth transitions between phases, the model conditions the generation of the current phase on the last $e$ frames of the previous phase. This grounds the temporal progression into a continuous trajectory.
Ego-Relative Partner Conditioning: To achieve partner-aware coordination, the model conditions the generation of Actor $i$ on the partner's (Actor $j$ ) motion, transformed into Actor $i$ 's ego-centric coordinate frame. This allows the model to directly observe relative position, orientation, and interaction geometry.
Training Objective: The fine-tuning objective combines the original flow-matching loss with losses for temporal smoothness, relative distance, and relative orientation, ensuring the model preserves atomic motion priors while learning social coordination.

Key Contributions

Problem Formulation: The paper formulates text-driven 3D HHI generation as a social structure modeling and grounding problem, identifying phase progression and partner-aware coordination as the core organizing principles rather than treating it as a simple two-person extension of solo generation.
Capability Boundary Analysis: Through a diagnostic study, the authors demonstrate that LLMs are effective at social structure planning (recovering phase decompositions and roles) but inadequate for direct continuous 3D motion execution due to a lack of kinematic precision and dynamic consistency.
Planner-Executor Paradigm: A novel framework where an LLM planner reorganizes HHI datasets into fine-grained, motion-aligned annotations, and a Solo-to-Social executor adapts a pretrained solo model to ground these structures into coordinated motion.
S2S Framework: The introduction of phase-wise self-conditioning and ego-relative partner conditioning, combined with LoRA, to effectively transfer solo motion priors to social interaction tasks.

Experimental Results

The proposed method was evaluated on standard HHI benchmarks (InterHuman and InterX) against five existing baselines (ComMDM, in2IN, InterGen, InterMask, TIMotion) and a variant using only the social planner with the raw solo model.

Quantitative Performance: The proposed method achieved the highest R-Precision (24.67% Top-1) and the lowest Fréchet Inception Distance (FID) (0.65), indicating superior text-motion alignment and motion realism. It also achieved the best scores in user studies for Phase Consistency (2.57 ranking) and Partner Coordination (2.78 ranking).
Qualitative Improvements: Visual comparisons show that the method produces clearer phase transitions (e.g., distinct approach, contact, and release stages) and more coherent partner-aware behaviors (e.g., proper blocking, handshaking, and lifting coordination) compared to baselines, which often suffer from premature contact, static interactions, or role mismatches.
Ablation Studies: Removing social structure planning caused a significant drop in R-Precision. Removing motion facts degraded motion consistency. Removing self-conditioning hurt phase smoothness, while removing partner conditioning severely degraded inter-person spatial coordination (Relative Distance and Orientation Errors).

Significance and Claims

The paper claims that treating social structure as the central abstraction for HHI generation is crucial for achieving coherent interactions. By decoupling the semantic reasoning of LLMs from the kinematic execution of specialized motion models, the framework successfully bridges the gap between high-level interaction organization and low-level motion realization.

The authors state that their work validates the importance of both social structure planning (to organize the interaction) and motion-level grounding (to ensure physical plausibility). They position this approach as a step toward more controllable and semantically grounded generation for interactive human motion, with potential applications in animation, virtual avatars, and embodied AI, while acknowledging the need for future work to extend these concepts to multi-person scenarios and more complex social dynamics.