💻 computer science

Stability-Driven Motion Generation for Object-Guided Human-Human Co-Manipulation

This paper proposes a stability-driven flow-matching framework for generating natural and effective human-human co-manipulation motions by integrating object affordance-based strategies, an adversarial interaction prior for realistic poses, and a stability-driven simulation to refine interaction states.

Original authors: Jiahao Xu, Xiaohan Yuan, Xingchen Wu, Chongyang Xu, Kun Li, Buzhen Huang

Published 2026-04-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiahao Xu, Xiaohan Yuan, Xingchen Wu, Chongyang Xu, Kun Li, Buzhen Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you and a friend are trying to move a giant, awkwardly shaped sofa through a narrow hallway. You aren't just walking; you have to coordinate your steps, grip the sofa at the right spots, and adjust your posture so neither of you trips or drops the furniture. If one of you moves too fast or grabs the wrong spot, the whole thing becomes a disaster.

This paper is about teaching computers to be that perfect team of movers. The researchers built a new AI system called StaCOM that can generate realistic, coordinated videos of two people carrying an object together, based on a path the object needs to take.

Here is how they did it, explained with some everyday analogies:

1. The Problem: The "Solo Dancer" vs. The "Duet"

Most existing AI motion generators are like solo dancers. They are great at making one person walk, run, or wave at a static background. Some can even make two people dance together, but they usually ignore the heavy furniture in the middle.

If you try to force a "solo dancer" AI to move a sofa, it might look like the people are floating, slipping through the sofa, or grabbing it in impossible ways. They lack the physics sense of "Oh, if I lift here, my friend needs to lift there to keep it balanced."

2. The Solution: A Three-Step Recipe for Perfect Teamwork

The authors created a system that acts like a choreographer, a physics coach, and a safety inspector all rolled into one.

Step A: The "Smart Grip" (Affordance-Informed Strategy)

The Analogy: Imagine you are handed a weirdly shaped vase. Before you pick it up, your brain instantly scans it and says, "I can't grab the handle; it's too slippery. I should hold the base."
What the AI does: The system looks at the object's shape and figure out the best places to grab it. It doesn't just guess; it calculates "graspability maps." This ensures the virtual humans don't try to grab the object through its middle or in a way that would make it fall. It gives the AI a clear plan: "Grab here, lift there."

Step B: The "Natural Flow" (Flow Matching & Adversarial Prior)

The Analogy: Think of Flow Matching as a river. The AI starts with a chaotic splash of water (random noise) and gently guides it into a smooth, flowing river that matches the path the sofa needs to take.
The Twist: Sometimes, a river can flow too stiffly. To fix this, they added an Adversarial Interaction Prior. Imagine a strict dance instructor watching the two virtual people. If one person looks stiff, or if they aren't looking at each other, or if their timing is off, the instructor yells, "No, that looks robotic! Try again!"
The Result: This forces the AI to learn how humans actually move when cooperating—looking at each other, adjusting their speed, and moving naturally, not just mechanically.

Step C: The "Physics Safety Net" (Stability-Driven Simulation)

The Analogy: This is the most important part. Imagine you are rehearsing a stunt in a video game. You think you did it perfectly, but when you actually try it, you drop the box.
What the AI does: Before finalizing the movement, the system runs a mini-physics simulation (like a video game engine). It pretends to lift the object. If the virtual humans start floating, or if the object wobbles and falls, the system says, "Wait, that's impossible!"
The Fix: It uses a mathematical "tuning knob" (called CMA-ES) to nudge the humans' poses just enough so that the object stays stable and gravity works correctly. It's like a safety net that catches the AI before it makes a physics-breaking mistake.

3. The Final Result

When you put all these parts together, the AI generates a video where:

Two people grab a box in the most logical spots.
They walk in sync, looking natural and reacting to each other.
The box doesn't float, fall, or pass through their hands.

Why This Matters

This isn't just about making cool videos. This technology is a huge step forward for:

Robotics: Teaching real robots how to carry heavy things together without dropping them.
Video Games & VR: Creating NPCs (non-player characters) that can help you carry loot or move furniture in a way that feels real, not glitchy.
Animation: Making movies where crowds move heavy props without needing a team of animators to fix every single frame.

In short, the paper teaches computers that moving together is harder than moving alone, and it solves the problem by combining smart planning, dance lessons, and physics checks.

1. Problem Statement

The paper addresses the challenge of generating coordinated motion for two humans co-manipulating a shared object (e.g., lifting or carrying a heavy item). Existing motion generation methods face three primary limitations in this domain:

Single-Agent Bias: Most methods are designed for single-person scenarios or static environments, lacking mechanisms for inter-agent communication and mutual adaptation.
Lack of Physical Dynamics: Recent multi-person methods often focus on social interactions (e.g., dancing) without object manipulation, failing to account for payload-induced dynamics, gravity, and stability.
Physical Implausibility: Directly adapting existing frameworks often results in unnatural poses, hand-object penetration, floating artifacts, and unstable object states because they ignore physical constraints like contact forces and torque limits.

The goal is to generate motions that satisfy three criteria: Intention (aligning with object affordances and goals), Naturalness (realistic individual poses and synchronized interactions), and Effectiveness (physically stable transport).

2. Methodology

The authors propose a Flow-Matching Framework that integrates object affordances, adversarial priors, and physics-based simulation. The pipeline consists of four key components:

A. Core Framework: Flow Matching

Base Model: The authors use Flow Matching to learn a continuous vector field that maps noise to clean motion data. This allows for deterministic generation and efficient conditioning.
Representation: Humans are represented using SMPL-X parameters (joint rotations, shape, and global translation). The object is represented by its mesh, 6D pose trajectory, and Basis Point Set (BPS) features to capture shape context.
Training: The model minimizes a mean-squared flow objective, supplemented by an L1 loss on SMPL parameters and a foot-contact loss to prevent sliding.

B. Affordance-Informed Manipulation Strategy

Goal: To guide the motion flow toward successful manipulation by predicting where and how to grasp.
Mechanism: A diffusion-based network predicts contact anchors (position, normal, offset) conditioned on the object's geometry and affordance probabilities (graspability scores).
Integration: During inference, these predicted anchors act as explicit gradient cues. The flow matching process is adjusted via a differentiable distance loss to keep human wrists aligned with these high-probability grasp regions, ensuring geometric consistency.

C. Adversarial Interaction Prior

Goal: To enhance the naturalness of individual poses and the synchronization between the two agents.
Mechanism: Two discriminators are trained:
1. Pose Discriminator ( $D_{body}$ ): Evaluates individual joint rotations and shape coefficients to ensure realistic articulation.
2. Interaction Discriminator ( $D_{int}$ ): Evaluates the relationship between the two agents (relative root transformations, timing, symmetry) to ensure coordinated behavior.
Integration: During generation, the gradients from these discriminators are used to refine the flow prediction, penalizing misaligned reactions and encouraging cooperative cues.

D. Stability-Driven Simulation

Goal: To resolve physical artifacts like floating, penetration, and object instability that purely data-driven methods produce.
Mechanism: A sampling-based optimization module is inserted into the flow matching process (specifically at the penultimate step).
- The current motion state is fed into a physics engine (PyBullet) with a PD controller.
- The CMA-ES algorithm samples corrective offsets ( $\Delta x$ ) to the body poses to minimize a cost function comprising similarity loss (keeping motion close to the target) and stability loss (minimizing external forces/torques that would cause the object to fall or drift).
Integration: The optimized, physically stable trajectory is fed back into the final integration step of the flow matcher, ensuring the output is both physically plausible and natural.

3. Key Contributions

Unified Framework: A novel flow-matching framework specifically designed for dual-human co-manipulation, balancing intention, naturalness, and physical effectiveness.
Affordance-Guided Strategy: An explicit manipulation strategy that derives grasp plans from object affordances, providing geometrically consistent contact cues for the motion generator.
Adversarial Coordination: The introduction of an interaction prior that specifically targets collaborative cues (timing, symmetry) rather than just individual pose realism.
Physics-Driven Refinement: A stability-driven simulation module using CMA-ES and PD control to iteratively refine unstable states, significantly reducing penetration and improving grasp stability without requiring end-to-end reinforcement learning policy training.

4. Experimental Results

The method was evaluated on the Core4D dataset (collaborative human-object-human interactions) and Inter-X.

Quantitative Performance:
- Contact Accuracy: Achieved 0.44 (vs. 0.21 for OMOMO and 0.11 for ComMDM), indicating significantly better hand-object alignment.
- Penetration: Reduced to 0.05 (vs. 0.15–0.19 in baselines), demonstrating superior physical plausibility.
- Distributional Fidelity (FID): Achieved 25.5, outperforming baselines and indicating more realistic motion distributions.
- Interaction Fidelity (IDF): Achieved 0.22, showing better spatial relationship modeling.
Qualitative Analysis: Visual comparisons show that baseline methods suffer from slipping contacts, orientation misalignment, and delayed responses. The proposed method maintains stable grasps and synchronized lifting throughout the trajectory.
Ablation Studies:
- Removing the simulation caused a sharp drop in contact accuracy (0.44 $\to$ 0.37) and a spike in penetration (0.05 $\to$ 0.16).
- Removing the adversarial prior degraded the naturalness (FID increased) and coordination.
- The affordance strategy was crucial for improving contact accuracy from 0.35 to 0.40.

5. Significance

This work bridges the gap between data-driven motion generation and physical simulation in multi-agent robotics and computer graphics. By decoupling the generation of natural motion (via flow matching and priors) from the enforcement of physical stability (via sampling-based simulation), the authors achieve a level of realism and robustness previously unattainable in co-manipulation tasks. The approach is significant for applications in virtual reality, human-robot collaboration, and animation, where physically plausible and socially coordinated interactions are critical. The code is open-sourced, facilitating further research in collaborative manipulation.