CoInteract: Physically-Consistent Human-Object Interaction Video Synthesis via Spatially-Structured Co-Generation

1. Problem Statement

Current video generation models, particularly those based on Diffusion Transformers (DiT), excel at photorealistic rendering but struggle significantly with Human-Object Interaction (HOI) scenarios. Two primary failure modes are identified:

1. Structural Instability: Sensitive regions like hands and faces often suffer from "structural collapse," where fingers merge, blur, or lose anatomical coherence.
2. Physical Implausibility: Models frequently violate physical laws, most notably through hand-object interpenetration (hands passing through objects) or unnatural grasping poses. This occurs because standard RGB-centric models lack explicit 3D spatial understanding of interaction geometry and rely solely on appearance cues.

Existing solutions either require heavy preprocessing (extracting per-frame poses/masks) or lack explicit mechanisms to enforce interaction constraints, leading to poor generalization and unrealistic results.

2. Methodology: CoInteract Framework

CoInteract is an end-to-end framework designed to synthesize HOI videos conditioned on a person reference image, a product reference image, text prompts, and speech audio. It embeds structural priors directly into the Diffusion Transformer (DiT) backbone through two core innovations:

A. Spatially-Structured Co-Generation (Dual-Stream Training)

Instead of generating only RGB video, CoInteract employs a dual-stream training paradigm within a shared DiT backbone:

• RGB Stream: Generates the standard photorealistic video.
• HOI Structure Stream: An auxiliary stream that generates a "silhouette-like" representation. In this stream, the human body is reduced to a mesh projection (silhouette), and the object retains its mask, effectively stripping away RGB texture while preserving interaction geometry.
• Mechanism: Both streams share the same transformer weights but use stream-specific modulation (scale/shift). The HOI stream attends to RGB tokens during training.
• Asymmetric Co-Attention:
  • Training: Uses a two-stage strategy. Stage 1 uses full bidirectional attention for convergence. Stage 2 applies an asymmetric attention mask where RGB tokens attend only to other RGB tokens, while HOI tokens attend to both streams.
  • Inference: The HOI branch is completely removed. Because the HOI stream's supervision (via the asymmetric mask) regularizes the shared backbone during training, the RGB generator inherits the learned physical constraints without any computational overhead at inference time.

B. Human-Aware Mixture-of-Experts (MoE)

To address the high-frequency details required for hands and faces:

• Architecture: A lightweight router dispatches tokens to specialized experts based on spatial location (face and hand bounding boxes).
• Experts: Includes a Shared Expert (reusing the original DiT FFN) and three specialized lightweight experts: Head, Hand, and Base.
• Routing: A spatially supervised router (trained with a cross-entropy loss) ensures tokens in critical regions are processed by experts dedicated to those anatomical structures, improving fidelity with minimal parameter overhead.

C. Data Curation & Representation

The framework utilizes a custom pipeline to create paired training data:

1. Decoupling: Separates person and product references.
2. Geometric Supervision: Uses SAM3 and SAM3D-body to generate object masks and human meshes, projecting them to create the texture-stripped HOI structure stream.
3. 3D RoPE: A 3D Rotary Positional Encoding assigns distinct spatiotemporal coordinates to motion frames, reference images, and the dual streams, ensuring the model understands temporal causality and spatial alignment.

3. Key Contributions

1. End-to-End HOI Framework: CoInteract is the first unified framework to embed human structural priors and interaction geometry constraints directly into the DiT backbone, eliminating the need for external preprocessing or post-processing.
2. Human-Aware MoE: Introduces a spatially supervised routing mechanism that significantly reduces artifacts in hands and faces with negligible parameter cost.
3. Zero-Overhead Co-Generation: Proposes a novel training-inference paradigm where an auxiliary HOI stream enforces physical plausibility during training but is discarded at inference, achieving structural consistency without increasing inference latency.
4. Asymmetric Attention: A novel masking strategy that allows the model to learn cross-stream dependencies while maintaining an independent, efficient RGB generation path.

4. Experimental Results

The authors evaluated CoInteract against state-of-the-art methods (e.g., AnchorCrafter, Phantom, Humo, InteractAvatar) on a curated dataset of 12K HOI clips.

• Quantitative Performance:
  • Interaction Plausibility: Achieved the highest VLM-QA score (0.72), significantly outperforming baselines (next best: 0.62), indicating superior logical consistency in interactions.
  • Hand Quality: Achieved the highest HQ score (0.724), demonstrating better structural stability in hand articulation.
  • Identity Preservation: Led in DINOid (0.671) and FaceSim (0.696), maintaining consistent character and product identity.
  • Temporal Coherence: Achieved the highest Smoothness score (0.9951).
• Qualitative Results: Visual comparisons show CoInteract successfully handles complex interactions (e.g., lifting bags, opening lids) without interpenetration, whereas baselines often exhibit hand-object clipping or unnatural grasping.
• User Study: In a blind study with 24 evaluators, CoInteract ranked first (lowest mean rank) across all criteria: Object Consistency, Human/Background Consistency, and Interaction Plausibility.
• Ablation Studies:
  • Removing the HOI stream caused a 33.3% drop in VLM-QA, proving the necessity of structural supervision.
  • Removing the MoE degraded hand quality (HQ dropped from 0.724 to 0.658).
  • The asymmetric mask strategy was crucial for maintaining high performance while keeping inference costs low (only 1.04x overhead compared to a standard model, versus 4.13x if the HOI branch were kept at inference).

5. Significance

CoInteract represents a significant leap forward in digital human and e-commerce applications. By shifting from purely appearance-based generation to structure-aware generation, it solves the critical "physics gap" in video synthesis. The ability to generate physically plausible, high-fidelity human-object interactions without heavy preprocessing or inference overhead makes it highly practical for real-world deployment in virtual marketing, remote education, and digital advertising. The methodology of using an auxiliary structural stream to regularize a shared backbone offers a promising direction for future research in controllable generative models.