Technical Summary: GIRAF

Problem Statement

Synthesizing realistic, full-body human interactions with articulated objects is a critical challenge for embodied AI and computer graphics, with applications in robotics training and virtual agents. Existing approaches face significant limitations:

• Narrow Scope: Current models typically focus either on simple full-body activities with static objects (e.g., sitting) or dexterous hand-only manipulation (e.g., grasping a cup), failing to unify these domains.
• Missing Transitions: Most methods assume the human is already in contact with or near the object, neglecting the seamless transition from locomotion (approaching) to interaction (manipulating).
• Generalization Issues: Models struggle to generalize to unseen object configurations, shapes, and positions because they often rely on object-specific contact definitions or lack robust training data.
• Data Scarcity: Acquiring large-scale, paired motion-scene datasets for diverse articulated objects is costly and difficult, creating a bottleneck for training generalizable models.

The core difficulty lies in jointly reasoning about locomotion, fine-grained hand-object contact, and the articulation of the object itself, while ensuring physical plausibility and consistency across diverse geometries.

Methodology

The authors propose GIRAF, a text-conditioned diffusion model designed to synthesize long-horizon, full-body motion sequences that capture the transition from locomotion to interaction with articulated objects. The framework relies on four core components:

1. Dynamic Basis Point Sets (BPS) for Object-Centric Representation

To address the challenge of generalizing contact across different object geometries, the authors introduce a unified, object-centric representation.

• Concept: Instead of encoding contact on fixed object surfaces (which fails on new shapes) or solely on hand keypoints (which lacks object correspondence), they define a fixed set of points ($P$) sampled within a unit sphere centered at the object origin.
• Integration: This representation unifies three features in a shared space:
  1. Distances from the basis points to the object surface ($od$).
  2. Distances from the basis points to hand end-effectors ($\tilde{J}_{hand}$).
  3. Binary contact labels indicating if a basis point's corresponding surface point is in contact with a hand joint.
• Benefit: This "voting scheme" over shared basis points is shape-agnostic, encouraging generalization to unseen object configurations while maintaining precise hand-object correspondence.

2. Mixed-Domain Training Strategy

To enable seamless transitions between locomotion and interaction, the model employs a specialized training regimen:

• FiLM Conditioning: Trainable embeddings via Feature-wise Linear Modulation (FiLM) layers are integrated into transformer blocks. These preserve interaction-specific features while allowing the model to naturally represent locomotion.
• Masking and Annealing: A locomotion mask ($M_{loco}$) identifies frames where the human is >0.5m from the object. The training schedule uses an annealing approach: early batches are balanced between locomotion and interaction, while later batches become interaction-heavy. This ensures the model learns smooth transitions without sacrificing locomotion fidelity.
• Homogeneous Batches: The authors note that training with homogeneous batches (rather than mixed within a single batch) is imperative for performance.

3. Contact-Based Data Augmentation

To mitigate data scarcity and improve generalization:

• Relocation and Rescaling: Objects are relocated on a discretized 3D grid and randomly rescaled.
• Contact Preservation: Hand-object contact points from the original sequence are mapped to corresponding points on the transformed object to preserve the interaction relationship.
• IK Solver: Human motion is recomputed using a CCD-based inverse kinematics solver with rotational limits to ensure natural articulation and physical plausibility during augmentation.

4. Scene-Aware Diffusion Noise Optimization

Post-training, a refinement stage improves physical plausibility:

• Process: Using DDIM sampling, the diffusion noise is optimized via gradient descent.
• Loss Functions:
  • Contact Loss: Aligns object motion with predicted hand contacts.
  • Collision Loss: Reduces hand-object penetration using a Signed Distance Field (SDF).
  • Smoothness Loss: Suppresses jitter in global joint positions over time.

Key Contributions

1. Unified Framework: Introduction of a text-conditioned diffusion model capable of generating unified full-body locomotion and interaction with articulated objects.
2. Object-Centric Contact Representation: A novel Dynamic BPS approach that enables fine-grained, generalizable hand-object interactions by unifying contact, hand, and object surfaces in a shared space.
3. Robust Training Strategy: A combination of mixed-domain learning, FiLM-based conditioning, and contact-driven data augmentation that supports long-horizon motion synthesis in complex 3D environments.

Experimental Results

The method was evaluated on the ParaHome dataset (augmented with Babel for locomotion), covering four interaction categories: drawers, microwaves, refrigerators, and washing machines.

• Quantitative Performance: GIRAF outperformed strong baselines (LINGO and CHOIS) across all metrics:
  • Contact & Penetration: Achieved the lowest contact distance (1.869 cm) and penetration error (1.044 cm), with improved precision and recall.
  • Pose Accuracy: Reduced Mean Per-Joint Position Error (MPJPE) to 0.1143 and hand error to 0.1328.
  • Text-to-Motion Alignment: Achieved the best R-precision (0.3812) and lowest Fréchet Inception Distance (FID) (0.0634), demonstrating strong semantic alignment and diversity.
• Qualitative & Generalization:
  • The model successfully synthesized realistic interactions for unseen object configurations (e.g., different closet door types, varying drawer heights) without retraining.
  • It demonstrated controllability, allowing users to specify contact strategies (e.g., left hand, right hand, or both) via text prompts while maintaining physical plausibility.
  • Visual comparisons showed that GIRAF resolves fine-grained hand-object contacts and minimizes penetration, whereas baselines often resulted in floating hands or deep interpenetration.

Significance and Limitations

The paper claims that GIRAF represents a significant step forward in generating physically plausible, long-horizon human-object interactions by unifying locomotion and manipulation within a single model. Its ability to generalize to unseen object placements and shapes addresses a critical gap in current embodied AI research.

However, the authors maintain modesty regarding the current scope:

• Limitations: The model is limited to the articulated mechanisms observed during training (e.g., it struggles with novel mechanisms like rotational joints not seen in ParaHome).
• Artifacts: Occasional motion artifacts, such as brief foot sliding or joint jitter during transitions from locomotion to interaction, persist, likely due to the difficulty of jointly modeling mixed regimes with limited paired data.
• Scope: The current framework focuses on simple terrains and short-range manipulation, not yet supporting complex navigation scenarios like stair traversal with interaction.

Future work is proposed to extend the representation to broader articulated object categories and integrate more diverse locomotion behaviors.