SceneAdapt: Scene-aware Adaptation of Human Motion Diffusion

1. Problem Statement

The core challenge addressed by this work is the generation of human motion that satisfies two conflicting requirements simultaneously:

1. Semantic Diversity: The motion must be diverse and faithful to complex text prompts (e.g., "dribbling a basketball," "walking in a circle").
2. Scene Awareness: The motion must be physically consistent with the surrounding 3D environment (e.g., avoiding collisions with walls, furniture).

Current Limitations:

• Text-to-Motion (T2M) Models: Trained on large-scale text-motion datasets (e.g., HumanML3D), these models generate diverse motions but are "blind" to spatial constraints, often causing characters to walk through walls.
• Scene-Aware Models: Trained on scene-motion datasets (e.g., TRUMANS, HUMANISE), these models respect geometry but suffer from limited semantic diversity. They typically only learn simple actions (walking, sitting) because collecting large-scale, semantically rich text-scene-motion triplets is prohibitively expensive and difficult.
• The Gap: There is no existing large-scale dataset containing rich text, diverse motion, and precise 3D scene interactions. Existing methods either sacrifice semantic richness for scene safety or vice versa.

2. Methodology: SceneAdapt

The authors propose SceneAdapt, a two-stage adaptation framework that injects scene-awareness into a pre-trained Text-to-Motion Diffusion Model (MDM) without requiring large-scale text-scene-motion triplets. The framework leverages two disjoint resources: a text-motion dataset and a scene-motion dataset.

The core insight is using Motion Inbetweening (generating motion between keyframes) as a "bridge" task. Inbetweening does not require text, allowing the model to learn from scene-motion data while preserving the semantic manifold learned from text-motion data.

Stage 0: Pre-training

Starts with a pre-trained Text-to-Motion Diffusion Model (MDM) trained on a large text-motion dataset (HumanML3D).

Stage 1: Adaptation for Motion Inbetweening

• Goal: Adapt the MDM to generate motion between given keyframes without text.
• Mechanism: Introduces the Context-aware Keyframing (CaKey) layer.
  • Sparse Modulation: Unlike standard modulation that affects all frames, CaKey applies affine modulation ($\gamma \cdot a + \beta$) only to the latent representations of keyframe indices. Non-keyframe latents remain unmodulated to preserve the original generation capabilities.
  • Context-Awareness: The modulation parameters ($\gamma, \beta$) are predicted by MLPs that take the diffusion timestep, the latent representation, and the self-attention activation as inputs. This allows the modulation to be aware of the current generation context.
• Training: The base MDM is frozen; only CaKey layers are trained using a motion inbetweening objective (null text embedding + keyframe mask).

Stage 2: Adaptation for Scene-Aware Inbetweening

• Goal: Inject geometric scene constraints into the inbetweening capability.
• Mechanism: Introduces Scene-conditioning (SceneCo) layers.
  • Scene Representation: The 3D scene is voxelized and processed by a Voxel ViT to extract patch-wise embeddings (local context) rather than a single global vector.
  • Cross-Attention: SceneCo layers use cross-attention where motion latents query the voxel patch embeddings. This allows each frame to attend to relevant local parts of the scene (e.g., a foot attending to the floor nearby).
  • Prior Preservation: To prevent the new scene layers from degrading the original text-to-motion capabilities, the training uses a prior loss. This involves training on the original text-motion pairs (with a learnable null scene embedding) alongside the scene-motion pairs.
• Training: CaKey layers are frozen; SceneCo layers and the Voxel ViT are trained.

Inference

For generation, the model takes a text prompt and a 3D scene.

• Keyframes: An all-zero mask is used (no keyframes provided).
• Guidance: Two Classifier-Free Guidance (CFG) scales are used: $w_t$ for text and $w_s$ for scene. This allows balancing semantic alignment vs. collision avoidance.
• Goal Pose: The system can optionally accept a goal pose (treated as a sparse keyframe) to enable functional interactions (e.g., sitting on a specific chair).

3. Key Contributions

1. SceneAdapt Framework: A novel two-stage adaptation pipeline that successfully injects scene-awareness into pre-trained T2M models using only disjoint text-motion and scene-motion datasets, bypassing the need for costly triplet data.
2. Context-aware Keyframing (CaKey): A specialized layer that modulates only keyframe latents based on context, enabling accurate inbetweening without distorting the pre-trained latent manifold.
3. Scene-conditioning (SceneCo): A cross-attention mechanism that utilizes patch-wise scene features, allowing frame-wise motion latents to dynamically interact with local geometric contexts.
4. State-of-the-Art Performance: The method achieves a superior trade-off between semantic diversity and physical consistency compared to optimization-based and feed-forward baselines.

4. Experimental Results

The method was evaluated on a constructed test set combining HumanML3D semantics with TRUMANS scene geometry.

• Quantitative Metrics:

  • Scene Awareness: Significantly outperforms baselines in Collision-Frame Ratio (CFR), Mean Max Penetration (MMP), and Joint-Collision Ratio (JCR). For instance, at $w_s=0.5$, it achieves a CFR of 0.220 compared to 0.429 for AffordMotion.
  • Semantic Quality: Maintains high R-Precision (text alignment) and low FID (motion naturalness). At $w_s=0.5$, R-Precision is 0.750 (close to the base MDM's 0.798), whereas other scene-aware methods often drop significantly.
  • Efficiency: Inference time is 1.69s per sample, orders of magnitude faster than optimization-based methods (DNO: 332s, DARTControl: 362s).

• Qualitative Results:

  • Visualizations show that SceneAdapt generates motions that avoid obstacles (unlike base MDM) while following complex text prompts (unlike HUMANISE-trained models).
  • It successfully handles tasks like "walking in a circle" or "dribbling" without penetrating walls.

• Ablation Studies:

  • CaKey Design: Removing sparse modulation or context-awareness leads to significant performance drops in inbetweening.
  • Two-Stage Necessity: Skipping Stage 1 (inbetweening) and going directly to scene adaptation causes the model to lose its original text-to-motion capabilities.
  • Scene Representation: Voxel-based patch embeddings outperform global class embeddings, point clouds, and meshes, proving the importance of local spatial context.

5. Significance and Impact

• Bridging the Data Gap: SceneAdapt solves the critical bottleneck of lacking large-scale text-scene-motion datasets by creatively leveraging existing disjoint datasets through the inbetweening proxy task.
• Practical Applicability: By being a feed-forward approach (no iterative optimization), it is fast enough for real-time applications in gaming, VR, and robotics.
• Generalizability: The framework demonstrates that geometric constraints can be learned and injected into generative models without retraining the entire model from scratch, preserving the rich semantic knowledge of large pre-trained models.
• Future Directions: The authors note limitations in handling dynamic object states (e.g., opening doors) and multi-human interactions, pointing toward future work in physics-based control and complex human-object interactions.