GMT: Goal-Conditioned Multimodal Transformer for 6-DOF Object Trajectory Synthesis in 3D Scenes

1. Problem Definition

The paper addresses the challenge of synthesizing controllable, realistic 6-DOF (six degrees of freedom) object manipulation trajectories within complex 3D environments.

• Core Challenge: Existing methods often rely on 2D representations or partial 3D data, failing to capture full scene geometry. Furthermore, most learning-based approaches focus on Human-Object Interaction (HOI), treating objects as passive entities whose motion is inferred from human hand movements. This limits generalization to different robot morphologies and lacks explicit control over the object's final state.
• Specific Difficulties:
  • Perception: Depth sensors suffer from noise, occlusion, and sparsity, making accurate 3D object localization difficult.
  • Constraints: Generated trajectories must be physically plausible (avoiding collisions, maintaining stability) and respect spatial constraints (e.g., not passing through tables).
  • Goal Conditioning: The system must integrate geometry, semantics, context, and a specific target pose to generate a trajectory that reaches a desired goal.

2. Methodology: The GMT Framework

The authors propose GMT, a multimodal transformer framework that treats the object as the primary dynamic entity. The model predicts a continuous sequence of 6-DOF poses (position + orientation) conditioned on observed history, scene context, and a target goal.

A. Problem Formulation

The task is formulated as conditional sequence modeling: $P(\hat{X}_{H+1:T} | X_{1:H}, G, S)$.

• Input:
  • $X_{1:H}$: Observed trajectory history (3D position + 6D rotation).
  • $S$: Scene context (Point cloud $P$ and semantic fixture bounding boxes $B$).
  • $G$: Target goal state (desired final position and orientation).
• Output: Future trajectory $\hat{X}_{H+1:T}$.

B. Multimodal Scene Encoding

The model constructs dedicated representations for four distinct modalities:

1. Trajectory Features: Temporal motion context coupled with semantic category features to prevent overfitting to geometry alone.
2. Spatial Features (Geometric):
   • Global: Encoded via PointNet++ from the raw point cloud.
   • Local: A novel feature propagation mechanism interpolates local features from the $k$-nearest neighbors of the scene point cloud to the object's bounding box corners at each timestep. This provides the trajectory with awareness of immediate surroundings (e.g., floor vs. tabletop).
   • Fixtures: A self-attention module processes semantic fixture bounding boxes to model inter-object constraints (e.g., a chair cannot penetrate a table).
3. Semantic Features: Uses a frozen CLIP encoder to embed object labels and natural language action descriptions, allowing the model to understand affordances (e.g., a bed vs. a desk).
4. Goal Features: The target state is encoded via a linear layer. This acts as a controllable "knob" to disambiguate between multiple plausible futures (e.g., placing an object on a table vs. the floor).

C. Multimodal Feature Fusion

Instead of naive concatenation, GMT employs a Multimodal Transformer inspired by Perceiver IO:

• Latent Array: A learnable latent array acts as an information bottleneck, fusing heterogeneous modalities (geometry, semantics, goals) via cross-attention and self-attention.
• Hierarchical Fusion: The architecture prioritizes hard geometric constraints over soft semantic cues to ensure physical validity.
• Direct Prediction: Unlike many diffusion models, the fused latent representation is fed directly into a prediction head without a separate decoding stage, which the authors found more stable for long-horizon control.

D. Training Objective

The model is trained using a composite loss function:

• Translation & Orientation Loss ($L_{trans}, L_{ori}$): L1 loss on future trajectory predictions.
• Reconstruction Loss ($L_{rec}$): Ensures the model accurately reconstructs the observed history.
• Destination Loss ($L_{dest}$): Explicitly constrains the final predicted pose to match the ground truth goal, ensuring goal alignment.

3. Key Contributions

1. Object-Centric Perspective: Shifts focus from human-centric HOI to object-centric trajectory generation, enabling cross-embodiment transfer via Inverse Kinematics (IK) without retraining for specific robot morphologies.
2. Multimodal Fusion Strategy: Introduces a tailored conditioning strategy that fuses:
   • Geometric conditioning via point-cloud-to-box feature propagation.
   • Semantic conditioning via hierarchical CLIP embeddings.
   • Contextual and goal-oriented conditioning.
3. State-of-the-Art Performance: Demonstrates superior performance over strong baselines (CHOIS, GIMO) in both spatial accuracy and orientation control.

4. Experimental Results

The method was evaluated on two datasets: Aria Digital Twin (ADT) (synthetic, noise-free) and HD-EPIC (real-world, noisy, sparse annotations).

• Metrics: Average Displacement Error (ADE), Final Displacement Error (FDE), Fréchet Distance (FD), Angular Consistency (AC), and Collision Rate (CR).
• Performance on ADT: GMT achieved an ADE of 0.366m and FDE of 0.072m, significantly outperforming GIMO (ADE 0.982) and CHOIS (ADE 0.853). It achieved the lowest Fréchet distance (0.438) and highest angular consistency (0.402), indicating superior trajectory smoothness and alignment.
• Performance on HD-EPIC: Despite the challenges of real-world noise and occlusion, GMT maintained the best performance across all metrics (e.g., FDE 0.034m vs. 0.589m for CHOIS).
• Ablation Studies:
  • Removing goal conditioning caused a massive drop in performance, confirming its necessity for long-horizon planning.
  • Removing geometric features degraded spatial accuracy significantly.
  • Using only the first frame as input resulted in high collision rates (83.2%), proving the need for dynamic context.

5. Significance and Impact

• Decoupling Perception and Control: By generating 6-DOF object trajectories directly, the framework decouples perception from control. These trajectories serve as a universal intermediate representation that can be instantiated by any robotic arm via IK, facilitating generalization across different robots.
• Physical Plausibility: The hierarchical fusion strategy ensures that generated trajectories respect spatial constraints, reducing collision rates compared to baselines that often fail to account for scene geometry.
• Goal-Directed Planning: The explicit goal conditioning allows for controllable synthesis, enabling robots to plan specific manipulation tasks (e.g., "move object to X") rather than just mimicking human motion patterns.
• Generalization: The model demonstrates strong generalization from synthetic environments to real-world, cluttered scenes, establishing a new benchmark for learning-based manipulation planning.

Conclusion: GMT represents a significant step forward in robotic manipulation by moving away from human-centric imitation toward a flexible, goal-conditioned, and physically grounded object trajectory synthesis framework.