Technical Summary: Articulate That Object Part (ATOP)

Problem Statement

The paper addresses the challenge of generating plausible and accurate 3D part articulations for static 3D objects based on text prompts. While digital 3D asset repositories (e.g., Objaverse, ShapeNet) have grown exponentially, the vast majority of these models exist only in their rest (unarticulated) states. Existing datasets with motion annotations, such as PartNet-Mobility and Shape2Motion, are limited in scale (few thousand models) and rely on expensive human annotation, making them insufficient for training generalizable models.

Current approaches face three primary hurdles:

1. Lack of Articulation Awareness: State-of-the-art (SOTA) video diffusion models (e.g., WAN2.1) generate high-quality visual content but fail to respect the piecewise rigidity of everyday objects, often producing physically impossible deformations.
2. Coarse Control: Text-only prompts are too ambiguous to control the precise movement of specific object parts. Alternative "drag" interaction methods suffer from ambiguity in identifying which part to animate, leading to incorrect motions (e.g., lifting a chair seat instead of folding it).
3. 2D-to-3D Transfer Difficulty: Lifting single-view motion to multi-view consistency is challenging. Existing pipelines that combine Image-to-Video (I2V) and Video-to-Multi-View (V2MV) generators often result in structural distortions and geometric inconsistencies, preventing accurate 3D articulation estimation.

The authors propose to solve this in a few-shot setting, aiming to generate accurate 3D motion parameters for unseen objects using only a small number of reference motion samples, without relying on large-scale annotated 3D motion datasets.

Methodology

The proposed framework, ATOP (Articulate That Object Part), operates in two main stages:

1. Few-Shot Finetuning for Personalized Multi-View Motion Generation

The first stage involves adapting a pre-trained multi-view image generator (ImageDream) to become a controllable, part-aware, multi-view motion diffusion model.

• Model Architecture: The authors employ an "inflated" U-Net architecture, extending 2D convolutional blocks to 3D (spatio-temporal) to handle video data.
• Key Modules:
  • Correspondence-Aware Spatial Attention: To ensure geometric consistency across multiple views, the self-attention mechanism is modified to aggregate information across views at each frame index, preserving the 3D structure of the target object.
  • Cross-Attention for Appearance Injection: Multi-view images of the object's rest state are encoded via a CLIP image encoder and fused with the latent space. This ensures the generated motion preserves the specific appearance and structure of the target object.
  • Controllable Part-Motion Module: To achieve spatial control, the model accepts a binary mask of the part to be articulated. An adaptive affine transformation modulates the latent features based on this mask, allowing the model to learn temporal motion patterns specific to the masked region.
• Training: The model is finetuned on a small collection of reference multi-view motion videos (few-shot). The diffusion model is frozen, and only the newly added controllable motion modules are trained.
• Inference: Given a target static mesh, the system renders multi-view images and part masks. These, along with a text prompt, are fed into the finetuned model to hallucinate personalized, multi-view motion videos where the specific part moves while the rest of the object remains rigid.

2. 3D Motion Axis Optimization

The second stage transfers the generated 2D multi-view motion back into 3D space to estimate the articulation parameters (motion axis and origin).

• Differentiable Rendering: The input mesh is parameterized as a set of 3D Gaussians. A differentiable renderer projects these Gaussians into the latent space of the diffusion model.
• Score Distillation Sampling (SDS): The system optimizes the 3D representation by minimizing the difference between the rendered latent and the noise predicted by the personalized diffusion model (guided by the text prompt and masks). This aligns the 3D geometry with the plausible 2D motion priors.
• Algorithmic Refinement: To handle noise in the reconstructed sparse point clouds, the authors propose an algorithmic approach for motion axis estimation. Instead of continuous regression, they approximate the moving part with an Oriented Bounding Box (OBB) and search a discrete set of possible axis origins and directions (derived from the OBB's principal components). The best fit is selected by minimizing the Chamfer distance against the articulated point cloud.

Key Contributions

1. First Annotation-Free Framework: ATOP is presented as the first learning framework capable of generating plausible and accurate part articulations on static 3D objects from text prompts without requiring human-annotated 3D part motions.
2. Motion Personalization via Few-Shot Finetuning: The paper introduces a method to inject articulation awareness into diffusion models using a few reference samples. This approach personalizes motion generation to specific object shapes while maintaining multi-view consistency.
3. Part-Level Spatial Control: By utilizing part masks as input conditions, the method overcomes the ambiguity of text-only or drag-based controls, enabling precise animation of specific object parts.
4. Robust 2D-to-3D Transfer: The combination of personalized multi-view generation and SDS-based optimization allows for the faithful transfer of motion from 2D video to 3D parameter estimation, outperforming two-stage pipelines (I2V + V2MV).

Results

The authors evaluate ATOP on the PartNet-Mobility and ACD datasets, as well as qualitatively on Objaverse.

• Quantitative Performance: On PartNet-Mobility, ATOP significantly outperforms baselines like Shape2Motion (S2M) and OPD in terms of Mean Angular Error (MAE) and Mean Position Error (MPE). For instance, the mean MAE drops from ~7.04 (S2M) and ~12.15 (OPD) to 2.63 for ATOP.
• Video Generation Quality: In multi-view video generation, ATOP achieves superior scores in FVD (938.97 vs. >1300 for baselines), PSNR (20.87 vs. ~11-12), and LPIPS (0.128 vs. ~0.49), indicating better temporal coherence, text alignment, and spatial consistency.
• Generalization: The method demonstrates strong zero-shot generalization on the ACD dataset, which contains structurally distinct objects not seen during training. It successfully estimates motion parameters for objects like armoires and nightstands despite significant structural differences from training samples.
• Ablation Studies: Experiments confirm that the algorithmic optimization (discrete search) yields more accurate results than direct continuous optimization. Additionally, the method is robust to noisy masks when trained with data augmentation.

Significance and Claims

The paper claims that ATOP represents a significant step toward animating the vast existing library of static 3D assets. By leveraging the generative priors of diffusion models and personalizing them through few-shot learning, the method bypasses the need for large-scale, expensive 3D motion datasets.

The authors emphasize that their approach is distinct from works that generate new articulated assets (e.g., CAGE, NAP); instead, ATOP enhances existing static meshes by inferring dynamic attributes. While the current method is limited to revolute and prismatic joints and does not model complex hierarchical kinematic chains, it establishes a foundation for scalable, text-driven 3D animation. The authors note that the "missing interior" problem in many 3D models remains a challenge, but the method successfully handles the external articulation of diverse shapes.