Technical Summary: Articulation in Prime

Problem Statement

The paper addresses the fundamental challenge of retrieving the 3D kinematics of articulated objects from a single, casually captured monocular video. Articulated objects consist of rigid parts connected by joints (revolute or prismatic) that constrain their motion. Existing approaches face significant limitations:

• Complex Acquisition: Many methods require multi-view setups, 3D scans paired with interaction videos, or specific acquisition protocols impractical for real-world use.
• Fragility: Methods relying on long-term point tracking or wide-baseline matching often fail under severe occlusions, rapid camera ego-motion, or when local features are weak.
• Generalization: Learning-based methods struggle to generalize beyond their specific training categories, even when trained on synthetic data with ground truth.

The goal is to develop a category-agnostic framework that segments an object into articulated parts and recovers joint parameters (axis, type, and motion) from a single video sequence containing partial observations and occlusions.

Methodology

The authors propose Articulation in Prime, an unsupervised optimization framework that treats articulated object understanding as a primitive-fitting problem. Instead of relying on unstable point tracks or category-specific priors, the method uses geometric primitives as a proxy representation.

Core Components

1. Input and Preprocessing:

   • The input is a sequence of depth maps and camera poses extracted from a video (using tools like ViPE for depth/pose and SAM2 for object masks).
   • Depth maps are backprojected into a sequence of partial 3D point clouds in world coordinates.

2. Primitive Representation:

   • The object is represented by a set of convex superquadrics (geometric primitives). Superquadrics are chosen for their balance between expressiveness and compactness.
   • Each primitive $Q_q$ is parameterized by 3D rotation, translation, scale, and shape parameters ($\epsilon$).

3. Part Segmentation and Joint Modeling:

   • Soft Assignment: Primitives are softly assigned to parts ($P_p$) via a differentiable allocation mechanism. At convergence, each primitive belongs to exactly one part.
   • Joint Constraints: Each part is assigned a joint type (revolute or prismatic). The method optimizes for more parts than expected; superfluous parts are automatically discarded if they receive no primitives.
   • Kinematics: Joint parameters are modeled using twist vectors from screw theory.
     • Revolute joints are defined by an axis vector $\omega$ and a point $q$.
     • Prismatic joints are defined by a direction vector $v$.
     • Motion is quantified by a scalar amount $\theta_{t,p}$ per timestep.

4. Visibility-Aware Optimization:

   • A critical component is handling partial observations. The method computes visibility probabilities ($\gamma_{t,y,q}$) for points on primitives, inspired by NeRF/3DGS, to account for occlusions by other primitives.
   • This allows the system to distinguish between points that are truly missing and points that are occluded.

5. Objective Function:
   The framework jointly optimizes part segmentation, primitive parameters, and joint kinematics by minimizing a global loss function $L = L_{obs} + L_{reg}$:

   • Observation Loss ($L_{obs}$):
     • Reconstruction ($L_{Q \to X}^{rec}, L_{X \to Q}^{rec}$): Ensures primitives fit the observed point clouds and vice-versa. Crucially, it only considers visible points on primitives to avoid bias from partial data.
     • Flow Loss ($L_{flow}$): Ensures the predicted motion of parts aligns with the scene flow estimated between consecutive frames.
   • Regularization Loss ($L_{reg}$):
     • Sparsity: Encourages a minimal number of parts and primitives.
     • Existence: Uses binary cross-entropy to determine if a part or primitive "exists" (is active).
     • Motion: Penalizes unrealistic velocities.
     • Matching: Encourages primitives to be assigned to a single part (one-hot assignment).

Key Contributions

1. Primitive-Based Optimization Framework: A category-agnostic approach that formulates articulated object understanding as a primitive-fitting problem, avoiding the brittleness of point-tracking methods and the generalization limits of learning-based approaches.
2. Visibility-Aware Formulation: A novel mechanism that explicitly handles partial observations and occlusions inherent in single-view videos by computing visibility probabilities for primitive points.
3. Joint Discovery and Segmentation: The method simultaneously recovers part segmentation, joint types (revolute/prismatic), joint axes, and motion magnitudes without requiring category-specific training or external mesh databases.
4. New Benchmarks: The introduction of AiP-synth and AiP-real, two datasets featuring significant camera motion, heavy occlusions, and diverse object categories (e.g., globes, excavators, cameras) to better evaluate robustness in unconstrained settings.

Results

The method was evaluated on four benchmarks: Video2Articulation-S, Arti4D, and the newly introduced AiP-synth and AiP-real.

• Quantitative Performance: The proposed method significantly outperforms existing baselines (Video2Articulation, ReArt, Articulate-Anything, Artipoint) across all metrics, including mean Intersection-over-Union (mIoU) for segmentation, axis/position error for joints, and motion state accuracy.
  • On AiP-synth, the method achieved near-perfect accuracy (e.g., 0.52° axis error for revolute joints vs. 51.20° for Video2Articulation) and 100% joint type accuracy.
  • On AiP-real, it maintained robustness (0.86 mean mIoU) where other methods failed or produced high errors, particularly on objects with heavy occlusions like globes and excavators.
• Robustness: The method demonstrated resilience to low-quality depth maps (Arti4D dataset) and complex camera trajectories (circular and ping-pong paths), whereas baselines relying on dense correspondences or static assumptions degraded significantly.
• Ablation Studies: Removing reconstruction losses, flow loss, or regularization terms resulted in significant performance degradation, confirming the necessity of the joint optimization of geometry, motion, and sparsity.

Significance and Claims

The paper claims to be the first to frame articulated object reconstruction and joint parameter estimation strictly as a primitive-fitting task driven by optimization rather than learning or dense correspondence tracking.

• Generalization: The approach demonstrates strong generalization across diverse object categories without category-specific training, relying instead on the geometric and dynamic coherence of the primitives.
• Casual Input: It proves that casual monocular video provides sufficient geometric regularity to recover articulated motion, eliminating the need for multi-view setups, 3D scans, or interaction videos.
• Limitations: The authors modestly acknowledge that while superquadrics provide a coarse approximation sufficient for segmentation and kinematics, they are insufficient for generating detailed mesh reconstructions required for high-fidelity applications like digital twins. Future work could explore more expressive primitive representations.

The work establishes that geometric priors (superquadrics) combined with visibility-aware optimization can effectively solve the articulated object understanding problem in unconstrained, real-world video scenarios.