💻 computer science

PAOLI: Pose-free Articulated Object Learning from Sparse-view Images

The paper presents PAOLI, a novel method that reconstructs articulated objects from sparse-view images with unknown camera poses by first establishing robust correspondences between independently reconstructed parts and then jointly optimizing geometry, appearance, and kinematics through a progressive disentanglement strategy.

Original authors: Jianning Deng, Kartic Subr, Hakan Bilen

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jianning Deng, Kartic Subr, Hakan Bilen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a toy robot or a folding chair, and you want to teach a computer exactly how it moves so a robot arm can pick it up or a video game can animate it. Usually, to do this, you need to take hundreds of photos of the object from every possible angle, with perfect cameras, and you need to know exactly where the camera was for every single shot. It's like trying to solve a puzzle where you have to buy a new box for every single piece.

PAOLI is a new method that says, "Wait a minute. We can do this with just four blurry, random photos and no idea where the camera was."

Here is how it works, using some everyday analogies:

1. The Problem: The "Two Different Worlds"

Imagine you take four photos of a door in the "closed" position, and then four photos of the same door in the "open" position.

The Old Way: Computers try to build a 3D model of the closed door and a 3D model of the open door separately. But because the camera was in a different spot for each set, the computer builds two models that are floating in different universes. One is tilted left, the other is tilted right. They don't line up. It's like trying to glue two maps of the same city together when one map is upside down and the other is rotated 90 degrees.
The PAOLI Solution: Instead of trying to guess the camera position first, PAOLI builds the two 3D models anyway, but then it tries to stretch and warp one model to fit perfectly onto the other, like molding clay.

2. The Secret Sauce: The "Magic Deformation Field"

This is the core of the paper.

The Analogy: Imagine you have a rubber sheet with a drawing of a closed door on it. You want to stretch that rubber sheet so the drawing looks like an open door.
How PAOLI does it: It doesn't just guess. It creates a "force field" (a mathematical deformation field) that gently pulls and pushes every tiny point of the 3D model. It does this while looking at the photos to make sure the colors and shapes still look real.
Why it's special: Old methods tried to match specific dots (like matching the corner of the door to the corner of the door). But if the door is smooth and has no texture (like a white wall), there are no dots to match. PAOLI matches the whole shape at once, like stretching a piece of taffy. This works even on smooth, boring surfaces.

3. The "Detective Work": Separating the Moving Parts

Once the two models are stretched to match each other, the computer needs to figure out: "Which parts moved, and which parts stayed still?"

The Analogy: Imagine a group of people holding hands in a circle. If the whole group spins, everyone moves. But if one person lets go and walks away, they are the "moving part."
How PAOLI does it: It uses a smart algorithm (like a detective) to look at the stretched model. It says, "Okay, this chunk of the model stayed rigid (stiff) relative to the other chunk. That must be the hinge!" It separates the static base (the wall) from the moving part (the door) and calculates the exact axis where the door spins.

4. The "Polishing" Phase

The first guess is usually a bit messy. So, PAOLI runs a three-step cleaning process:

Stabilize: Make sure the big, rigid parts are locked in place.
Refine: Fix the geometry (the shape) and re-label any parts that were accidentally grouped with the wrong piece.
Polish: Adjust the colors and lighting so the final 3D model looks exactly like the photos.

Why This Matters

Real Life: You don't need a fancy studio with 100 cameras. You can just take a few photos with your phone in your living room.
Robots: This helps robots understand how to open drawers, turn doorknobs, or fold laundry without needing a manual for every single object.
Games & VR: You can scan a real-world object, figure out how its joints work, and instantly put it into a video game where it moves realistically.

In short: PAOLI is like a master sculptor who can look at a few random snapshots of a folding chair, build a 3D model of it, figure out exactly how the legs fold, and tell you the math behind the movement—all without needing a ruler, a protractor, or a perfect studio setup.

1. Problem Definition

The paper addresses the challenge of reconstructing and modeling articulated objects (rigid parts connected by joints allowing rotation or translation) from sparse, unposed images.

Current Limitations: State-of-the-art methods (e.g., ArticulatedGS, PARIS) require dense multi-view observations (often ~100 views) and precise, ground-truth camera poses. They assume a shared, pre-calibrated coordinate system, which is impractical for real-world deployment.
The Gap: Existing approaches fail when given only a few images (e.g., 4 views) per articulation state where camera poses are unknown.
Core Challenge: The primary difficulty is not just 3D reconstruction, but establishing robust 3D-to-3D correspondences between independently reconstructed models that exist in different, unaligned coordinate frames. Traditional sparse keypoint matching fails on textureless surfaces or large articulations, and standard Structure-from-Motion (SfM) assumes static scenes.

2. Methodology: PAOLI

The proposed method, PAOLI, operates in three main stages to create a "digital twin" of the object, including geometry, appearance, and kinematics.

Stage 1: Independent Initialization

Input: Two sets of sparse images ( $I_s$ and $I_t$ ), each containing $K$ (e.g., 4) unposed views of the object in different articulation states.
Process: The method uses FreeSplatter (a sparse-view 3D Gaussian Splatting method) to independently reconstruct the 3D geometry ( $G_s$ and $G_t$ ) and estimate initial camera parameters for each set.
Refinement: Since the initial extrinsics are not pixel-perfect, the method freezes the Gaussians and optimizes only the camera extrinsics using a photometric rendering loss to ensure better alignment before the next stage.

Stage 2: Dense Correspondence via Deformation Fields

This is the core innovation. Instead of matching sparse keypoints, PAOLI learns a dense deformation field to align the source model ( $G_s$ ) to the target model ( $G_t$ ).

Deformation Network ( $F_{deform}$ ): A lightweight feed-forward network predicts rigid transformations (rotation, translation, scaling) for every Gaussian point in the source model based on its positional encoding.
Constraints: The network is constrained to generate low-frequency deformations to enforce local rigidity (preventing individual points from moving arbitrarily) while capturing global motion.
Optimization: The network is trained from scratch for each object by minimizing:
1. Chamfer Distance ( $L_{CD}$ ): Between the means of the deformed source Gaussians and the target Gaussians.
2. Photometric Loss ( $L_{photo}$ ): Between the rendered images of the deformed source and the target.
Correspondence: Once the source is deformed to match the target, dense 3D-to-3D correspondences are established using a 1-Nearest-Neighbor (1-NN) search.

Stage 3: Joint Optimization & Segmentation

With dense correspondences established, the method separates static and moving parts and refines the model.

Initial Segmentation: The correspondences are fed into TEASER++, a robust pose estimation algorithm, to identify the largest rigid component (root) and outliers (leaf/moving parts), providing initial part labels and rigid transformations.
Progressive Disentanglement Strategy: A three-phase joint optimization refines the geometry, segmentation, and kinematics:
1. Phase 1 (Rigid Pose Refinement): Freezes geometry and non-rigid displacements; optimizes only rigid transformations ( $\pi_r, \pi_l$ ) to stabilize coarse poses.
2. Phase 2 (Geometry & Segmentation Refinement): Unfreezes geometry and introduces a learnable per-Gaussian displacement term ( $\delta\mu$ ) to correct non-rigid errors. It iteratively updates part labels: points with large displacements are re-assigned to the part with the smallest alignment error.
3. Phase 3 (Appearance & Pose Polish): Locks geometry and displacements; fine-tunes part poses and Gaussian colors ( $c$ ) using photometric loss against original images.
Output: The final output includes the 3D Gaussian geometry, part segmentation, and kinematic parameters (joint axis, angle, pivot point).

3. Key Contributions

Problem Formulation: First work to address articulated object reconstruction from sparse, unposed images, identifying the 3D-to-3D correspondence between unaligned reconstructions as the central bottleneck.
Dense Deformation Correspondence: Proposes a robust, test-time optimization-based deformation field that outperforms sparse keypoint methods and pre-trained dense matching networks. It leverages both 3D geometric consistency and 2D photometric appearance.
Complete Pipeline: Demonstrates the first end-to-end solution that successfully recovers full articulated models (geometry, parts, and kinematics) from minimal data (4 views per state) without camera supervision.

4. Experimental Results

The method was evaluated on the PartNet-Mobility dataset and real-world objects.

Quantitative Performance:
- Rendering: Achieved high PSNR (~~26-29 dB) and SSIM (~~0.95) on novel views, comparable to methods trained with dense, posed data.
- Kinematics: Significantly lower angular error (e.g., ~0.9° vs. >30° for baselines) and position error compared to ArticulatedGS (AGS) and other baselines.
- Geometry: Low Chamfer Distance (CD), indicating accurate 3D shape recovery.
Baseline Comparison:
- AGS (ArticulatedGS): Failed to converge with only 4 views, even with ground-truth poses.
- GaussReg (Sparse Keypoints): Failed completely due to insufficient and noisy correspondences.
- FM-Dense (Pre-trained Dense): Highly unstable; most correspondences were rejected as outliers, leading to incorrect segmentation.
Real-World Validation: Successfully reconstructed real-world objects (e.g., drawers, scissors, Lego toys) with only 4 input images per state, demonstrating practical applicability.

5. Significance

Reduced Capture Requirements: PAOLI drastically lowers the barrier for 3D articulated modeling, requiring only 4 unposed images instead of hundreds of calibrated views.
Robustness: By solving the correspondence problem via a learned deformation field rather than relying on pre-trained models or sparse features, it handles textureless surfaces and large articulations effectively.
Practical Application: This enables downstream applications like robotic manipulation and digital twin creation in uncontrolled, real-world environments where dense scanning and calibration are infeasible.
Limitations: The method assumes rigid parts and requires at least two distinct articulation states. It may struggle if the initial 3D reconstructions are extremely misaligned (e.g., front vs. back views) or for objects with very thin structures undergoing large motion.