💻 computer science

Category-Level 3D Correspondence in Camera Space via Morphable Object Priors

This paper introduces HouseCorr3D, a large-scale benchmark with 178k images and 3D keypoint annotations, alongside Morpheus, a method that learns morphable object priors to achieve state-of-the-art category-level 3D correspondence in camera space without explicit correspondence supervision.

Original authors: Leonhard Sommer, Artur Jesslen, Basavaraj Sunagad, Adam Kortylewski

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Leonhard Sommer, Artur Jesslen, Basavaraj Sunagad, Adam Kortylewski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand the world, not just by seeing pictures, but by understanding the 3D shape of objects.

The problem with current technology is that while robots are getting good at spotting where an object is (its pose), they struggle to understand what the object is made of. If a robot sees a mug, it knows where the handle is. But if it sees a weirdly shaped, squashed mug, or a mug that is half-hidden behind a book, it gets confused. It doesn't know that the "handle" of the squashed mug is still the same functional part as the handle of the normal mug.

This paper introduces a new way to solve this, called HouseCorr3D, and a new tool to do it called Morpheus.

Here is the breakdown in simple terms:

1. The Core Idea: The "Universal Template"

Think of every object category (like "mugs" or "chairs") as having a universal, invisible template.

The Analogy: Imagine a master clay mold for a "chair." Even if you squish the clay to make a tiny stool or stretch it to make a giant throne, the mold knows that the "seat" is always in the middle and the "legs" are always at the bottom.
The Innovation: Previous methods tried to match pixels in a 2D photo (like matching a red dot on a screen). This paper says, "No, let's match the 3D clay itself." They teach the computer to predict how that invisible template deforms to fit the specific object in the photo.

2. The New Benchmark: "HouseCorr3D"

To test if this works, the authors built a massive new playground called HouseCorr3D.

What it is: A giant library of 178,000 images of 50 common household items (like bottles, shoes, and toys).
The Secret Sauce: Unlike other datasets that only show you what is visible, this one includes "Amodal" labels.
- The Analogy: If you look at a ball half-buried in sand, you can only see the top. A normal dataset only labels the top. HouseCorr3D labels the entire ball, including the part buried in the sand. It teaches the AI to "imagine" the whole object, even the parts it can't see.
Symmetry: It also handles tricky symmetries. If you have a round cup, the "front" and "back" are the same. The dataset knows this so it doesn't punish the AI for guessing the "back" when the "front" was asked for.

3. The Method: "Morpheus"

The authors created an AI system named Morpheus (named after the shapeshifter in The Matrix, though here it's about shape, not dreams).

How it works: Instead of trying to memorize every single mug, Morpheus learns the "shape language" of mugs.
- When it sees a new mug, it asks: "How do I stretch and squish my universal mug template to look exactly like this one?"
- Once it figures out the shape, it can instantly point to the "handle" or the "rim" on any mug, even if that mug is twisted, broken, or hidden behind other things.
The Magic: The paper claims that Morpheus learned to do this without being explicitly taught where the handles are. It just learned the rules of shape deformation, and the "correspondence" (knowing which part is which) emerged naturally as a side effect.

4. Why This Matters (According to the Paper)

Beyond 2D: Current methods are like looking at a flat map; they get lost when the terrain changes. This method builds a 3D model in the camera's view, so it understands depth and occlusion.
Robotic Hands: For a robot to pick up a cup, it needs to know where the handle is, even if the handle is hidden from the camera's view. This system allows the robot to "fill in the blanks" of the invisible parts.
No "Cheating": The paper shows that you don't need to manually label every single point on every single object to get this right. If you teach the AI the rules of shape, it figures out the rest.

Summary

The paper says: "We built a new test (HouseCorr3D) that forces AI to understand the whole 3D shape of objects, including hidden parts. We built a new AI (Morpheus) that learns a flexible 'template' for each object type. By learning how to stretch this template, the AI automatically learns to find matching parts (like handles or wheels) across different objects, even when they are hidden or shaped weirdly."

The result is a system that is much better at understanding 3D objects in a camera view than previous methods, setting a new standard for how robots and VR systems might understand the physical world.

Technical Summary: Category-Level 3D Correspondence in Camera Space via Morphable Object Priors

Problem Definition

The paper addresses the challenge of monocular category-level 3D correspondence in camera space. While recent advances have improved category-level pose estimation, existing representations often fail to capture the fine-grained semantics required for reasoning about object parts, functions, and interactions.

The task is formally defined as follows: Given two RGB-D images ( $I_q$ and $I_t$ ) of objects from the same category and a query 3D point $x_q$ in the camera space of $I_q$ , the goal is to predict the corresponding 3D point $x_t$ in the camera space of $I_t$ that represents the same semantic part. Unlike prior approaches that map pixels to normalized canonical spaces (e.g., NOCS), this work targets correspondences directly in 3D camera space. This approach aims to resolve ambiguities inherent in 2D matching (viewpoint dependence, occlusion, symmetry) and avoids the undefined nature of object-centric spaces where scale and center are not well-defined.

A critical gap identified is the lack of benchmarks that support amodal correspondence (inferring parts occluded or not visible) and explicit symmetry handling (where multiple geometric positions are semantically equivalent) for diverse household objects.

Methodology: Morpheus

The authors propose Morpheus, a framework that learns morphable category-level shape priors to establish semantically consistent 3D correspondences without explicit correspondence supervision. The core idea is to represent all objects within a category as identity-preserving deformations of a shared template mesh.

1. Mesh-Based 3D Correspondence Prediction

Morpheus mediates all predictions through a shared deformable template. For an input RGB-D image, the model predicts:

An instance-specific deformation of the template mesh.
A 6D object pose ( $\omega$ ), estimated using a pretrained pose diffusion network.

Correspondence is established via mesh transfer:

A query 3D point $x_q$ is projected onto the surface of the deformed query mesh ( $M_q^{def}$ ).
The point's location is encoded as barycentric coordinates relative to the underlying mesh face.
Because all instances share the same mesh topology, these coordinates are transferred directly to the target deformed mesh ( $M_t^{def}$ ), yielding the corresponding point $x_t$ in the target camera space.
This reduces the problem to predicting the pose and deformation of a shared template rather than directly matching points between images.

2. 3D Morphable Priors

The framework utilizes a hybrid volumetric mesh representation:

Canonical Shape: Represented as a Signed Distance Field (SDF) $\phi_{sdf}$ , which is converted to a mesh via Differentiable Marching Tetrahedra. This allows for flexible modeling of intricate geometries.
Instance-Specific Deformations: An affine deformation field $\phi_a$ adapts the canonical mesh to specific instances. This field is driven by a latent code $l$ extracted from the input image by a deformation encoder (based on a DINOv2 backbone).
The deformation is defined as an affine mapping that displaces vertices based on the latent code, preserving the semantic identity of template vertices across deformations.

3. Training Objectives

Morpheus is trained using geometric supervision without explicit correspondence labels. Semantic alignment emerges implicitly because every instance must deform the same canonical template while remaining consistent with observed data. The training loss includes:

2D Loss: Amodal object mask reconstruction (MSE) and distance transform overlap to ensure the deformed mesh matches the visible and occluded regions of the object.
3D Loss: Chamfer distance between the deformed mesh vertices and ground-truth mesh vertices.
Regularization: Eikonal loss (to enforce SDF properties), $\ell_2$ deformation regularization (to prevent excessive deformation), and edge-based smoothness regularization.
Pose Supervision: 6D pose supervision is provided to stabilize optimization and reduce local minima.

Key Contributions

Task Formulation: The paper identifies monocular category-level 3D correspondence in camera space as a necessary step beyond pose-centric representations toward semantically aligned 3D understanding.
HouseCorr3D Benchmark: The authors introduce HouseCorr3D, the first large-scale benchmark for this task. It comprises:
- 178,000 images across 50 household object categories.
- 280 unique object instances.
- 3D keypoint annotations directly on CAD models.
- Amodal correspondence labels for occluded regions.
- Explicit symmetry annotations to handle geometric ambiguities fairly.
Morpheus Framework: A method that learns morphable priors to produce semantically consistent 3D correspondences directly in camera space by disentangling canonical shape, deformation, and object pose.
Empirical Validation: Demonstration that semantic 3D object understanding can emerge without direct correspondence supervision, achieving state-of-the-art results on the proposed benchmark.

Results

Experiments on HouseCorr3D show that Morpheus substantially outperforms existing baselines, including:

2D Feature Matching: DINOv2, MagicPony2D, and NOCS (lifted to 3D).
3D Space Matching: GenPose++ and MagicPony.

Key Findings:

Performance: Morpheus achieves the highest PCK@0.1 (Percentage of Correct Keypoints) across 2D, 3D modal (visible), and 3D amodal (occluded) settings.
Occlusion Handling: Unlike 2D methods which fail on occluded parts, Morpheus successfully reconstructs occluded regions due to its amodal supervision and shared prior.
Symmetry and Deformation: Morpheus handles object symmetries and shape variations better than methods relying on normalized object spaces (like NOCS) or entangled canonical/camera spaces (like MagicPony).
Generalization: Evaluation on a filtered subset of the real-world ROPE dataset (from Omni6DPose) confirms that the model generalizes to real data, with performance drops attributed primarily to noisier depth and annotations rather than model failure.

Significance and Claims

The paper claims that establishing correspondences in 3D camera space is a fundamental shift from previous 2D or normalized 3D object space approaches. By leveraging morphable object priors, the authors demonstrate that:

Semantic 3D correspondences can be learned implicitly through shared deformable representations, eliminating the need for expensive, explicit correspondence supervision.
This approach enables amodal reasoning, allowing systems to infer the complete 3D structure of objects even when parts are occluded, which is critical for robotic manipulation and interaction.
The HouseCorr3D benchmark fills a critical void in the literature, providing the first large-scale dataset with amodal and symmetry-aware annotations for evaluating category-level 3D correspondence from single images.

The work suggests that future progress in embodied robotics and AR/VR requires moving beyond 2D feature matching to robust 3D reasoning that accounts for full object geometry, including occluded regions and symmetries.