💻 computer science

PASR: Pose-Aware 3D Shape Retrieval from Occluded Single Views

PASR (Pose-Aware 3D Shape Retrieval) is a framework that improves single-view 3D shape retrieval by distilling knowledge from a 2D foundation model into a 3D encoder and employing a test-time, analysis-by-synthesis optimization to jointly search for the best shape and pose, ensuring robustness against occlusions.

Original authors: Jiaxin Shi, Guofeng Zhang, Wufei Ma, Naifu Liang, Adam Kortylewski, Alan Vuile

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiaxin Shi, Guofeng Zhang, Wufei Ma, Naifu Liang, Adam Kortylewski, Alan Vuile

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a photo of a partially hidden object—say, a chair tucked behind a table—and you want to find the exact 3D model of that chair in a massive digital warehouse.

Current AI models usually try to do this by "guessing" based on a general vibe. They look at the photo and say, "This looks 80% like a wooden chair," and then they grab the closest thing they have. But if the chair is half-covered or at a weird angle, they often get confused and grab a stool or a sofa instead.

The researchers behind PASR have created a smarter way to solve this. Here is how it works, explained through a few simple analogies.

1. The "Master Artist" (Knowledge Distillation)

Imagine you want to teach a student how to draw 3D objects, but you only have 2D sketches to work with. To make the student an expert, you pair them with a Master Artist (this is the "2D Foundation Model" like DINOv3).

The Master Artist knows everything about colors, textures, and shapes in 2D. The student (the 3D Encoder) tries to draw a 3D object, and the Master Artist looks at the drawing and says, "Not quite! The way the light hits that corner is wrong." By constantly trying to match the Master's vision, the student learns to understand 3D shapes so deeply that they can "see" the 3D structure even when looking at a flat 2D photo.

2. The "Flashlight and Shadow" Method (Analysis-by-Synthesis)

This is the "secret sauce" of PASR. Most AI models look at a photo and try to match it to a memory. PASR does something different: it reconstructs the scene.

Think of it like this: You have a dark room and a mysterious object. Instead of just guessing what it is, you take a 3D model from your warehouse, shine a flashlight on it at a certain angle, and look at the shadow it casts on the wall.

If the shadow doesn't match the shape of the object in your photo, you don't give up.
You rotate the model and change the angle of the light until the shadow perfectly matches the outline of the object in your photo.

In the paper, this is called "Test-time Optimization." The AI doesn't just pick a shape and move on; it actively "fidgets" with the 3D model—turning it and tilting it—until the 3D features "click" into place with the pixels in the photo.

3. Why is this a big deal? (The "Occlusion" Superpower)

Because PASR uses this "shadow-matching" technique, it is incredibly good at dealing with occlusions (when something is blocking the view).

Imagine you are playing "I Spy," but someone is holding a piece of paper over half of the object. A normal AI might see a "leg" and a "seat" and think it's a stool. But PASR says, "Wait, if I rotate this chair model to this specific angle, the part that is visible matches perfectly, and the part that is hidden is exactly where the paper is blocking the view."

It uses the visible parts to "solve" the puzzle of the hidden parts.

Summary: The Three Big Wins

It’s a Multi-Tasker: Because it’s so good at matching shapes and angles, it doesn't just find the object; it also tells you exactly how it’s tilted (Pose Estimation) and what it is (Classification).
It’s a Generalist: It can find shapes it has never even seen before because it understands the logic of geometry, not just a list of memorized objects.
It’s Robust: It doesn't get "distracted" by messy backgrounds or objects being partially hidden.

In short: PASR doesn't just look at a picture and guess; it plays a high-speed game of "match the shadow" to find the truth.

Technical Summary: PASR (Pose-Aware 3D Shape Retrieval)

PASR: Pose-Aware 3D Shape Retrieval from Occluded Single Views addresses the fundamental challenge of retrieving a 3D mesh from a single 2D RGB image, a task complicated by real-world factors such as partial occlusions, varying camera poses, and the gap between synthetic 3D models and real-world object instances.

1. The Problem

Existing single-view 3D shape retrieval methods generally fall into two categories:

Multimodal Alignment: Mapping 3D features into pre-existing vision-language spaces (e.g., CLIP). These often fail to generalize to real-world images because they rely on precise alignments of synthetic data.
Joint Embedding: Learning a common space for 2D and 3D features. These are typically "view-agnostic" (holistic/global), making them highly vulnerable to partial occlusions and unable to capture fine-grained geometric details or pose information.

2. Methodology

The authors propose a novel framework that reformulates retrieval as a feature-level analysis-by-synthesis problem. Instead of a single forward pass, the model iteratively optimizes the match between a 3D candidate and the 2D query.

A. Training: 2D-to-3D Knowledge Distillation

3D Encoder: A point-based encoder (PointNeXt) extracts multi-scale, point-level features. By aggregating features from multiple scales via K-Nearest Neighbors (KNN) and interpolation, the model captures both local geometric details and global semantic context.
Distillation: To bridge the 2D-3D gap, the model distills knowledge from a 2D foundation model (DINOv3). The 3D point features are projected into a 2D feature map using a differentiable renderer under ground-truth poses. A cosine similarity loss is applied between the projected 3D features and the 2D foundation model's feature map, ensuring the 3D encoder learns rich, semantic, and spatial representations.

B. Inference: Two-Stage Retrieval

Stage 1: Initial Search (Coarse): Since the camera pose is unknown, the model performs a discrete search. It renders the features of all candidate 3D shapes from a set of predefined poses (e.g., 192 views) and compares them to the query image's feature map. This provides a coarse ranking and an initial pose estimate.
Stage 2: Test-Time Optimization (Fine): For the top- $k$ candidates, the model performs analysis-by-synthesis. It uses the AdamW optimizer to iteratively refine the 3D pose (elevation, azimuth, and in-plane rotation) to minimize the alignment loss between the query's feature map and the rendered 3D feature map. This makes the model inherently robust to occlusions, as it searches for the best geometric fit.

3. Key Contributions

Explicit Point-Level Representation: Unlike holistic methods, PASR learns fine-grained, view-aware 3D point features.
Analysis-by-Synthesis Framework: By using test-time optimization, the model can "reason" about the best pose and shape, providing robustness against partial occlusions.
Multi-Task Capability: The framework is unified; the same architecture achieves state-of-the-art 3D shape retrieval, competitive 3D pose estimation, and accurate category classification.
Generalization: The method demonstrates a superior ability to transfer to "unseen" 3D meshes (shapes not present in the training set).

4. Results

State-of-the-Art Performance: PASR achieves significant improvements on the Pix3D and Pascal3D datasets. On Pix3D, it reached an overall Top-1 accuracy of 81.59%, outperforming previous bests by a wide margin.
Robustness to Occlusion: While existing methods' performance drops sharply as occlusion increases (L1 to L3), PASR maintains much higher accuracy. On Pascal3D under heavy (L3) occlusion, it outperformed the strongest baseline (CMIC) by 8.29%.
Unseen Shape Transfer: In experiments transferring from Pix3D to Pascal3D (disjoint datasets), PASR achieved 63.40% Top-1 accuracy, whereas supervised baselines like CMIC dropped to 9.13%.

5. Significance

PASR represents a shift from "black-box" feed-forward retrieval to an interpretable, optimization-based approach. By leveraging the power of 2D foundation models to supervise 3D geometry, it solves the critical problem of bridging the gap between synthetic 3D databases and messy, occluded real-world imagery. This has significant implications for downstream robotics, autonomous navigation, and 3D scene understanding.