UniPR: Unified Object-level Real-to-Sim Perception and Reconstruction from a Single Stereo Pair

This paper introduces UniPR, the first end-to-end framework that performs unified object-level perception and reconstruction from a single stereo pair by leveraging geometric constraints and a novel pose-aware shape representation to overcome the inefficiencies and cumulative errors of traditional modular pipelines, supported by the new large-vocabulary LVS6D dataset.

The Gist

Imagine you are a robot trying to pick up a coffee mug from a messy table. To do this successfully, you need more than just a picture; you need to know exactly how big the mug is, where it is in 3D space, and what shape it has. If you guess wrong, you might crush the mug or miss it entirely.

For a long time, robots have struggled with this because their "vision" systems are like a team of specialists who don't talk to each other. One person finds the object, another cuts it out of the picture, a third guesses its size, and a fourth tries to build a 3D model. By the time the information reaches the robot's hand, errors have piled up, and the robot is confused.

Enter UniPR (Unified Object-level Real-to-Sim Perception and Reconstruction). Think of UniPR not as a team of specialists, but as a super-photographer who is also a master sculptor and a mathematician all rolled into one.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Blindfolded Sculptor"

Most current AI methods for turning 2D photos into 3D objects are like a sculptor working in the dark. They look at a flat photo and guess the depth.

• The Issue: Without a second eye (stereo vision), they can't tell if a toy car is small and close, or huge and far away. They often get the proportions wrong (making a tiny cup look like a giant bucket).
• The Old Way: The old method was like an assembly line. Step 1: Find the cup. Step 2: Cut it out. Step 3: Guess the shape. Step 4: Guess the pose. If Step 1 makes a tiny mistake, Step 4 becomes a disaster.

2. The Solution: The "All-in-One Super-Brain"

UniPR changes the game by doing everything at once in a single glance.

• One Shot, Whole Scene: Instead of processing objects one by one, UniPR looks at the entire scene (like a stereo camera pair, which is like human eyes) and instantly identifies every object, their sizes, and their 3D shapes simultaneously. It's like looking at a messy desk and instantly knowing the exact dimensions and location of the pen, the cup, and the laptop all at the same time.
• The "Real-to-Sim" Magic: The goal is to move things from the "Real World" to a "Simulation World" (like a video game for robots) perfectly. UniPR ensures that if a real cup is 10cm tall, the digital cup is exactly 10cm tall. This is crucial for robots to interact safely with the real world.

3. The Secret Sauce: "Pose-Aware" Shapes

This is the paper's biggest innovation.

• The Old Problem: Imagine trying to describe a chair. Usually, AI tries to force the chair into a "standard box" (a canonical space) before describing it. But if the chair is tilted or rotated, this "box" breaks, and the AI gets confused about the size.
• The UniPR Fix: UniPR uses a Pose-Aware Shape Representation. Think of this as a mold that can stretch and rotate. Instead of forcing the object into a rigid box, UniPR understands the object's shape exactly as it is sitting there, tilted or turned. It doesn't need to "normalize" the object first; it just sees the shape and the pose together. This prevents the "squishing" or "stretching" errors that happen when objects are rotated.

4. The "Eyes" of the System: Stereo Vision

UniPR uses stereo images (two pictures taken from slightly different angles, just like your left and right eyes).

• Why it matters: This gives the system depth perception. It's the difference between looking at a flat photo of a ball and holding a real ball. Because it has depth, it knows the true scale of the object. It doesn't have to guess if the object is a toy or a life-size version.

5. The Result: Speed and Accuracy

• Speed: Because it processes the whole scene in one go (instead of an assembly line), it is 100 times faster than previous methods. It's like the difference between building a house brick-by-brick with a team of one, versus using a 3D printer that builds the whole room in a minute.
• Accuracy: It preserves the true physical proportions. If you ask a robot to grab a specific object, UniPR gives it a blueprint that is mathematically accurate, not just a "good guess."

Summary Analogy

Imagine you are trying to pack a suitcase.

• Old Methods: You ask one friend to list the items, another to measure them, and a third to draw them. By the time you get the list, the measurements are wrong, and you can't fit the items in the suitcase.
• UniPR: You have a super-intelligent packing assistant who looks at the pile of clothes, instantly knows the size and shape of every item, understands how they are folded, and tells you exactly how they fit in the suitcase—all in a split second, without needing to measure anything manually.

In short: UniPR is a new AI system that lets robots "see" the 3D world with perfect accuracy and speed, allowing them to interact with real objects as if they were in a perfect video game simulation. It's a huge leap forward for making robots that can actually help us in our daily lives.

Technical Summary

1. Problem Statement

The paper addresses the critical challenge of Real-to-Sim (Real-to-Simulation) transfer in robotics. For robots to interact effectively with the physical world in simulation, they require accurate 3D reconstructions of objects that preserve metric scale, physical proportions, and spatial relationships.

Existing approaches suffer from three main limitations:

• Modular Pipelines: Traditional methods decompose the task into separate stages (detection, segmentation, pose estimation, shape reconstruction). This leads to error propagation, where mistakes in early stages (e.g., segmentation) degrade later stages, and a lack of global context.
• Inefficiency: Processing objects sequentially (one by one) is computationally expensive and slow, making it unsuitable for dynamic robotic applications.
• Scale Ambiguity & Proportion Errors: Monocular (single-camera) image-to-3D generative models often fail to preserve true physical proportions due to inherent scale ambiguity. They also struggle to handle multiple objects in a single scene simultaneously.

2. Methodology: UniPR

The authors propose UniPR, the first end-to-end, unified framework that performs object-level perception and 3D reconstruction directly from a single stereo image pair in a single forward pass.

Core Components:

1. Pose-Aware Shape Representation (PASR):

   • Concept: Instead of relying on predefined "canonical spaces" (standardized orientations for object categories, which limit scalability), PASR jointly encodes object pose and geometry directly in the observation space.
   • Spherical Voxel Space: To handle arbitrary rotations without scale ambiguity, the method uses a spherical voxel space (unit sphere) rather than a cubic one. This ensures objects remain within bounds regardless of rotation, preventing the re-normalization issues found in cubic spaces.
   • VAE Architecture: A Pose-Aware Shape Variational Autoencoder (VAE) is trained to compress object surface point clouds into lightweight latent embeddings. The decoder reconstructs the 3D shape (occupancy) from these embeddings.

2. Triplane-View (TPV) Encoder:

   • The network uses a Triplane-View mechanism to aggregate stereo features into a global coordinate system.
   • It initializes three feature planes ($T_{UV}, T_{UD}, T_{VD}$) representing different views of the 3D space.
   • Stereo Cross-Attention: Features from left and right stereo images are lifted onto these planes using back-projection and cross-attention, effectively fusing geometric depth cues.

3. End-to-End Transformer Decoder:

   • Object Queries: Similar to DETR, the decoder uses object queries to extract instance-specific features from the TPV features.
   • Parallel Processing: Unlike sequential pipelines, UniPR processes all objects in the scene simultaneously through these queries.
   • Outputs: For each object, the network predicts:
     • 3D Position ($x, y, z$)
     • Scale ($s$)
     • Semantic Category (via CLIP integration using 2D projections)
     • Shape Embedding (mean $\mu$ and variance $\sigma^2$ of the latent distribution).
   • Shape Decoding: The predicted shape embedding is fed into the pre-trained VAE decoder to generate the final 3D mesh in camera coordinates.

4. Training Strategy:

   • Loss Functions: The model is trained using Binary Cross Entropy (BCE) for occupancy, KL-divergence regularization for the latent space, and L1 loss for position/scale.
   • Matching: A Hungarian algorithm is used for one-to-one matching between predictions and ground truth.

3. Key Contributions

• First End-to-End Real-to-Sim Framework: UniPR eliminates intermediate modules (detection, segmentation), preventing error accumulation and enabling whole-scene processing in a single inference.
• Pose-Aware Shape Representation (PASR): A novel representation that removes the need for category-specific canonical definitions, allowing the model to scale to hundreds of object categories while preserving true physical proportions.
• Stereo-Based Metric Accuracy: By leveraging stereo vision, UniPR resolves scale ambiguity, ensuring reconstructed objects have accurate metric dimensions essential for robotic manipulation.
• LVS6D Dataset: The authors constructed a large-scale stereo dataset containing 6,300+ objects across 192 categories, specifically designed for large-vocabulary 6D pose estimation and shape reconstruction.
• Efficiency: The framework achieves up to 100× faster generation compared to sequential baselines by processing multiple objects in parallel.

4. Experimental Results

The paper evaluates UniPR on the new LVS6D dataset, public stereo datasets (TOD, SS3D), and against state-of-the-art (SOTA) image-to-3D models.

• Reconstruction Quality:
  • Shape Proportion: UniPR significantly outperforms generative models like Hunyuan3D and Trellis in Shape Proportion Error (SPE), achieving 3x improvement in accuracy.
  • Geometric Fidelity: It achieves superior Chamfer Distance (CD) and F-Score compared to baselines, even when baselines are given perfect ground-truth masks and poses.
• Pose Estimation:
  • On the LVS6D dataset, UniPR achieves an Average Precision (AP) of 0.752 on the "Hard" subset (complex objects), significantly outperforming the previous SOTA stereo method, Coders (0.483).
  • It demonstrates robust generalization to unseen object categories and complex intra-class variations.
• Efficiency:
  • Inference Time: UniPR processes a full scene with 5 objects in 0.63 seconds, whereas sequential methods (like Trellis) take 43.08 seconds for the same scene.
• Real-World Validation:
  • Real-robot experiments confirm that the metric-accurate predictions allow a robot to successfully grasp objects using a simple top-down policy, validating the practical utility for real-to-sim transfer.

5. Significance

UniPR represents a paradigm shift in robotic perception. By unifying detection, pose estimation, and shape reconstruction into a single, efficient, and metric-accurate pipeline, it solves the "scale ambiguity" and "error propagation" problems that have hindered real-to-sim transfer.

• Scalability: The elimination of canonical spaces allows the system to handle a vast vocabulary of objects without retraining for new categories.
• Robotics Impact: The ability to generate physically accurate, metric-scale 3D models of arbitrary objects in real-time from a single stereo pair is a foundational step toward autonomous robots that can interact with unstructured, real-world environments without prior object models.
• Dataset Contribution: The release of LVS6D provides a crucial benchmark for future research in large-vocabulary 6D pose estimation and 3D reconstruction.