💻 computer science

Single-View 3D Shape and Pose Estimation of Transparent Objects Using Statistical Shape Models

This paper proposes a method for estimating the complete 3D shape and pose of transparent objects from a single RGB-D image by leveraging YOLO11 for initial detection and iteratively optimizing a category-specific statistical shape model to overcome the limitations of conventional depth sensors.

Original authors: Toshiro Nakano, Kaihei Okada, Tokuo Tsuji, Hiroaki Seki, Toshihiro Nishimura, Yosuke Suzuki, Tetsuyou Watanabe

Published 2026-08-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Toshiro Nakano, Kaihei Okada, Tokuo Tsuji, Hiroaki Seki, Toshihiro Nishimura, Yosuke Suzuki, Tetsuyou Watanabe

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to take a perfect 3D photo of a glass of water or a crystal vase using a standard depth camera. It's like trying to map a ghost; the light bounces off the glass, bends through it, or passes right through it, leaving the camera confused and the resulting image full of holes and glitches. This is a huge problem for robots. If a robot arm wants to pick up a glass bottle, it needs to know exactly where the bottle is, how it's tilted, and what its full shape looks like—even the parts hidden from view. Without that, the robot might try to grab empty air or crush the object. Scientists have tried to fix this with fancy new cameras or by taking hundreds of photos from every angle, but those solutions are often too slow, too expensive, or too complicated for a robot to use in real-time.

This paper tackles that messy problem by teaching a robot to "guess" the full shape of a transparent object using just one quick snapshot. The authors use a clever trick called a "Statistical Shape Model" (SSM). Think of an SSM like a digital clay mold that has learned the average shape of a specific category, like wine glasses. It knows that a wine glass usually has a stem, a bowl, and a rim, and it knows how those parts can stretch or shrink to make different glasses. By combining this "memory" of what a glass usually looks like with a single photo and a depth scan (which shows the table the glass is sitting on), the method can fill in the missing holes and figure out the object's exact position and orientation. The researchers found that this approach is not only more accurate at guessing the hidden parts of the object than previous methods but also much faster, making it a promising step toward robots that can safely handle delicate, see-through items.

The core of this study is a new method for estimating the complete 3D shape and pose (position and orientation) of transparent objects from a single RGB-D image (a photo that includes color and depth data). The authors argue that existing solutions have significant flaws: some require expensive, specialized cameras; others need dozens of photos from different angles, which takes too long; and many deep-learning models that try to "fill in" the missing depth often fail when the lighting or background changes. The paper explicitly rules out the idea that a robot can reliably measure transparent objects using standard depth sensors alone, noting that reflection and refraction create "missing and erroneous depth values." Instead of trying to fix the broken depth data directly, the proposed method replaces the broken data with a smart, mathematically generated model.

The process works like a game of "Hot and Cold" played with a 3D puzzle. First, the system uses a tool called YOLO11 to look at the color photo and identify what the object is (e.g., a bottle), which way it is facing, and its outline. It also looks at the depth image to find the flat surface (the table) the object is resting on. With this information, the system pulls up a "Statistical Shape Model" (SSM) from its database. This model is a flexible 3D template built from scanning many real examples of that object type. The system places this digital template into the scene, roughly where the object should be.

Then, the real magic happens. The system compares the outline of the digital template (projected onto the screen) with the actual outline of the object seen in the photo. If they don't match, the system tweaks the template's shape, rotation, and size, and then checks again. It does this over and over, using a mathematical technique called gradient descent to slowly nudge the template until its shadow perfectly aligns with the real object's silhouette. Crucially, the system also corrects the template's position based on the table surface to ensure it's sitting correctly. This iterative process allows the robot to "see" the parts of the object that are hidden or invisible to the camera, reconstructing the entire 3D shape, including the back side.

The authors tested this method on a dataset of plastic bottles, wine glasses, and cups, including scenarios where the objects were upright, lying on their side, or even upside down. They also tested it when objects were partially hidden behind one another. The results showed that their method produced significantly smaller errors in depth estimation compared to two other popular methods, ClearGrasp and TDCNet. On the researchers' own dataset, their method had a mean absolute error (MAE) of 14.5 mm, while ClearGrasp had an error of 74.9 mm and TDCNet had 39.2 mm. This suggests that the SSM approach is much more robust when the camera or lighting conditions change, unlike the other methods which seemed to struggle when moved away from their specific training data.

Furthermore, the study demonstrated that this method is faster than a previous version of the SSM technique. While the older method took an average of 4.32 seconds per object, the new method reduced this to 0.506 seconds. This speedup was achieved by optimizing how the system calculates the shape changes and by using a graphics card (GPU) to handle the heavy lifting in parallel. The researchers also found that the accuracy of the final 3D shape improved as they used more training models to build the SSM; with eight training models, the error in the reconstructed shape dropped to a mean of 2.77 mm, compared to 4.16 mm with only four models.

However, the paper is careful to note the limitations. The method relies on knowing the category of the object in advance; it cannot guess the shape of a transparent object it has never seen before or for which it doesn't have a pre-built model. It also assumes the object is sitting on a detectable flat surface. If the table is uneven or the object is floating, the method might struggle. Additionally, because the system is guessing the shape based on a single view, it relies heavily on the "prior knowledge" stored in the SSM. If the real object is very different from the average shape the model learned, the guess might be slightly off, particularly regarding the thickness of the object along the viewing direction.

In conclusion, the paper presents a method that successfully estimates the complete 3D shape and pose of transparent objects using a single view and a statistical model. It suggests that by combining visual outlines with a "memory" of typical shapes, robots can overcome the limitations of standard depth cameras. The findings indicate that this approach is more accurate and faster than current alternatives for handling transparent objects in various orientations and even when partially hidden, offering a practical path toward robots that can manipulate delicate, see-through items with confidence.

Technical Summary: Single-View 3D Shape and Pose Estimation of Transparent Objects Using Statistical Shape Models

Problem Statement

Transparent objects, such as those made of glass or plastic, present a significant challenge for robotic perception and manipulation. Conventional RGB-D sensors struggle to measure these objects accurately due to light reflection, refraction, and transmission, resulting in missing or erroneous depth values in the transparent regions. This unreliability hinders stable robotic grasping, which often relies on single-view observations from a camera mounted on the robot hand.

Existing solutions face several limitations:

Specialized Hardware: Methods using polarization or light-field cameras require dedicated, non-standard imaging devices.
Multiview Requirements: Techniques like Neural Radiance Fields (NeRFs) offer high accuracy but require multiple images and camera poses, making them unsuitable for real-time, single-view robotic tasks.
Learning-Based Depth Completion: While deep learning methods (e.g., ClearGrasp, TDCNet) can infer missing depth from single RGB-D images, their performance is highly dependent on training conditions. They often fail to generalize to different cameras, lighting, or object arrangements without costly retraining. Furthermore, they typically output only the visible surface depth rather than a complete 3D geometry, limiting their utility for grasp planning.
Generative Models: Large generative models can create 3D shapes from single images but often lack physical measurement accuracy, have high inference costs, and do not explicitly utilize transparent-object cues like missing depth or supporting planes.

Methodology

The authors propose a method to estimate the complete 3D shape and pose (position and orientation) of transparent objects from a single-view RGB-D image using a category-specific Statistical Shape Model (SSM). The approach avoids reliance on unstable depth measurements within the transparent regions by leveraging shape priors and geometric constraints.

1. Offline Preparation

SSM Construction: For specific object categories (bottles, cups, wineglasses), an SSM is constructed from multiple 3D training models. These models are aligned to ensure vertex correspondence using a shrink-wrap technique. The SSM represents shape variations via Principal Component Analysis (PCA), allowing the generation of new 3D shapes by varying weight parameters.
Instance Segmentation Training: A YOLO11-based model is trained to detect the target category, orientation, and instance mask from RGB images.

2. Online Inference

The inference pipeline consists of two main stages: initialization and optimization.

Observation Extraction:
- RGB Image: YOLO11 extracts the object category, orientation, and instance mask. Canny edge detection generates the target contour ( $C_{tgt}$ ).
- Depth Image: The depth image is back-projected to a point cloud to estimate the supporting plane (the surface the object rests on) using RANSAC.
Initialization:
- The SSM is initialized based on the detected category and orientation.
- For sideways or upside-down objects, the initial "sideways" direction is determined by analyzing the principal component vector of the target contour projected onto the supporting plane.
- The initial pose is refined by intersecting the target reference point with the estimated supporting plane.
Optimization Loop:
- Objective: Minimize the region error ( $e$ ) between the observed target contour ( $C_{tgt}$ ) and the projected SSM contour ( $C_{ssm}$ ). The error metric accounts for partial occlusion by masking out foreground occlusions.
- Parameter Updates: A gradient descent algorithm optimizes the SSM shape parameters ( $b$ ), rotation parameters ( $\theta, \phi, \psi$ ), and a scale factor ( $s$ ).
- Position Correction: During optimization, the SSM position is repeatedly corrected to align with the supporting plane and the target contour. Under partial occlusion, a horizontal position correction is applied separately to minimize region error.
- Acceleration: The deformation and projection steps are parallelized on a GPU to accelerate the sequential CPU-based process used in previous work.

Key Contributions

Complete 3D Shape and Pose Estimation: The method estimates the full 3D geometry (including unseen rear surfaces) and pose of transparent objects from a single RGB-D image, utilizing a category-specific SSM.
Robustness to Imaging Conditions: Unlike learning-based depth completion methods, the proposed approach does not require retraining for different cameras or environments. It relies on geometric priors and the supporting plane, making it adaptable to varying imaging conditions.
Extended Capabilities:
- Multiple Orientations: Unlike previous SSM-based methods restricted to upright objects, this method handles sideways and upside-down configurations.
- Partial Occlusion: By using instance-level contour information, the method can estimate shapes even when objects are partially occluded by other transparent items.
- Efficiency: The method significantly reduces processing time compared to previous SSM-based approaches through GPU parallelization and efficient gradient-based optimization.

Experimental Results

The method was evaluated on a custom RGB-D dataset (Ours-normal and Ours-occluded) and public datasets (ClearGrasp subset and TransCG).

Depth Estimation Accuracy:
- On the custom dataset, the proposed method achieved a Mean Absolute Error (MAE) of 14.5 mm and RMSE of 52.2 mm, outperforming ClearGrasp (MAE 74.9 mm) and TDCNet (MAE 39.2 mm).
- On the ClearGrasp subset, the proposed method maintained superior accuracy (MAE 11.2 mm) compared to ClearGrasp (31.4 mm) and TDCNet (40.0 mm).
- The method demonstrated consistent performance across different object categories and orientations (upright, sideways, upside-down), whereas learning-based methods showed significant performance drops when transferred to datasets with different distributions.
Complete 3D Shape Accuracy:
- Evaluated against ground-truth 3D scans, the method achieved a mean complete-shape RMSE of 2.77 mm.
- It outperformed the previous SSM-based method (Okada et al.) in all categories for upright objects (e.g., 2.39 mm vs. 3.41 mm mean RMSE).
- The method successfully estimated shapes under partial occlusion with a mean RMSE of 3.36 mm.
Processing Time:
- The proposed method processed objects in 0.506 seconds per object, substantially faster than the previous SSM method (4.32 seconds).
- While slower than TDCNet (0.0173 seconds per image), the proposed method provides complete 3D models necessary for robotic manipulation, which TDCNet does not.

Significance and Claims

The paper claims that the proposed method offers a practical solution for robotic perception of transparent objects by bridging the gap between single-view efficiency and complete 3D geometric accuracy. By utilizing Statistical Shape Models, the method circumvents the need for specialized sensors, multiview acquisition, or environment-specific retraining.

The authors emphasize that providing complete 3D geometry, including surfaces not visible to the camera, is critical for robotic manipulation tasks such as grasp planning and contact-state estimation, which cannot be reliably performed using only the visible surface depth provided by conventional depth sensors or learning-based completion methods. The ability to handle multiple orientations and partial occlusions further extends the applicability of the method to complex, real-world robotic scenarios.