💻 computer science

GHOST: Geometry-Guided Hallucination of Opaque Surface Textures

The paper proposes GHOST, a geometry-guided preprocessing framework that leverages visual foundation models to synthesize opaque, texture-rich representations of transparent objects, thereby significantly improving the performance of downstream depth estimation and 3D reconstruction tasks without requiring model retraining.

Original authors: Langxu Zhao, Zuan Gu, Tianhan Gao

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Langxu Zhao, Zuan Gu, Tianhan Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to take a 3D photo of a glass vase. To your eyes, it looks beautiful, but to a standard computer camera, it's a nightmare. Why? Because most 3D cameras work on a simple rule: "If light hits a surface, it bounces back." This is called the Lambertian assumption. But glass? Glass plays by different rules. It lets light pass right through (transparency) and bounces it off in shiny, confusing ways (specular reflection).

When a standard 3D camera tries to measure the depth of that glass vase, it gets confused. It sees the background through the glass and thinks the glass is flat or missing entirely. It's like trying to measure the height of a ghost by looking at the wall behind it.

Enter GHOST (Geometry-Guided Hallucination of Opaque Surface Textures). Think of GHOST not as a new camera, but as a magical "pre-painting" tool. Before the 3D camera even tries to do its job, GHOST steps in and "repaints" the glass. It doesn't change the shape of the vase; it just gives the glass a fake, solid, opaque skin that looks like a real object, so the 3D camera can finally "see" it.

The Four-Step Magic Trick

The authors built a pipeline with four distinct steps to pull off this illusion, and they tested it rigorously on datasets like ClearGrasp and ClearPose.

1. The Mask Maker (TransDINO)
First, the system needs to know exactly where the glass is. It uses a tool called TransDINO to draw a perfect outline around the transparent object.

The Analogy: Imagine a super-precise artist who can trace the invisible edges of a glass cup on a piece of paper.
The Proof: In tests, this step was incredibly sharp. On the ClearGrasp dataset, TransDINO achieved a 92.44% accuracy (measured by mIoU), beating other top methods like EBLNet (64.99%) and TROSNet (71.40%). It's so good it can even handle tricky, stacked glass objects where others fail.

2. The "Un-Glassing" Artist (TransDecomp)
Once the outline is drawn, the system has to figure out what the glass is actually made of. It separates the image into two parts: the alpha matte (how see-through it is) and the foreground (the actual color and shine of the glass itself).

The Analogy: Think of this like a chef separating the clear broth from the solid noodles. The system assumes that if the glass is thin, the light doesn't bend too much, so it can mathematically "un-mix" the background from the glass.
The Catch: If the glass is super thick or bends light wildly (strong refraction), this step admits it might struggle to get the perfect "noodle" color. Instead, it lowers the "see-through" value, telling the next step, "Hey, this part is basically solid, treat it like a rock."

3. The Shape Detective (DAF-Net)
Now the system needs to know the curves of the glass. Standard cameras can't see the curves of clear glass because there are no shadows or textures to grab onto. DAF-Net steps in to guess the surface angles (normals).

The Analogy: Imagine a blind sculptor who can't see the statue but can feel the air currents around it. DAF-Net uses deep "feeling" features from a model called DINOv3 to guess the bumps and curves of the glass, even though it's invisible.
The Result: This step creates a map of the glass's 3D shape, which is usually impossible for standard tools to get right.

4. The Texture Painter (GeoSemTransNet)
Finally, the system takes all that information—the outline, the "un-glassed" color, and the 3D shape map—and paints a new, fake image. It replaces the transparent glass with a solid, textured object that has the exact same 3D shape.

The Analogy: This is like taking a wireframe model of a ghost and filling it with clay and paint, making it look like a solid statue.
The Goal: The goal isn't to make the glass look real to a human; it's to make it look real to a 3D camera. The camera now sees a solid object and can calculate its depth perfectly.

Does It Actually Work?

The authors didn't just guess; they ran the numbers. They took existing, top-tier 3D models (like DA3, MoGe2, and VGGT) that usually fail miserably on glass, fed them these "GHOST-painted" images, and watched what happened.

The Depth Test:
When they used the MoGe2 model on raw glass images, the depth error (RMSE) was 0.018. But when they used the GHOST-painted version, the error dropped to 0.014.

The Big Win: The accuracy of the model getting the depth right within a tiny margin (δ1.05) jumped from 86.7% to 93.3%.
The Shape Test: The angle error for the surface normals (how well the model guessed the curves) dropped from 35.64 degrees to 24.37 degrees. That's a huge improvement in understanding the shape.

The 3D Reconstruction Test:
When they tried to build a 3D model of the glass using these painted images, the results were even more dramatic.

Using the DUSt3R model, the "Accuracy" (how close the 3D model is to the real thing) improved from 0.57 to 0.25 (remember, lower is better here).
The "F-Score" (a measure of overall quality) skyrocketed from 77.36 to 88.66.

What This Is NOT (And What It Rules Out)

It's important to know what GHOST doesn't do, because the authors were very clear about that.

It's not a "Depth Completion" fix: Some old methods try to fix the depth after the camera messes up. The authors argue this is inefficient and requires retraining the camera model for every new type of glass. GHOST rejects this; it fixes the input so the camera doesn't need retraining.
It's not a NeRF (Neural Radiance Field): Other researchers try to use complex, slow 3D models (NeRFs) that need hours of training for a single scene. GHOST explicitly avoids this. It's fast (0.38 seconds per image on a powerful RTX 4090 GPU) and works with standard, off-the-shelf models.
It's not magic for everything: The paper notes that if the glass has very strong refraction (bending light a lot), the "un-glassing" step might not get the perfect color. However, the system is smart enough to signal that the object is "mostly solid," so the next step still works.

The Bottom Line

The paper suggests that by "hallucinating" a solid texture onto transparent objects, we can trick standard 3D cameras into seeing what they usually can't. It's a clever workaround that doesn't require building new cameras or training new AI models from scratch.

The authors showed that this method significantly boosts the performance of existing tools. On the ClearGrasp dataset, their method helped standard models achieve depth accuracy that rivals or beats specialized, glass-only models. It's a "plug-and-play" solution: you take your glass object, run it through GHOST, and suddenly your 3D scanner works like a charm.

As the authors put it, this creates a new way to solve the "perception silo" problem, where we usually have to build a separate, narrow tool just to see glass. With GHOST, we might finally be able to see glass using the same tools we use for everything else.

Technical Summary: GHOST (Geometry-Guided Hallucination of Opaque Surface Textures)

1. Problem Statement

Transparent objects present a fundamental challenge for mainstream depth estimation and 3D reconstruction systems. These systems typically rely on the Lambertian diffuse reflection assumption, which is violated by transparent surfaces due to light transmission, specular reflection, and refraction. Consequently, standard models suffer from severe geometric degradation, often predicting background depth rather than the actual surface of the glass or transparent object.

Existing solutions face significant limitations:

Depth Completion & NeRF: Methods like depth completion often require expensive ground-truth data (obtained via specialized hardware or powder spraying) and specific end-to-end training, leading to poor generalization on unseen materials. NeRF-based approaches suffer from low inference efficiency, requiring multi-view inputs and lengthy per-scene training, creating a "perception silo" that forces the abandonment of versatile State-of-the-Art (SOTA) models.
Segmentation: Current segmentation models struggle with the coupling of object appearance and background due to refraction and reflection, often failing under low contrast or complex occlusion.

The core challenge is to enable generic, off-the-shelf 3D vision models to process transparent objects without retraining, while avoiding the high cost of acquiring ground-truth depth labels for transparent materials.

2. Methodology

The authors propose GHOST (Geometry-guided Hierarchical Opaque Style Transfer), a universal preprocessing framework. Instead of modifying downstream models, GHOST "repaints" transparent regions into structurally consistent opaque representations using visual foundation models. The pipeline consists of four topologically interconnected modules:

2.1. TransDINO: Transparent Object Localization

TransDINO is a hybrid segmentation framework designed to extract high-quality semantic masks for transparent objects.

Architecture: It utilizes a U-Net structure with a dual-stream encoding mechanism. One stream extracts multi-scale spatial features using a pre-trained ResNet-34 backbone, while the other processes features from DINOv3 (a visual foundation model) to capture dense semantic boundaries.
Fusion: A novel TransFusionChannelAttn module aligns heterogeneous features using spatial cross-attention, treating RGB features as queries and DINOv3 semantic features as keys/values. A dual-pooling channel attention mechanism suppresses noise, and a learnable scaling factor stabilizes the deep fusion training.

2.2. TransDecomp: Alpha and Foreground Decomposition

TransDecomp decomposes the transparent object region into an alpha matte ( $\alpha_{trans}$ ) and a foreground grayscale map ( $F_g$ ).

Physical Assumption: Under the "weak refraction" assumption (where light displacement is small), the interaction is approximated as alpha blending. For strong refraction, the network generates lower alpha values to signal non-transparency.
Architecture: A U-Net with gated convolutions and a Transformer bottleneck. Gated convolutions distinguish valid object features from background noise, while the ViT bottleneck captures long-range dependencies to resolve refractive ambiguities.
Dual-Head Decoding: The network outputs an Opacity Head (predicting $\alpha$ ) and a Foreground Head (predicting intrinsic color/specular reflections). These are constrained by the object mask and trained using a reconstruction loss against the original image, guided by a pre-computed background prior generated via InpaintAnything.

2.3. DAF-Net: Surface Normal Estimation

DAF-Net recovers fine-grained surface normals ( $N$ ) to encode geometric curvature.

Architecture: It bridges semantic priors and local geometric cues by fusing frozen DINOv3 features with a Multi-Modal Spatial Adapter (MMSA) that extracts pixel-aligned geometric details from RGB, masks, alpha mattes, and foreground guidance.
Fusion: A Fidelity-Aware Fusion Block uses a Squeeze-and-Excitation (SE) mechanism to adaptively suppress noise from transparent regions.
Loss: The training objective combines a probabilistic Von Mises-Fisher (vMF) loss (to handle orientation ambiguity) and an edge-aware smoothness prior guided by the alpha matte.

2.4. GeoSemTransNet: Opaque Texture Synthesis

This module synthesizes the final texture-rich opaque RGB image.

Mechanism: It is an encoder-decoder network that fuses DINOv3 features via SPADE (Spatially-Adaptive Normalization) modules. The SPADE blocks modulate the activation of the texture encoder using geometric priors ( $N$ , mask, foreground) to ensure the synthesized texture adheres strictly to the object's 3D geometry.
Physics-Aware Loss: The generation is constrained by:
1. Shape-from-Shading: Ensuring luminance consistency with predicted surface normals.
2. Gradient Consistency: Aligning textural edges with geometric discontinuities.
3. Reconstruction Terms: Ensuring boundary and foreground consistency.

3. Key Contributions

TransDINO: A U-Net-based segmentation model injecting DINOv3 features for robust transparent object mask extraction, outperforming specialized models on multiple datasets.
TransDecomp: A dual-head ViT architecture that disentangles physical properties (transparency and foreground values) under weak refraction assumptions.
DAF-Net: A surface normal estimator leveraging deep semantic features for structural integrity and shallow features for local refraction/specular flow.
GeoSemTransNet: An encoder-decoder that hierarchically injects geometric priors via SPADE to synthesize geometry-guided opaque textures.
Universal Preprocessing: The framework allows existing SOTA depth estimation and reconstruction models to process transparent objects without retraining, effectively "repainting" the scene to restore photometric cues.

4. Experimental Results

The authors evaluated GHOST on the ClearGrasp, ClearPose, and Trans10K-v2 datasets.

Segmentation: TransDINO achieved state-of-the-art mIoU scores (e.g., 92.44% on ClearGrasp, 86.82% on Trans10K-v2), significantly outperforming specialized models like EBLNet and TROSNet.
Depth Estimation: When GHOST-processed images were fed into generic depth estimators (e.g., DA3, MoGe2, DepthPro, VGGT), accuracy improved drastically. For instance, using MoGe2, the RMSE dropped from 0.018 to 0.014, and $\delta_{1.25}$ accuracy improved from 0.999 to 1.000. The method outperformed specialized transparent object depth estimators (e.g., HDCNet, SwinDRNet).
3D Reconstruction: In feed-forward reconstruction tasks (using models like DUSt3R, $\pi^3$ , VGGT), GHOST-processed inputs led to significant improvements in Accuracy, Completeness, and F-Score. For example, DUSt3R's F-Score improved from 77.36 to 88.66.
Ablation Studies: Experiments confirmed that the full pipeline is necessary. Removing components (e.g., using only masks without normals or foreground cues) resulted in noise-like outputs or performance worse than the original raw images, highlighting the interdependence of the modules.

5. Significance and Claims

The paper claims that GHOST provides a universal preprocessing mechanism that resolves the "perception silo" created by specialized transparent object models. By transforming transparent regions into opaque, structurally consistent representations, GHOST enables:

Generalization: Off-the-shelf, versatile SOTA models can be applied to transparent objects without the need for task-specific retraining or expensive ground-truth depth acquisition.
Efficiency: It avoids the computational overhead of per-scene NeRF training and the hardware dependency of traditional depth completion.
Paradigm Shift: It offers a new approach to resolving transparent object bottlenecks in general 3D vision tasks by leveraging visual foundation models to hallucinate missing geometric and photometric cues.

The authors emphasize that this work advances the field of Machine Learning by demonstrating how foundation models can be integrated into a pipeline to overcome fundamental physical limitations (Lambertian assumptions) in computer vision tasks.