💻 computer science

VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching

VolFill is a generative framework that leverages a hybrid 3D VAE and latent Diffusion Transformer to reconstruct complete 3D scene geometries, including hidden structures, from a single RGB image by utilizing geometry foundation models and volumetric flow matching.

Original authors: Tuan Duc Ngo, Chuang Gan, Evangelos Kalogerakis

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tuan Duc Ngo, Chuang Gan, Evangelos Kalogerakis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a room through a keyhole. You can see the front of a sofa, a coffee table, and a lamp. But you can't see the back of the sofa, the wall behind the table, or the floor underneath the lamp. Most computer programs trying to "see" this room can only draw what is directly in front of the keyhole. If they try to guess the rest, they often end up drawing messy, floating dots or leaving big holes in the picture.

VolFill is a new computer program designed to solve this problem. It doesn't just guess what's hidden; it builds a complete, solid 3D model of the entire room, including the parts you can't see, from just a single photo.

Here is how it works, using some simple analogies:

1. The Problem: The "Pixel-Aligned" Trap

Think of most current 3D scanners like a painter who is only allowed to paint on the canvas where the light hits. If you look at a chair, they can paint the front legs perfectly, but the back legs? They just leave them blank or try to guess, often resulting in a wobbly, broken shape. They are stuck to the "visible surface" and can't imagine the rest of the object.

2. The Solution: The "Invisible Ink" Map (TUDF)

Instead of trying to guess individual points (like a cloud of dust), VolFill thinks of the room as a giant 3D grid of tiny cubes, like a massive block of Jell-O.

The Trick: It uses a special type of map called a TUDF. Imagine this map is like a "distance-to-surface" sensor. Every single cube in the grid knows exactly how far it is from the nearest wall, floor, or piece of furniture.
Why it helps: Even if a wall is hidden behind a sofa, the cubes behind the sofa still know how far they are from that hidden wall. This allows the computer to "fill in" the invisible parts perfectly, creating a solid, continuous shape rather than a scattered cloud of dots.

3. The Engine: The "Smart Compressor" and the "Dreamer"

To make this work, VolFill uses two main tools working together:

The Smart Compressor (Hybrid 3D VAE):
A full 3D grid of a whole room is huge and would crash a computer. VolFill uses a "compressor" to shrink this massive grid down into a tiny, compact summary (a latent space).
- Analogy: Imagine taking a giant, detailed blueprint of a city, folding it up until it fits in your pocket, but keeping all the important details intact so you can unfold it later. This compressor is smart enough to know that empty air doesn't need much detail, so it focuses its memory on the furniture and walls.
The Dreamer (Diffusion Transformer):
Once the grid is compressed, VolFill uses a "Dreamer" to fill in the missing parts.
- Analogy: Imagine you have a sketch of a room, but half of it is erased. The Dreamer is an artist who looks at the visible part and says, "Okay, based on how rooms usually look, the hidden part must look like this." It doesn't just guess randomly; it follows the "rules" of how 3D objects fit together.

4. The Guide: The "Double-Check" System

To make sure the Dreamer doesn't hallucinate crazy shapes (like a floating ceiling), VolFill uses a Dual-Conditioning strategy. It acts like a detective with two sources of evidence:

The Photo: It looks at the high-level "vibe" of the image (is it a kitchen? a bedroom?).
The Visible Geometry: It uses a pre-trained "expert" (called MoGe2) to get a precise map of exactly what is visible.
- Analogy: The Dreamer is the artist, but the "Visible Geometry" is a strict supervisor holding a ruler. The supervisor says, "You can imagine the hidden back of the sofa, but you must make sure it connects perfectly to the front legs we can actually see." This keeps the hidden parts physically realistic.

5. The Result: A Solid, Clean Model

When VolFill finishes its work, it doesn't give you a messy cloud of points. Because it used the "distance map" (TUDF) method, it can instantly turn that grid into a smooth, solid mesh (like a 3D printed object).

Comparison: Other methods might give you a sofa that looks like a bag of marbles (noisy points) or a sofa with a ghostly second layer behind it (artifacts). VolFill gives you a clean, sharp, solid sofa where the hidden back is just as smooth as the front.

In short: VolFill takes a single photo, compresses the world into a smart summary, uses an AI "dreamer" guided by strict geometric rules to imagine the hidden parts, and then unfolds it into a perfect, solid 3D room that includes everything you can't see.

Technical Summary: VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching

1. Problem Formulation

The paper addresses the challenge of amodal 3D scene reconstruction from a single RGB image. Unlike traditional pixel-aligned methods that recover only visible surfaces (e.g., depth maps or pointmaps), the goal is to predict the complete 3D geometry within the observed view frustum, including surfaces occluded by foreground objects.

This task is inherently ill-posed due to the lack of visual evidence for hidden structures. Existing approaches face significant limitations:

Pixel-aligned methods (e.g., MoGe2, DepthPro) are restricted to visible surfaces and cannot infer occluded regions.
Optimization-based methods require dense multi-view captures and hours of per-scene computation.
Object-centric generative models struggle with complex, unbounded scenes.
Scene-level amodal baselines (e.g., LaRI, NOVA3R) often produce fragmented meshes, layered artifacts ("multiple wall" effects), or noisy, disintegrated point clouds due to unstructured representations or fixed ray-layer constraints.

2. Methodology

VolFill is a generative framework that predicts a complete Truncated Unsigned Distance Function (TUDF) voxel grid from a single image, rather than relying on deterministic regression. The architecture consists of three core components:

A. TUDF Representation

The scene geometry is represented as a continuous scalar field $V(p)$ , defined as the distance to the nearest surface (including occluded ones), truncated to a maximum value $\tau$ .

Unsigned Formulation: An unsigned distance function is used because indoor scenes are often non-watertight (e.g., single-sided walls), making signed distance functions (SDFs) geometrically ambiguous.
Dynamic Discretization: The method employs a frustum-aligned discretization strategy that computes a dynamic, isotropic voxel size based on the visible geometry's bounding box, ensuring the grid focuses on the immediate captured scene.

B. Hybrid 3D VAE

To bridge the gap between high-resolution TUDF grids ( $256^3$ ) and the latent space required for generative modeling, a Hybrid Sparse-Dense 3D VAE is proposed:

Encoder: Processes sparse TUDF inputs using sparse 3D convolutions to compress the grid by $16\times$ into a compact, dense latent space ( $16^3 \times C$ ).
Decoder: Utilizes a hybrid dense-to-sparse decoding schedule:
1. Dense Upsampling: Transposed convolutions upsample the latent to $64^3$ .
2. Structure Prediction: A binary occupancy head predicts a mask to sparsify the feature map, discarding empty regions.
3. Sparse Upsampling: Sparse convolutions upsample the features to the final $256^3$ resolution.
Loss Function: The VAE is trained with a composite loss including L1 distance for active voxels, BCE and Dice losses for occupancy, and KL-divergence for latent regularization.

C. Latent Diffusion Transformer with Flow Matching

The generative core is a Diffusion Transformer (DiT) trained with a rectified flow matching objective. It learns to transport noise toward the distribution of scene TUDF latents.

Dual-Conditioning Strategy: To ensure robust generalization and physical plausibility, the DiT is conditioned on two sources:
1. Global Geometric Priors: Features from a frozen geometry foundation model (MoGe2) are injected via cross-attention to provide high-level spatial context.
2. Visible-Latent Inpainting: The visible point map from MoGe2 is encoded into a "visible latent" ( $z_{vis}$ ) and fused with the noisy latent via a zero-initialized MLP. This anchors the generation to observed geometry while guiding the completion of occluded regions.
Inference: The model integrates the learned ODE from $t=0$ to $t=1$ using Euler steps, followed by decoding through the VAE to produce the final TUDF grid.

3. Key Contributions

VolFill Framework: A generative framework utilizing volumetric flow matching to recover complete scene-level geometry from single-view images, moving beyond pixel-aligned regression.
Hybrid 3D VAE: An architecture enabling efficient spatial compression of high-resolution TUDF grids into a compact latent space, facilitating high-fidelity reconstruction of complex amodal structures.
Dual-Conditioning Strategy: A novel approach leveraging geometry foundation models (MoGe2) to integrate high-level image tokens with explicit visible geometry, grounding amodal reasoning in observed data.

4. Experimental Results

The method was evaluated on the SCRREAM and NRGB-D datasets, outperforming current baselines (including LaRI, NOVA3R, and object-centric models like TRELLIS and TripoSG).

Quantitative Performance: VolFill achieves state-of-the-art results in Chamfer Distance (CD) and F-score (FS) metrics, particularly for occluded and complete geometry. On SCRREAM, it achieves a CD of 3.03 and FS0.02 of 54.83 for the complete scene, significantly outperforming NOVA3R (CD 3.43, FS 45.12) and LaRI (CD 5.19, FS 30.85).
Metric Scale: In metric-scale evaluations, VolFill significantly outperforms baselines in F-score (44.90 vs. 22.23 for NOVA3R), demonstrating faithful metric-scale reconstruction.
Qualitative Quality: Visual comparisons show VolFill produces sharp, high-fidelity geometry with smooth, topologically consistent meshes. In contrast, LaRI suffers from layered artifacts and holes, while NOVA3R yields noisy, unstructured point scatters.
Ablation Studies:
- The Hybrid VAE design offers the best balance of reconstruction quality, latency, and memory usage compared to dense-only or sparse-only decoders.
- Dual-conditioning (combining MoGe2 features and visible latents) significantly outperforms single-conditioning strategies.
- MoGe2 is identified as the superior foundation model for conditioning compared to DINOv2 or VGGT, highlighting the importance of explicit 3D spatial anchors over general semantic features.

5. Significance and Claims

The paper claims that VolFill provides a robust foundation for holistic spatial understanding by shifting reconstruction from deterministic, pixel-aligned estimation to a structured generative manifold.

Structured Representation: By predicting a TUDF grid rather than unstructured point clouds, VolFill enables direct surface extraction via isosurfacing, resulting in clean, artifact-free meshes without the need for noisy post-processing (e.g., Poisson reconstruction).
Scalability: The volumetric approach scales well with scene complexity and imposes no constraints on the number of recoverable occluded layers, unlike ray-based methods.
Physical Plausibility: The generative framework, guided by strong geometric priors, synthesizes physically plausible completions for under-determined regions where visual evidence is absent.

Limitations: The authors acknowledge that small holes may occasionally persist in unobserved regions due to the lack of visual evidence, and inference remains slower (1.4s on RTX 4090) than regression-based baselines due to the iterative generative process. Future work is suggested to address topological gaps through increased model capacity and data diversity.