💻 computer science

First Shape, Then Meaning: Efficient Geometry and Semantics Learning for Indoor Reconstruction

The paper introduces FSTM, a unified two-step framework that first optimizes geometry using RGB and geometric cues before estimating semantics, thereby achieving faster training, improved robustness, and higher recall in indoor 3D reconstruction compared to existing multi-SDF methods.

Original authors: Remi Chierchia, Léo Lebrat, David Ahmedt-Aristizabal, Olivier Salvado, Clinton Fookes, Rodrigo Santa Cruz

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Remi Chierchia, Léo Lebrat, David Ahmedt-Aristizabal, Olivier Salvado, Clinton Fookes, Rodrigo Santa Cruz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect 3D model of a messy living room using only a stack of 2D photos. You want the model to not only look like the room (the geometry) but also know exactly which object is a chair, which is a lamp, and which is a book (the semantics).

This paper introduces a new method called FSTM (First Shape, Then Meaning) that solves a major problem in how computers currently do this: they usually try to learn the shape and the labels at the same time, which confuses the computer and makes the training slow and messy.

Here is how FSTM works, explained through simple analogies:

The Problem: Trying to Learn Two Things at Once

Think of the old way of doing this (called "Multi-SDF") like trying to teach a student to play the piano and solve advanced math equations simultaneously, starting on day one.

The Confusion: The student gets overwhelmed. They might learn to press the right keys but forget the math, or vice versa.
The Slowdown: In the computer world, this means the system has to run a separate "math engine" for every single object in the room. If there are 50 objects, the computer has to run 50 engines at once. This is incredibly slow and causes the system to crash or give up on small details (like the legs of a chair).

The Solution: FSTM's Two-Step Approach

The authors propose a smarter strategy: First Shape, Then Meaning.

Step 1: The "Warm-Up" (Geometry Only)

Imagine you are sculpting a lump of clay. Before you paint it or tell anyone what it is, you focus entirely on getting the shape right.

In this phase, the computer looks at the photos and learns the 3D structure of the room. It ignores the labels (it doesn't care if something is a chair or a table yet).
It uses the photos and some "guesses" about depth and angles to build a smooth, accurate skeleton of the scene.
Why this helps: By focusing only on the shape first, the computer builds a solid foundation without getting distracted by the confusing task of labeling.

Step 2: The "Painting" (Adding Meaning)

Once the clay sculpture is perfectly formed, you now pick up the paintbrush.

Now that the shape is stable, the computer starts learning the semantics (the labels). It looks at the 2D photos again to figure out which parts of the clay are "chairs" and which are "lamps."
Because the shape is already solid, the labels snap into place perfectly. The computer can now "read" the 3D model and say, "Ah, this specific curve is definitely a chair leg."

Why This is Better

The paper claims this method is like a master chef who first perfects the dough before adding the toppings, rather than trying to knead the dough and sprinkle the cheese at the exact same second.

It's Faster: Because the computer doesn't have to run 50 separate engines for 50 objects, it runs one efficient engine for the whole room. The paper says it trains 2.3 times faster than previous methods.
It's More Accurate: The old methods often gave up on small objects or made them look like blurry blobs. FSTM recovers tiny details (like the thin legs of a chair) that others miss.
It Handles Clutter Better: In a room full of many objects, the old methods get confused and lose track of things. FSTM keeps track of almost everything, recovering up to six times more objects in complex scenes.

The Result

The final output is a 3D model that is not only geometrically precise (the walls are straight, the chairs have real legs) but also semantically rich (you can click on the chair and the computer knows it's a chair).

The authors tested this on both fake, perfect computer-generated rooms and real, messy photos from actual buildings. In both cases, FSTM built better, faster, and more detailed 3D worlds than the previous state-of-the-art methods.

In short: Don't try to learn the map and the street names at the same time. First, draw the map perfectly. Then, add the names. That is the secret to building better 3D worlds.

Technical Summary: FSTM – Efficient Geometry and Semantics Learning for Indoor Reconstruction

Problem Statement

Neural Surface Reconstruction, particularly using Signed Distance Functions (SDFs), has become a standard for indoor 3D reconstruction. However, many downstream applications (robotics, editing, interaction) require not only geometric fidelity but also per-object semantic understanding. Existing approaches that integrate semantics often rely on multi-SDF formulations (e.g., ObjectSDF++, RICO), where each object is represented by a separate SDF. These methods suffer from three primary limitations:

Computational Overhead: Regularization terms (Eikonal, smoothness) must be computed for every object SDF, causing training time to scale poorly with scene complexity.
Optimization Instability: The use of min-pooling to merge object SDFs into a scene representation introduces gradient discontinuities, leading to unstable training and slow convergence.
Poor Scalability in Cluttered Scenes: Multi-SDF architectures struggle to maintain detail and recall in complex, multi-object environments, often missing small objects or generating oversimplified geometry in under-observed regions.

Furthermore, naive joint optimization of geometry and semantics in a single network often degrades reconstruction quality, as early semantic supervision can bias the geometry network toward semantic consistency at the expense of geometric accuracy.

Methodology: FSTM

The authors propose FSTM (First Shape, Then Meaning), a unified framework that learns geometry and semantics through a single global SDF using a two-step training strategy.

1. Unified Representation

Unlike multi-SDF approaches, FSTM represents the entire scene with a single learnable SDF ( $SDF_\theta$ ). To enable semantic estimation, the architecture extends the MonoSDF backbone with a semantic decoder head. For a 3D query point $x$ , the geometry MLP outputs an SDF value and a feature vector $g(x)$ , which is mapped to semantic logits $s_C(x)$ . These distributions are integrated along camera rays via volume rendering to enable dense per-ray semantic supervision.

2. Two-Step Training Strategy

To address the conflict between geometric and semantic optimization, FSTM employs a staged approach:

Step 1: Geometry Warm-up: The network is trained exclusively on geometric cues (RGB, monocular depth, and normal priors) with semantic supervision disabled ( $\lambda_5 = 0$ ). This ensures the geometry converges to a high-fidelity state without being biased by potentially noisy or conflicting semantic signals.
Step 2: Joint Optimization: Semantic supervision is introduced, and the network is jointly optimized for both geometry and semantics for the remainder of the training budget.

3. 3D Segmentation

At test time, object-level segmentation is achieved via a lightweight post-processing step. An iso-surface extraction (e.g., Marching Cubes) generates a mesh from the zero-level set of the single SDF. For each mesh face, a ray is cast from the face center along the normal direction to query the learned semantic field. The face is assigned the class with the highest integrated probability, enabling dense, object-level labeling without requiring separate object-specific networks.

Key Contributions

Single-SDF Formulation: Demonstrates that a single global SDF, rather than a complex multi-SDF architecture, is sufficient for high-quality, object-aware reconstruction. This avoids gradient discontinuities and reduces computational overhead.
Two-Step Optimization: Introduces a geometry-first training schedule that prevents the destructive interference of early semantic gradients, leading to more stable convergence and higher quality reconstructions compared to joint optimization.
Efficient Semantic Integration: Shows that semantic priors can actively refine geometric fidelity when properly scheduled, rather than merely acting as a byproduct of reconstruction.
Scalability: The method scales efficiently with scene complexity, avoiding the linear increase in training time associated with the number of objects in multi-SDF designs.

Experimental Results

The method was evaluated on Replica (synthetic) and ScanNet++ (real-world) datasets.

Performance on Replica: FSTM outperforms state-of-the-art methods (ObjectSDF++, RICO, MonoSDF) in both scene-level and object-level metrics (Chamfer Distance, F-Score, Normal Consistency). It achieves a 2.3× speedup in training time compared to multi-SDF baselines.
Performance on ScanNet++: FSTM demonstrates superior robustness to real-world imperfections (blurry frames, pose errors). It recovers significantly more objects than baselines (up to 6× more in dense scenes) and maintains higher recall.
Qualitative Improvements: FSTM recovers fine geometric details (e.g., chair legs) and small objects that multi-SDF methods often miss or oversimplify.
Limited Supervision: Under sparse semantic supervision, FSTM maintains stable scene-level reconstruction quality, whereas naive joint optimization and multi-SDF methods degrade significantly.

Significance and Claims

The paper claims that FSTM establishes a streamlined foundation for scalable, semantically informed 3D scene understanding. By decoupling geometry and semantics learning, the authors show that:

Semantics enhance geometry: Properly integrated semantic supervision improves geometric fidelity, particularly in ambiguous or under-observed regions.
Simplicity is effective: Complex multi-object coordination is not necessary for high-quality, object-aware reconstruction; a single SDF with a structured training schedule is sufficient.
Efficiency: The approach offers a practical trade-off, sacrificing the real-time inference of 3D Gaussian Splatting methods to achieve the high-fidelity, topologically correct surfaces required for industrial inspection and surgical planning, while remaining significantly faster and more robust than existing multi-SDF neural surface methods.

The authors acknowledge limitations, noting that the method does not natively produce watertight meshes for individual objects (requiring post-processing for applications like 3D printing) and that geometric improvements diminish when 2D semantic annotations are extremely scarce (e.g., fewer than 10 views).