GS4City: Hierarchical Semantic Gaussian Splatting via City-Model Priors

1. Problem Statement

Recent advancements in 3D Gaussian Splatting (3DGS) have enabled high-fidelity, real-time rendering of large-scale scenes. However, extending 3DGS to semantic urban understanding faces significant challenges:

• Limitations of 2D Foundation Models: Current semantic 3DGS methods rely heavily on 2D foundation models (e.g., CLIP, GroundingDINO). While effective for generic object discovery, they struggle with urban facades characterized by repetitive patterns, reflective materials, and strong occlusions. This leads to ambiguous boundaries and a failure to preserve the hierarchical part-whole structure of buildings (e.g., distinguishing windows from walls, or walls from the whole building).
• Representation Mismatch: Structured 3D city models (like CityGML) contain rich, hierarchical semantic data (LoD3) with explicit parent-child relations (Building $\to$ Wall $\to$ Window). However, these models define semantics on discrete polygonal surfaces, whereas 3DGS represents scenes using alpha-composited Gaussian primitives without explicit surface ownership. Directly transferring labels from CityGML to Gaussians is unreliable due to geometric misalignment and floating artifacts.

Goal: To create a 3DGS representation that is not only photorealistic but also hierarchical, geometry-aligned, and queryable, effectively bridging the gap between structured city models and unstructured Gaussian scenes.

---

2. Methodology: GS4City

GS4City introduces a pipeline that incorporates CityGML priors to supervise the learning of semantic Gaussian representations. The framework operates in three main stages (illustrated in Figure 2 of the paper):

A. City-Model Semantic Extraction (Two-Pass Raycasting)

To generate reliable supervision masks from the LoD3 CityGML model:

1. Preprocessing: The CityGML hierarchy is flattened into a semantic table where each triangle face inherits IDs for the Feature (Building), Surface (Wall/Roof), and Part (Window/Door) levels.
2. Global Pass: Standard raycasting projects the mesh to generate initial masks.
3. Part Pass (Recovery): Single-pass raycasting often fails to detect recessed elements (e.g., windows inside a wall) because the wall surface occludes them. GS4City performs a second pass specifically on part-level geometries.
   • It checks if a part-level hit is consistent with the global pass (i.e., the parent surface ID matches and the depth difference is within a tolerance $\tau$).
   • If consistent, the fine-grained part ID overrides the surface ID.
   • Result: A high-fidelity, hierarchical instance map ($M_{city}$) that preserves the building structure and recovers fine details like windows and doors.

B. Mask Fusion and Instance Linking

Since CityGML only covers buildings, the method must integrate non-building objects (trees, cars, people) using foundation models.

1. Foundation Model Filtering: Raw masks from 2D foundation models are filtered. Masks overlapping significantly with the CityGML building region are disambiguated using a vision-language model to distinguish between "building parts" and "foreground objects."
2. Cross-View Association: To ensure scene consistency, image-based masks are linked across views. The pre-trained 3DGS scene acts as a geometric bridge. Masks are merged into global groups if they share high geometric overlap with existing Gaussian clusters and have semantically similar vision-language features.
3. Fusion: The CityGML-derived masks ($M_{city}$) and the cleaned foundation-model masks ($M_{img}$) are fused.
   • Priority Rule: Image-based masks take precedence for non-building regions, while CityGML masks dominate building regions.
   • Feature Aggregation: Each fused instance is assigned a vision-language feature vector, enabling open-vocabulary querying.

C. Gaussian Identity Learning

The fused 2D supervision is lifted into the 3D Gaussian field.

1. Identity Encoding: Each Gaussian primitive $g_j$ is augmented with a learnable identity code $e_j$.
2. Rendering: These codes are rendered using the standard alpha-compositing rule to produce feature maps $E(u)$.
3. Loss Functions:
   • 2D Identity Supervision ($L_{2D}$): Cross-entropy loss between the rendered identity logits and the fused 2D mask $M_t$.
   • 3D Spatial Regularization ($L_{3D}$): To prevent weakly observed Gaussians from drifting, a consistency loss is applied. It penalizes disagreements between a Gaussian's predicted class distribution and those of its $k$-nearest neighbors in 3D space.
4. Outcome: The trained Gaussians possess a compact identity encoding that links them to the CityGML semantic hierarchy (Feature $\to$ Surface $\to$ Part) and associated metadata.

---

3. Key Contributions

1. GS4City Framework: A novel method for hierarchical semantic 3DGS that integrates structured city-model priors (CityGML LoD3) to overcome the limitations of purely 2D-driven approaches.
2. Robust Supervision Pipeline: A Two-Pass Raycasting mechanism that explicitly recovers fine-grained facade elements (windows/doors) often lost in single-pass projections, combined with a Multi-Source Mask Fusion strategy to handle the full scene (buildings + non-buildings).
3. Semantic Lifting: A learning strategy using compact Gaussian identity encoding with joint 2D supervision and 3D spatial regularization. This enables accurate coarse segmentation (Building vs. Non-Building) and fine-grained semantic parsing (Walls, Roofs, Windows) while supporting open-vocabulary queries.
4. Metadata Integration: The method preserves the link between Gaussian primitives and the original CityGML attributes, enabling structured urban information retrieval.

---

4. Experimental Results

The method was evaluated on two datasets: TUM2TWIN and Gold Coast LoD3.

• Coarse Segmentation (Building vs. Non-Building):
  • GS4City significantly outperformed baselines (LangSplat, Gaga).
  • IoU Gains: Achieved 93.1% on TUM2TWIN and 95.4% on Gold Coast, surpassing the next best baseline by up to 15.8 IoU points.
• Fine-Grained Segmentation:
  • GS4City achieved the highest mean Intersection over Union (mIoU) across both datasets.
  • Improvements: +8.2 mIoU on TUM2TWIN (30.5% vs 22.3%) and +14.2 mIoU on Gold Coast (44.4% vs 30.2%).
  • Specific Categories: The largest gains were observed in structured building elements (Roof, Wall, Window, Door) and cars, where the CityGML priors provided strong geometric constraints.
• Qualitative Analysis:
  • GS4City produced sharper boundaries and correctly isolated windows and doors, whereas baselines often merged these into walls or produced noisy, fragmented predictions.
  • It successfully avoided assigning building labels to non-building regions (e.g., water surfaces).
• Efficiency: An ablation study on the cross-view association threshold showed that GS4City achieves the best accuracy-efficiency trade-off (time-normalized mIoU) compared to baselines.

---

5. Significance and Impact

• Bridging the Gap: GS4City successfully bridges the gap between rigorous, structured city models (CityGML) and photorealistic, unstructured 3DGS representations.
• Urban Digital Twins: By preserving the hierarchical part-whole relationships (Building $\to$ Surface $\to$ Part) and integrating metadata, GS4City enables semantically queryable and structure-aware urban reconstructions. This is crucial for applications like city-scale asset management and digital twins.
• Beyond Appearance: The work demonstrates that 3DGS can evolve from purely appearance-centered rendering to a representation capable of hierarchical urban understanding, offering a new paradigm for large-scale semantic scene reconstruction.

Code Availability: The authors have released the code at https://github.com/Jinyzzz/GS4City.