💻 computer science

Semantic-guided Gaussian Splatting for High-Fidelity Underwater Scene Reconstruction

This paper proposes SWAGSplatting, a semantic-guided multimodal framework that enhances high-fidelity underwater 3D reconstruction by integrating CLIP-based semantic priors and an adaptive Gaussian reallocation strategy to overcome the challenges of spatially varying visibility and information imbalance.

Original authors: Zhuodong Jiang, Haoran Wang, Guoxi Huang, Brett Seymour, Nantheera Anantrasirichai

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhuodong Jiang, Haoran Wang, Guoxi Huang, Brett Seymour, Nantheera Anantrasirichai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a beautiful 3D photo of a sunken ship or a vibrant coral reef, but you are doing it while wearing a pair of goggles filled with thick, murky fog. The water is scattering the light, making everything look blue, blurry, and full of floating dust (marine snow).

This is the challenge of underwater 3D reconstruction. Computers usually struggle here because they rely on clear pictures to build a 3D model. When the water is murky, the computer gets confused, filling in the gaps with guesswork or leaving holes in the model.

The paper you shared introduces a new method called SWAGSplatting (a catchy name for Semantic-guided Water-scene Augmented Gaussian Splatting). Think of it as giving the computer a "smart assistant" and a "smart budget manager" to fix these blurry underwater photos.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Blurry Fog" Effect

Standard 3D computer vision works like a painter looking at a clear canvas. It sees sharp edges and bright colors, so it knows exactly where to put a brick or a leaf.

Underwater: The "canvas" is covered in fog. The computer sees a blob and doesn't know if it's a fish, a rock, or just a cloud of dirt.
The Result: Traditional methods get "lazy." They focus too much on the clear parts of the image and ignore the blurry parts, or they get confused by the noise, resulting in a 3D model that looks like a melted wax statue.

2. The Solution: SWAGSplatting

The authors added three "superpowers" to the computer's brain to solve this.

Superpower #1: The "Smart Assistant" (Semantic Guidance)

Imagine you are trying to describe a blurry shape to a friend over a bad phone connection.

Old Way: You say, "It's a gray blob." The friend guesses it's a rock.
SWAGSplatting Way: You have a Smart Assistant (powered by AI called CLIP) who looks at the whole picture and whispers, "Hey, that gray blob is actually a shipwreck."

The computer now knows the context. Even if the pixels are blurry, the AI knows, "This area is a ship, so it should be solid and structured." It forces the 3D model to respect the shape of the ship, even if the water is hiding the details. It's like drawing a sketch of a ship based on the idea of a ship, not just the blurry pixels.

Superpower #2: The "Smart Budget Manager" (Adaptive Reallocation)

Imagine you are building a 3D model using a limited number of LEGO bricks (the computer calls these "Gaussians").

Old Way: The computer sees a clear, bright rock and dumps 100 bricks on it. It sees a dark, blurry fish and only puts 2 bricks on it. The rock looks perfect, but the fish looks like a flat, pixelated mess.
SWAGSplatting Way: The computer acts like a Smart Budget Manager. It realizes, "Wait, I have too many bricks on the clear rock, and not enough on the blurry fish!"
- It takes some bricks away from the easy parts (where we already have enough detail).
- It moves those bricks to the hard, blurry parts (where we need them most).
- Result: The whole model gets a balanced upgrade. The blurry fish suddenly gets enough bricks to look like a real fish.

Superpower #3: The "Two-Stage Training" (Stage-wise Optimization)

Imagine learning to play a song on the piano.

Old Way: You try to hit every single note perfectly while also getting the rhythm right, all at once. You get overwhelmed and make mistakes.
SWAGSplatting Way: It uses a Two-Stage Plan:
1. Stage 1 (The Rough Sketch): The computer focuses only on getting the shape right. "Where is the ship? Where is the rock?" It ignores the tiny details and colors for now.
2. Stage 2 (The Polish): Once the shape is solid, the computer locks the shape in place and focuses entirely on the colors and fine details. "Okay, now let's make the coral look red and the sand look sandy."
- Result: The model doesn't get confused trying to do everything at once. It builds a strong foundation first, then paints the masterpiece.

The Final Result

When you put all these tools together, the computer can look at a murky, dark, noisy underwater video and produce a crystal-clear 3D model.

Without SWAGSplatting: The 3D model looks like a foggy, distorted ghost of the scene.
With SWAGSplatting: The 3D model looks sharp, the shipwrecks are solid, the coral reefs are detailed, and the colors are true to life, even though the original video was terrible.

Why Does This Matter?

This isn't just about making pretty pictures.

Archaeologists can explore shipwrecks without diving.
Robots can navigate underwater caves safely.
Biologists can study coral reefs without touching them.

In short, SWAGSplatting teaches computers to "see" through the water by using context clues (what the object is) and smart resource management (putting effort where it's needed most), turning murky underwater chaos into high-definition 3D reality.

1. Problem Statement

Underwater 3D reconstruction faces unique challenges due to the physical properties of water, including light scattering, wavelength-dependent attenuation, and backscatter. These factors cause:

Spatially Non-Uniform Visibility: Foreground objects often have clearer signals than background regions, which are obscured by haze and scattering.
Imbalanced Representation: Existing 3D Gaussian Splatting (3DGS) methods rely on gradient-driven densification based on photometric signals. In degraded underwater conditions, this leads to an imbalance where well-observed regions accumulate redundant Gaussians (overfitting), while low-visibility or degraded areas remain under-represented, resulting in blurred structures and geometric inconsistencies.
Lack of High-Level Guidance: Current methods primarily optimize based on low-level pixel differences, lacking semantic priors to guide the reconstruction of salient objects (e.g., shipwrecks, coral) amidst noise.

2. Methodology: SWAGSplatting

The authors propose SWAGSplatting (Semantic-guided Water-scene Augmented Gaussian Splatting), a multimodal framework that integrates semantic priors and adaptive resource allocation into the 3DGS pipeline.

A. Semantic-Guided Gaussians

Augmented Primitives: Each 3D Gaussian primitive is extended with a learnable semantic feature vector ( $f_s$ ).
Semantic Supervision: The framework uses vision-language models (BLIP3-o for captioning and Grounded-SAM for segmentation) to identify regions of interest in input images. These regions are encoded using CLIP to generate reference semantic embeddings.
Semantic Consistency Loss ( $L_s$ ): A loss function is introduced to align the learnable semantic features of Gaussians within a specific region to the corresponding CLIP embedding. This acts as an auxiliary regularizer, enforcing object-level coherence and preventing the allocation of density to semantically incoherent noise (e.g., floating particles) without altering geometric parameters directly.

B. Adaptive Gaussian Primitive Reallocation

Problem Addressed: Standard 3DGS densification creates an imbalance where high-signal regions get too many Gaussians, while low-signal regions get too few.
Mechanism: The method introduces a dynamic reallocation strategy that operates periodically during training:
1. Importance Scoring: Calculates an importance score ( $\tilde{S}_i$ ) based on the gradient contribution of each Gaussian to the rendered image.
2. Error Scoring: Calculates an error score ( $\tilde{E}_i$ ) based on the contribution of each Gaussian to the reconstruction loss.
3. Pruning and Cloning: The bottom 10% of primitives (lowest importance) are pruned. Their "budget" is reallocated to the top 10% of primitives (highest error) via cloning.
Result: This redistributes representation capacity from redundant regions to areas with high reconstruction error, ensuring balanced spatial coverage without increasing the total number of primitives.

C. Stage-wise Optimization Strategy

To ensure stability and high fidelity, the training follows a coarse-to-fine two-stage approach:

Stage 1 (Reconstruction): Focuses on global geometry and appearance. Uses an $L_1$ -based reconstruction loss with SSIM, depth supervision, and semantic alignment.
Stage 2 (Refinement): Freezes geometric and semantic parameters. Focuses purely on appearance refinement using an $L_2$ loss to enhance fine details and color consistency, while disabling semantic and depth regularization.

D. Loss Functions

The total objective function combines standard reconstruction losses with new terms:

$L_s$ : Semantic consistency loss.
$L_2$ : Pixel-wise MSE for fine detail refinement (weighted higher in Stage 2).
$L_h$ (Hinge Loss): A weak regularization enforcing the physical ordering of attenuation coefficients ( $\beta_{red} > \beta_{green} > \beta_{blue}$ ) to reflect wavelength-dependent absorption.

3. Key Contributions

Semantic-Augmented 3DGS: The first framework to integrate learnable semantic features into 3D Gaussian primitives for underwater scenes, using CLIP embeddings to enforce object-aware regularization.
Adaptive Reallocation: A novel strategy to rebalance Gaussian distribution by pruning low-importance primitives and cloning high-error ones, mitigating the imbalance caused by gradient-driven densification in degraded environments.
Stage-wise Optimization: A training scheme that decouples global structure learning from fine-grained appearance refinement, improving convergence stability.
Comprehensive Evaluation: Extensive testing on real-world datasets demonstrating state-of-the-art performance.

4. Experimental Results

The method was evaluated on three public underwater datasets: SeaThru-NeRF, Submerged3D, and S-UW.

Quantitative Performance: SWAGSplatting outperformed six state-of-the-art baselines (including 3DGS, UW-GS, RUSplatting, and SeaThru-NeRF) across all metrics (PSNR, SSIM, LPIPS).
- On Submerged3D (sparse views, severe degradation), it achieved an average 1.80 dB PSNR gain over UW-GS.
- On SeaThru-NeRF, it achieved an average 0.67 dB PSNR gain over the best baseline (RUSplatting).
Qualitative Performance: Visual comparisons show that SWAGSplatting preserves fine structural details (e.g., coral spines, shipwreck textures) and reduces artifacts (blurring, color distortion) significantly better than competitors.
Ablation Study: Removing any component (Semantic Guidance, Stage-wise Optimization, or Adaptive Reallocation) resulted in performance degradation, confirming the necessity of each module. The semantic guidance component provided the most significant individual boost.

5. Significance

This work addresses a critical bottleneck in underwater photogrammetry: the inability of standard neural rendering methods to handle spatially varying visibility and semantic ambiguity. By integrating high-level semantic priors and adaptive resource management, SWAGSplatting enables:

Robust Reconstruction: High-fidelity 3D models in turbid, low-light, and sparse-view conditions where traditional methods fail.
Efficiency: Maintaining real-time rendering capabilities while improving reconstruction quality through smarter primitive distribution rather than brute-force densification.
Application Impact: The method supports critical applications in marine ecology, underwater archaeology, and robotics by providing accurate, semantically consistent 3D visualizations of submerged environments.

The code is made available to the community to further advance underwater 3D reconstruction research.