CoRe-GS: Coarse-to-Refined Gaussian Splatting with Semantic Object Focus

CoRe-GS is a coarse-to-refine Gaussian Splatting framework that accelerates 3D reconstruction for robotic applications by selectively optimizing only task-relevant points of interest, thereby significantly reducing training time and mitigating artifacts while maintaining high-quality semantic segmentation.

The Gist

Imagine you are a rescue drone pilot flying over a disaster zone. Your mission is urgent: you need a crystal-clear 3D map of one specific thing—maybe a collapsed building or an injured person—to guide a rescue team. You don't care about the trees, the sky, or the empty parking lots nearby; you only care about that one spot.

However, most current 3D mapping tools are like a perfectionist chef who insists on chopping every single vegetable in the kitchen, even the ones you aren't going to use, before they can serve you the soup. They try to build a perfect 3D model of the entire scene first, which takes a long time and uses up a lot of computer power. By the time they are done, the rescue team might have already waited too long.

CoRe-GS is the new, smart kitchen assistant that changes the game. Here is how it works, broken down into simple steps:

1. The "Rough Sketch" Phase (Coarse)

Instead of trying to paint a masterpiece of the whole room immediately, CoRe-GS first makes a quick, rough sketch of the entire scene. It's fast and a bit blurry, but it's enough to tell you, "Okay, the car is here, and the person is there."

• The Analogy: Think of this like using a charcoal pencil to quickly outline a drawing. You aren't worrying about the perfect shading yet; you just need to know where the important objects are.

2. The "Spotlight" Phase (Refine)

Once the rough sketch is done, the operator (you) points to the specific object you need (the "Point of Interest" or POI). CoRe-GS then turns off the lights on everything else and shines a bright spotlight only on that specific object.

• The Analogy: Imagine you are at a crowded party. Instead of trying to learn everyone's name at once, you just focus on the one person you need to talk to. CoRe-GS ignores the chatter of the crowd and zooms in to polish only that one person's face until it looks perfect.

3. The "Magic Filter" (Cleaning Up Floaters)

Here is the tricky part. When you zoom in on just one object, the computer sometimes gets confused and creates "ghosts" or "floaters"—tiny, floating specks of color that look like dust motes or glitches around the edges of your object.

• The Problem: Previous methods often left these ghosts behind, making the 3D model look messy.
• The CoRe-GS Solution: The system uses a clever color filter. It calculates a "magic color" that is the opposite of everything in the real photo. If a floating ghost appears, it usually has a weird color that doesn't match the real object. CoRe-GS spots these weird colors and instantly wipes them away, like using a magnet to pull out stray paperclips from a pile of sand.

Why Does This Matter?

• Speed: Because it stops wasting time on the background, CoRe-GS is much faster. In tests, it finished tasks in seconds that took other methods minutes or even hours.
• Quality: The final result for the specific object you care about is sharper and cleaner than if you had tried to do the whole scene at once.
• Real-World Use: This is perfect for robots, drones, and emergency responders who need to make decisions right now. They don't need a perfect map of the whole world; they need a perfect map of the problem area.

In a nutshell: CoRe-GS is like a laser-focused editor. Instead of rewriting the entire encyclopedia to fix one typo, it finds the page, zooms in, fixes the typo perfectly, and throws away the rest of the book. It saves time, saves energy, and gives you exactly what you need, when you need it.

Technical Summary

Here is a detailed technical summary of the paper "CoRe-GS: Coarse-to-Refined Gaussian Splatting with Semantic Object Focus."

1. Problem Statement

In time-critical robotic applications (e.g., disaster response, tele-guidance), operators often need to rapidly analyze specific Points of Interest (POIs) within a 3D scene rather than the entire environment.

• Inefficiency of Current Methods: Existing Semantic Gaussian Splatting (GS) approaches optimize the entire scene uniformly. This incurs substantial computational costs even when only a small subset of the scene is operationally relevant.
• The "Floater" Problem: Methods that attempt to extract specific objects after full-scene training (post-processing) or during selective refinement often suffer from outliers (floaters). These are inconsistent 3D Gaussians that appear due to mismatches between semantic masks and the optimized geometry, degrading visual coherence.
• Goal: The authors aim to create a framework that prioritizes computational resources on task-relevant POIs, achieving high-fidelity reconstruction of specific objects while drastically reducing training time and eliminating floaters.

2. Methodology: CoRe-GS

The proposed CoRe-GS (Coarse-to-Refined Gaussian Splatting) framework operates in a three-stage pipeline:

A. Initial Scene Optimization (Coarse Stage)

• Fast Initialization: The process begins with a standard, rapid RGB-only Gaussian Splatting optimization to recover coarse scene geometry.
• Lightweight Semantic Refinement: Instead of full semantic training, a late-stage semantic refinement is applied only during the final $4 \times N$iterations (where$N$ is the number of training images).
  • Gaussians are augmented with object-level feature channels.
  • These features are fine-tuned using Cross-Entropy Loss alongside the standard Novel View Synthesis (NVS) loss (SSIM and L1).
  • A $1 \times 1$ convolution projects features to semantic labels.
• Outcome: This produces a "segmentation-ready" GS representation that allows for class-consistent labeling of Gaussians without the heavy computational cost of full semantic training.

B. Point of Interest (POI) Selection

• Once the initial representation is ready, the operator selects a target POI based on a Class ID (e.g., "car" or "injured person").
• Images containing the chosen class are retained, and the system isolates Gaussians predominantly associated with that POI.
• Binary masks are generated to isolate the target during the subsequent refinement stage.

C. POI Refinement (Refined Stage)

This stage focuses exclusively on the Gaussians associated with the selected POI, employing a novel Color-Based Filtering Mechanism to prevent floaters:

1. Furthest Color Extraction:
   • The system analyzes the color distribution of the input views.
   • It constructs a reduced RGB color space and uses a KD-tree to find a "furthest color" ($p^*$) that maximizes the minimum Euclidean distance from all existing image colors.
   • This $p^*$ acts as a dedicated background rendering color.
2. Periodic Scene Filtering:
   • During refinement, the system renders the scene using $p^*$ as the background.
   • Gaussians that render with colors close to $p^*$ (indicating they are likely artifacts or background floaters associated with the mask boundaries) are identified.
   • Pruning: If the Euclidean distance between a Gaussian's rendered color and $p^*$ falls below a threshold ($d_{remove}$), the Gaussian is marked as an artifact and pruned.
   • This filtering is applied periodically (every 1,000 iterations) to ensure a clean, artifact-free POI isolation.

3. Key Contributions

1. Coarse-to-Refine Framework: A novel approach that decouples global scene understanding from local POI optimization, enabling task-driven refinement.
2. Selective POI Optimization: A strategy that isolates and optimizes only the Gaussians relevant to the selected object, significantly reducing unnecessary background computation.
3. Geometry-Preserving Color Filtering: A mechanism that suppresses segmentation-induced floaters without requiring complex mask rasterization or back-projection, using a "furthest color" heuristic to identify and remove outliers.
4. Comprehensive Evaluation: Validation across diverse indoor and outdoor datasets (NeRDS 360, SCRREAM, Tanks and Temples) demonstrating superior speed and quality.

4. Experimental Results

The authors evaluated CoRe-GS against state-of-the-art methods like Gaussian Grouping (GG), GAGA, and SAGD.

• Training Time Reduction:
  • On NeRDS 360, CoRe-GS achieved a total runtime of ~114 seconds, compared to ~1,802 seconds for GG and ~2,416 seconds for GAGA.
  • On Tanks and Temples ("Train" and "Truck"), CoRe-GS took 353 seconds, while SAGD took 674 seconds, and GG failed due to Out-of-Memory (OOM) errors.
• Reconstruction Quality (NVS):
  • CoRe-GS outperformed baselines in PSNR, SSIM, and LPIPS across all datasets.
  • On SCRREAM (Scene 02), CoRe-GS showed a +14.9 dB improvement in PSNR over GG for the "chair" POI.
  • Masked Metrics: When evaluating only the POI region, CoRe-GS achieved significantly higher masked PSNR/SSIM scores (e.g., 24.8/0.842 vs. 22.2/0.738 for GG on NeRDS 360), proving it retains high quality within the object while removing noise.
• Segmentation Quality:
  • The initial "segmentation-ready" stage achieved competitive mIoU and mBIoU on the LERF-Mask dataset using only 5,000 iterations, whereas competitors required 30k–40k iterations.
• Visual Coherence:
  • Visual comparisons (Fig. 4, Fig. 5) confirmed that CoRe-GS successfully eliminates floaters and preserves sharp object boundaries, whereas methods like GG and GAGA often retained outliers or failed mask associations.

5. Significance and Impact

• Operational Efficiency: CoRe-GS addresses the critical need for rapid situational awareness in robotics. By focusing computation only on relevant objects, it enables near-real-time 3D reconstruction for emergency scenarios.
• Robustness: The color-based filtering mechanism solves a persistent issue in semantic GS: the generation of floaters during object extraction. This leads to cleaner, more usable 3D models for downstream tasks like manipulation or tele-operation.
• Scalability: The method demonstrates that high-quality semantic understanding does not require full-scene semantic optimization, making it scalable for large, complex environments where full training is computationally prohibitive.
• Modularity: The framework is modular, allowing for the integration of different segmentation models and adaptable to various robotic platforms (drones, ground robots).

In conclusion, CoRe-GS represents a significant step forward in efficient 3D scene reconstruction, proving that task-aware, selective refinement yields faster training times and higher-quality object-level results compared to uniform semantic optimization.