MOGS: Monocular Object-guided Gaussian Splatting in Large Scenes

MOGS is a monocular 3D Gaussian Splatting framework for large scenes that replaces costly LiDAR with object-anchored metric depth derived from sparse visual-inertial cues, significantly reducing training time and memory consumption while achieving rendering quality competitive with LiDAR-based approaches.

The Gist

Imagine you are trying to build a perfect, photorealistic 3D model of a whole city using only a single camera on a car. This is the dream of 3D Gaussian Splatting (3DGS), a technology that lets computers render 3D scenes so fast and realistic you can't tell the difference from reality.

However, there's a big problem: Scale and Cost.

The Problem: The "Expensive LiDAR" vs. The "Blind Camera"

Currently, the best way to build these huge 3D city models is to use LiDAR. Think of LiDAR as a super-expensive, high-tech laser scanner that shoots millions of laser beams to measure exact distances. It's like having a blind person with a magical cane that instantly knows exactly how far every wall and car is.

• The Catch: These scanners cost thousands of dollars, are heavy, and create so much data that they clog up computer memory and slow everything down. It's like trying to fill a swimming pool with a firehose; you get the water, but you might flood the house.

On the other hand, a regular monocular camera (like the one on your phone) is cheap and everywhere. But it's "blind" to depth. It sees a flat picture and has to guess how far away things are. In a small room, it's okay. But in a massive city scene, it gets confused, leading to a wobbly, distorted 3D model where cars float in the air or roads twist into impossible shapes.

The Solution: MOGS (The "Object Detective")

The authors of this paper created MOGS (Monocular Object-guided Gaussian Splatting). Think of MOGS as a clever detective that solves the "blind camera" problem without needing the expensive laser scanner.

Here is how it works, using a simple analogy:

1. The "Shape Guessing Game" (Multi-scale Shape Consensus)

Imagine you are looking at a photo of a city. You see a blurry patch that might be a bus, a flat road, or a round water tower.

• The Old Way: The computer tries to guess the distance of every single pixel individually. It's like trying to count every grain of sand on a beach to figure out the beach's shape. It's slow and prone to errors.
• The MOGS Way: MOGS says, "Wait, I recognize that shape! That's a bus."
  • It uses AI to identify objects (cars, buildings, roads).
  • It knows that a bus is usually a box, a road is a flat plane, and a water tower is a cylinder.
  • It finds just a few reliable distance points (from a cheap sensor) on the edge of the bus.
  • The Magic: Once it knows the bus is a "box" and has a few points on the edge, it can mathematically "fill in the blanks" for the rest of the bus. It assumes the whole object follows that shape. This turns a few blurry guesses into a solid, reliable 3D shape.

2. The "Group Hug" (Cross-object Depth Refinement)

Even with shape guesses, things can get messy. Maybe the computer thinks the bus is floating 10 feet above the road because it misjudged the scale.

• The MOGS Fix: MOGS acts like a strict project manager. It looks at all the objects together and says, "Hey, buses don't float! They sit on the road. And that building must be parallel to the street."
• It uses a second AI (called a "Foundation Model") that is really good at guessing relative depth (which things are closer than others, even if the exact distance is wrong).
• MOGS combines the exact shape of the bus (from step 1) with the relative layout of the city (from step 2). It smooths out the errors, ensuring the bus sits firmly on the ground and the buildings line up correctly.

The Result: Fast, Cheap, and Real

By using this "Object Detective" approach, MOGS achieves three amazing things:

1. It's Cheap: It doesn't need the expensive laser scanner. A standard camera and a small motion sensor are enough.
2. It's Fast: Because it builds the model based on "objects" (like a whole bus) rather than millions of individual dots, it trains 30% faster.
3. It's Memory Efficient: It uses 20% less computer memory, meaning you can run this on cheaper hardware.

The Bottom Line

Think of building a 3D city like building a house of cards.

• Old LiDAR methods are like using heavy, solid blocks. They are stable but heavy and expensive to ship.
• Old Camera methods are like using tissue paper. They are light and cheap, but they blow away in the wind (errors) and collapse.
• MOGS is like using cardboard cutouts of the furniture. You don't need to build every brick; you just need to know "that's a chair" and "that's a table." You use a few measurements to set the size, and the rest of the shape fills itself in perfectly.

The result? A stunningly realistic 3D city map built quickly, cheaply, and accurately, making it possible for self-driving cars and robots to understand the world without needing a million-dollar sensor suite.

Technical Summary

1. Problem Statement

While 3D Gaussian Splatting (3DGS) has achieved real-time, photorealistic rendering, scaling it to large scenes (e.g., autonomous driving environments) presents significant challenges:

• LiDAR Dependency: Current state-of-the-art large-scale systems rely on high-channel LiDAR for metric depth initialization. This is costly, generates dense point clouds that inflate memory usage, and slows down optimization, hindering fleet deployment and rapid iteration.
• Monocular Limitations: Pure monocular approaches lack reliable metric depth, leading to scale drift and geometric inconsistency in large scenes.
• Sparse SfM Issues: Visual-inertial Structure-from-Motion (SfM) provides metric depth but is sparse, often clustering only on high-texture edges. This leaves large, low-texture surfaces (roads, glass, sky) under-constrained, making interior shape estimation ill-posed.
• Geometric Inconsistency: Even with plausible local geometry, monocular cues often fail to enforce global consistency (e.g., parallelism, ground contact) between objects, leading to depth offsets and broken discontinuities.

2. Methodology: MOGS Framework

MOGS is a monocular 3DGS framework that replaces active LiDAR with object-anchored, metrized dense depth derived from sparse visual-inertial (VI) SfM cues and image semantics. The pipeline consists of three main stages:

A. System Overview

1. Input: RGB images and VI-SfM data (camera poses, sparse 3D points).
2. Semantic Segmentation: Uses Segment Anything (SAM) to generate fine-grained object masks.
3. Feature Association: Aligns sparse SfM 3D points (SuperPoint features) with the corresponding semantic masks.
4. Two-Stage Depth Generation:
   • Stage 1: Multi-scale Shape Consensus (generates coarse, object-level metric depth).
   • Stage 2: Cross-Object Depth Refinement (optimizes pixel-level depth for global coherence).
5. Output: Metrized dense depth maps used to initialize 3D Gaussians for high-fidelity rendering.

B. Key Modules

1. Multi-scale Shape Consensus Module

• Challenge: Addresses insufficient SfM coverage within objects (e.g., large roads or roofs where features are sparse).
• Mechanism:
  • Adaptive Merging: Iteratively merges small, under-supported semantic segments into larger "coarse objects" until a region accumulates enough SfM points to fit a parametric model. Merging is constrained by depth continuity to avoid merging objects at different elevations (e.g., a car on a road vs. the road itself).
  • Parametric Fitting: Fits canonical primitives (Plane, Cylinder, Ellipsoid) to the aggregated SfM points using RANSAC. The model with the highest confidence (based on inlier ratio and residual variance) is selected.
  • Propagation: The fitted parametric model propagates metric depth to all pixels within the object, creating dense, object-consistent priors. Non-parametric regions (e.g., foliage) are bypassed for later refinement.

2. Cross-Object Depth Refinement Module

• Challenge: Enforces geometric consistency across different objects and refines the coarse depth from Stage 1.
• Mechanism: Optimizes per-pixel depth using a three-term combinatorial objective:
  1. Geometric Consistency: Aligns the propagated metric depth with a scale-ambiguous but geometrically consistent dense depth from a Large Foundation Model (LFM), specifically Depth Anything.
  2. LFM Prior Anchoring: Softly penalizes deviations from the LFM estimate to preserve local shape details in weakly constrained regions without overriding the metric alignment.
  3. Edge-Aware Smoothness: Encourages smoothness within object interiors while preserving sharp discontinuities at object boundaries using image gradients.
• Optimization: Solves for affine scale and shift parameters per mask using Iteratively Reweighted Least Squares (IRLS).

3. Key Contributions

1. Object-Level Metric Propagation: Introduces a Multi-scale Shape Consensus module that transforms sparse SfM cues into dense, metrized depth priors by fitting parametric models to semantic objects, solving the "interior under-constraint" problem.
2. Global Coherence Optimization: Develops a Cross-Object Depth Refinement module that combines geometric consistency, LFM priors, and edge-aware smoothness to produce a globally coherent depth field, preventing inter-object scale drift.
3. Efficiency and Scalability: Demonstrates that a low-cost VI sensor suite can achieve rendering quality competitive with expensive LiDAR-based systems, significantly reducing memory and training time.

4. Experimental Results

Experiments were conducted on public datasets (KITTI-Depth and KITTI-360) comparing MOGS against state-of-the-art monocular depth estimators and 3DGS baselines.

• Depth Accuracy: MOGS outperforms LFM-only baselines (Depth Anything V2, Depth Pro, etc.) in Absolute Relative Error (AbsRel) and RMSE, achieving 0.058 AbsRel on KITTI-Depth.
• 3DGS Initialization Efficiency:
  • Training Time: Reduces iterations required to reach target quality by up to 30.4%.
  • Memory: Reduces the number of active Gaussian primitives by 19.8%, lowering peak GPU memory consumption.
• Rendering Quality:
  • Achieves PSNR ~20.0 dB on outdoor road sequences, comparable to the costly LiDAR-based baseline (GS-LIVM, PSNR ~20.4 dB) and significantly outperforming monocular baselines (MonoGS: 14.7 dB).
  • Produces sharper thin structures and suppresses "floaters" (artifacts) in long-range scenes.
• Ablation Studies: Removing either the Shape Consensus or Depth Refinement modules leads to significant drops in PSNR (up to 1.39 dB and 0.78 dB respectively) and increased depth errors, confirming the necessity of both components.

5. Significance

• Cost Reduction: MOGS enables high-fidelity 3D reconstruction for autonomous driving and large-scale mapping using low-cost monocular cameras and IMUs, eliminating the need for expensive high-channel LiDAR.
• Scalability: By reducing memory footprint and training time, MOGS makes real-time 3DGS optimization feasible for large-scale deployments and fleet operations.
• Robustness: The object-guided approach effectively handles the limitations of sparse SfM in low-texture environments, providing a robust solution for metric 3D reconstruction where traditional methods fail.

In summary, MOGS bridges the gap between the high accuracy of LiDAR-based 3DGS and the low cost of monocular systems by leveraging semantic object priors to "densify" sparse metric cues, enabling efficient and high-quality large-scale scene reconstruction.