IDSplat: Instance-Decomposed 3D Gaussian Splatting for Driving Scenes

IDSplat is a self-supervised 3D Gaussian Splatting framework that reconstructs dynamic driving scenes with explicit instance-level decomposition and physically consistent motion trajectories without requiring human annotations, leveraging zero-shot language-grounded tracking and coordinated-turn smoothing to enable practical, large-scale autonomous driving applications.

The Gist

Imagine you are trying to build a perfect, interactive 3D movie of a busy city street, complete with cars zooming by, pedestrians crossing, and trees swaying in the wind. You want to be able to pause the movie, pick up a specific car, move it to the other side of the road, or even delete it entirely, and have the rest of the scene look perfectly realistic.

This is exactly what IDSplat does, but for autonomous driving. It's a new computer program that rebuilds real-world driving scenes into a digital "twin" without needing a human to manually label every single car or person.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Spaghetti" Mess

Previous methods tried to rebuild these scenes by treating the whole world as a giant, tangled bowl of spaghetti. They used millions of tiny, floating dots (called "Gaussians") to represent the scene.

• The Issue: When a car drove by, the program didn't know which dots belonged to the car and which belonged to the road. The car and the road got mixed together.
• The Consequence: If you wanted to move the car, you'd accidentally drag a chunk of the road with it. Also, to make this work, humans usually had to spend hours drawing boxes around every car in the video to teach the computer what was moving. That's slow and expensive.

2. The Solution: The "Lego" Approach

IDSplat changes the game by treating the scene like a Lego set. Instead of a messy bowl of spaghetti, it builds the scene out of distinct, separate blocks.

• The "Blocks": It identifies every car, pedestrian, or cyclist as its own unique "instance" (a Lego block).
• The "Static Base": The road, buildings, and trees are the static baseplate.
• The Magic: Because the car is a separate block, the computer knows exactly where it is. It can move the car block without touching the road baseplate.

3. How It Learns Without a Teacher (Zero-Shot)

Usually, to teach a computer to recognize a car, you need thousands of photos where humans have drawn boxes around cars. IDSplat is a "self-taught" genius.

• The Detective Work: It uses a smart AI (called Grounded-SAM-2) that acts like a detective. You can tell it, "Find me all the cars," or "Find me all the people," and it does it instantly without needing prior training on your specific dataset.
• The 3D Lift: It takes these 2D "findings" from the video and uses the car's Lidar (a laser scanner that acts like a 3D ruler) to lift them into 3D space. Now, the computer knows exactly where the "car block" is in 3D space.

4. Tracking the Motion: The "Smoothie" Filter

Once the computer finds the cars, it needs to figure out how they move.

• The Rough Draft: Initially, the computer guesses the path of the car by matching features between frames. This is like trying to trace a path on a shaky hand-drawn map; it's a bit wobbly and has errors.
• The Smoothie: IDSplat uses a special "smoothing" technique (Coordinated-Turn Smoothing). Imagine taking that wobbly hand-drawn path and running it through a blender to make it a perfectly smooth curve. It filters out the jitters and ensures the car moves in a way that makes physical sense (cars don't teleport or spin 90 degrees instantly).

5. The Final Result: A Controllable World

After all this processing, IDSplat produces a high-definition 3D world where:

• Everything is Separated: You can click on a specific car and see only that car, or hide it.
• It's Realistic: You can render new camera angles (like a drone flying over the scene) and the cars will look real, with correct lighting and shadows.
• No Human Labels Needed: It figured all this out on its own.

Why Does This Matter?

Think of self-driving cars like a student learning to drive.

• Old Way: The student only learns by driving on real roads. This is dangerous, expensive, and you can't easily practice "what if" scenarios (like "what if a car suddenly jumps in front of me?").
• IDSplat Way: It creates a perfect digital simulator. Because the system understands that "Car A" is a separate object from "Road B," engineers can create infinite new driving scenarios. They can move cars, change their speeds, or remove them to test the self-driving car's reaction in a safe, virtual environment.

In short: IDSplat is like a magic camera that turns a chaotic video of traffic into a clean, organized, 3D Lego world where every piece is labeled and movable, all without a human ever having to draw a single box.

Technical Summary

1. Problem Statement

Reconstructing dynamic driving scenes is critical for developing autonomous systems via sensor-realistic simulation (digital twins). While recent methods using 3D Gaussian Splatting (3DGS) and Neural Radiance Fields (NeRF) achieve high-fidelity rendering, they face two major limitations in autonomous driving contexts:

1. Reliance on Annotations: State-of-the-art methods typically require costly, human-annotated 3D bounding boxes and object trajectories to separate static backgrounds from dynamic objects.
2. Lack of Instance Decomposition: Existing self-supervised approaches often model dynamic scenes using time-varying primitives (e.g., changing Gaussian attributes over time) without explicit object-level decomposition. This leads to "intertwined" static and dynamic elements, making it impossible to manipulate individual objects (e.g., for scenario generation or editing) or track them consistently.

The core challenge is to achieve self-supervised reconstruction of dynamic scenes that preserves instance-level decomposition and learns coherent motion trajectories without human annotations.

2. Methodology

The authors propose IDSplat, a framework that models dynamic objects as coherent rigid instances undergoing rigid transformations, rather than unstructured time-varying primitives. The pipeline consists of four main stages:

A. Scene Representation

• The scene is represented by a set of translucent 3D Gaussians.
• Static vs. Dynamic: Gaussians are assigned a discrete ID ($z$). $z=0$ represents the static background; $z>0$ represents dynamic object instances.
• Rigid Transformation: Each dynamic instance $z$ has a learnable SE(3) pose $T_{z,t}$ at time $t$. All Gaussians belonging to an instance are transformed rigidly according to this pose, preserving the object's geometry and appearance over time.
• Multi-sensor Rendering: The model supports both camera (RGB) and LiDAR rendering by assigning feature vectors for color, view-dependent effects, and LiDAR properties (intensity, ray drop).

B. Zero-Shot Instance Decomposition

To avoid human annotations, IDSplat employs a zero-shot pipeline:

1. 2D Mask Generation: Uses Grounded-SAM-2 with class prompts (e.g., "car," "pedestrian") to generate instance masks from video frames.
2. 3D Lifting: Projects temporally closest LiDAR points onto the image plane. Points falling within the 2D masks are assigned the corresponding instance ID.
3. Filtering: To handle sensor misalignment and outliers, the masks are eroded, and projected points are clustered using DBSCAN. Only the largest cluster per instance is retained.

C. Trajectory Estimation and Smoothing

1. Initial Pose Estimation: For each instance, a canonical frame is defined based on the densest LiDAR observation. Subsequent frames are registered to this canonical frame using DINOv3 feature correspondences and RANSAC (with the Umeyama estimator) to estimate rigid transformations.
2. Coordinated-Turn (CT) Smoothing: Initial RANSAC poses may be noisy or intermittent. The authors formulate a pose graph optimization problem using GTSAM.
   • Factors: Includes noisy measurement factors (from RANSAC) and CT motion model factors to enforce physical consistency (smooth velocity and curvature).
   • Outlier Rejection: An iterative process discards measurements with high residuals to refine trajectories.
   • Global Alignment: A single rotation is estimated to align the local motion model axis with the global motion direction.

D. Joint Optimization

The entire system (Gaussian parameters, instance poses, and trajectories) is optimized jointly via self-supervised reconstruction loss:

• Loss Function: Combines $L_1$, SSIM, and SSIM losses for images, plus specific LiDAR losses (depth, intensity, line-of-sight, and ray drop probability).
• Refinement: The trajectories are further refined during optimization using gradients from the reconstruction losses to correct minor misalignments.

3. Key Contributions

1. Self-Supervised Instance Decomposition: A novel framework that decomposes dynamic scenes into distinct object instances with learnable motion trajectories without requiring human annotations.
2. Zero-Shot 3D Tracking: Leverages foundation models (Grounded-SAM-2 and DINOv3) to extract instance masks and estimate poses, enabling generalization to new datasets and object classes without retraining.
3. Robust Trajectory Optimization: Introduces a coordinated-turn smoothing scheme combined with photometric consistency to handle tracking failures and sparse views, ensuring physically consistent motion.
4. Controllable Scene Editing: By maintaining explicit instance IDs and rigid transformations, the method enables targeted manipulation of objects (e.g., removal, repositioning, or trajectory editing) for simulation purposes.

4. Experimental Results

The method was evaluated on the Waymo Open Dataset and PandaSet, comparing against state-of-the-art self-supervised methods (DeSiRe-GS, AD-GS, CoDa-4DGS, SplatFlow) and supervised baselines (SplatAD).

• Novel View Synthesis (NVS): IDSplat achieves competitive or superior PSNR, SSIM, and LPIPS scores compared to self-supervised baselines. Notably, it matches the performance of the supervised SplatAD (which uses ground-truth bounding boxes) in many settings.
• Dynamic Region Performance: IDSplat significantly outperforms baselines in dynamic regions (DPSNR), especially under sparse view conditions (e.g., 25% training frames), where time-varying primitive methods fail due to interpolation gaps.
• LiDAR Rendering: The method achieves LiDAR rendering quality (depth and intensity) comparable to supervised methods.
• Generalization: When tested on the PandaSet dataset without hyperparameter tuning, IDSplat matched the performance of annotation-based SplatAD, demonstrating strong zero-shot generalization.
• Tracking Metrics: While not the primary focus, the optimized trajectories show reasonable tracking performance (MOTA/MOTP), though occasional ID switches occur due to the lack of explicit inter-frame identity association.

5. Significance and Impact

• Scalability: By removing the dependency on expensive human annotations for object trajectories, IDSplat makes large-scale dynamic scene reconstruction feasible for diverse datasets.
• Simulation & Safety: The explicit instance decomposition allows for "what-if" scenario generation (e.g., removing a pedestrian or changing a car's path), which is crucial for closed-loop training and safety testing of autonomous driving systems.
• Efficiency: Built on 3D Gaussian Splatting, the method retains the real-time rendering speeds of 3DGS while adding the complexity of dynamic object modeling.
• Future Direction: The paper highlights that while rigid body modeling works well for vehicles, deformable objects (pedestrians) are a limitation, suggesting future work in non-rigid decomposition.

In summary, IDSplat bridges the gap between high-fidelity neural rendering and practical autonomous driving simulation by providing a self-supervised, instance-aware, and editable 3D scene representation.