💻 computer science

ArmGS: Composite Gaussian Appearance Refinement for Modeling Dynamic Urban Environments

ArmGS introduces a composite Gaussian splatting framework with multi-granularity appearance refinement to achieve high-fidelity, real-time modeling of dynamic urban driving scenes by optimizing transformation parameters across local, global, and actor levels.

Original authors: Guile Wu, Dongfeng Bai, Bingbing Liu

Published 2026-02-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guile Wu, Dongfeng Bai, Bingbing Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to create a perfect, photorealistic movie of a busy city street for a self-driving car to practice on. You want the car to see the world exactly as it is: the sun glinting off a wet windshield, a pedestrian stepping off a curb, or a traffic light changing from red to green.

For a long time, computers struggled with this. Old methods were like trying to paint a masterpiece with a thick, slow brush—they looked okay but took forever to create and couldn't play back in real-time. Newer methods (using something called "3D Gaussian Splatting") are like using a high-speed spray gun; they are fast and look great, but they often miss the tiny, subtle details that change when the camera moves or the weather shifts. They might render a car perfectly in one frame, but when the camera moves slightly, the car looks a bit blurry or the colors look wrong.

Enter "ArmGS" (Composite Gaussian Appearance Refinement).

Think of ArmGS as a super-smart digital art director who doesn't just paint the scene once, but constantly tweaks the painting in three different ways to make it perfect, no matter how the camera moves or what time of day it is.

Here is how ArmGS works, broken down into three simple "layers" of refinement:

1. The "Local Tweaker" (The Fine-Grained Painter)

Imagine you are looking at a single tree in the distance. As the car drives past, the light hits the leaves differently.

The Problem: Standard 3D models treat the tree as a static object. It looks the same from every angle, which looks fake.
The ArmGS Solution: ArmGS gives every single tiny "pixel-cloud" (a Gaussian) its own little brain. It learns that this specific leaf should look slightly greener when the sun hits it from the left, and slightly darker when the car moves. It's like having a team of artists who can instantly touch up the color of every single leaf in the scene to match the exact lighting of that moment.

2. The "Global Director" (The Lighting Crew)

Now, imagine the whole scene. Suddenly, a cloud covers the sun, or the car drives from a sunny street into a dark tunnel. The entire image gets darker or changes color temperature.

The Problem: If you only tweak individual leaves (the Local Tweaker), you might fix the tree but miss the fact that the whole sky has turned gray.
The ArmGS Solution: ArmGS has a "Global Director" who looks at the whole picture. If the camera moves into the shade, this director says, "Okay, let's cool down the colors of the entire image just a bit." It ensures that the whole scene feels consistent, like a real movie set where the lighting changes naturally for the whole environment.

3. The "Actor Manager" (The Moving Parts Specialist)

Finally, think about the moving cars and people. A car isn't just a static object; it moves, its brake lights turn on, and its shape distorts slightly as it speeds by.

The Problem: Most systems treat moving cars like ghosts that just slide across the screen. They don't account for the fact that a car's appearance changes based on what it is doing (braking, turning) and where it is.
The ArmGS Solution: ArmGS treats moving objects (actors) like special guests. It uses a lightweight "deformation" tool that reshapes and re-colors these actors in real-time. If a car brakes, ArmGS knows to make the brake lights glow red and the car's shape compress slightly, just like in real life. It understands that a moving car is different from a parked one.

Why Does This Matter?

When you put these three layers together, ArmGS creates a simulation that is fast enough to run in real-time (so a self-driving car can use it instantly) but detailed enough to fool the human eye.

Without ArmGS: The simulation might look like a video game where the sky is a flat blue and the cars look like cardboard cutouts that don't change when the sun moves.
With ArmGS: The simulation looks like a high-definition movie. The rain reflects off the road, the traffic lights change color realistically, and the shadows shift as the car turns a corner.

The Result

The researchers tested this on some of the hardest driving datasets in the world (like Waymo and KITTI). They found that ArmGS didn't just look better; it was significantly more accurate than the current best methods. It successfully captured those "fine-grained" details—the tiny changes that happen every millisecond—that other methods missed.

In short, ArmGS is the difference between a static 3D model and a living, breathing digital city that self-driving cars can trust to learn from.

1. Problem Statement

The paper addresses the challenge of modeling dynamic urban environments for autonomous driving simulation. While existing data-driven methods have made progress:

Neural Radiance Fields (NeRFs): Achieve photorealism but suffer from long training times and fail to achieve real-time rendering.
3D Gaussian Splatting (3DGS): Offers high-fidelity reconstruction and real-time rendering but often neglects fine-grained variations between frames and camera viewpoints.
The Gap: In real-world driving, scene appearance changes significantly due to lighting, camera exposure, and object movement. Existing 3DGS-based approaches for driving scenes fail to model these subtle, multi-granularity variations, leading to the loss of fine details and suboptimal simulation results.

2. Methodology: ArmGS

The authors propose ArmGS (Appearance RefineMent with Gaussian Splatting), a framework that exploits composite 3D Gaussian Splatting enhanced by a multi-level appearance modeling scheme. The core idea is to optimize a set of transformation parameters to refine Gaussians from three distinct granularities:

A. Scene Representation (Composite 3DGS)

The driving scene is decomposed into three components:

Background Gaussians: Static world-coordinate Gaussians.
Dynamic Actor Gaussians: Object-centric Gaussians for moving entities (e.g., vehicles), transformed via pose matrices.
Sky Gaussians: Explicit cubemap representations for distant sky regions.

B. Multi-Level Appearance Refinement

ArmGS introduces three specific refinement modules to handle appearance variations:

Local Composite Gaussian Level Refinement:
- Goal: Capture frame-to-frame variations and local fine-grained changes.
- Mechanism: A low-dimensional learnable embedding ( $\epsilon$ ) is constructed for each frame. This is merged with a multi-resolution Hash grid position code and Gaussian color to form a latent representation ( $f_l$ ).
- Refinement: A lightweight MLP ( $D_l$ ) learns affine transformation parameters ( $\alpha_l, \beta_l$ ) to modulate the color of individual Gaussians: $c' = \alpha_l c + \beta_l$ .
Global Image Level Refinement:
- Goal: Capture holistic scene changes caused by global factors like sunlight intensity or camera exposure across viewpoints.
- Mechanism: Camera location and viewing direction are encoded into a viewpoint code ( $\phi$ ) and merged with the frame embedding ( $\epsilon$ ) to form a global latent representation ( $f_g$ ).
- Refinement: A global transformer ( $D_g$ ) learns affine parameters ( $\alpha_g, \beta_g$ ) to transform the final rendered image color: $C' = \alpha_g C + \beta_g$ .
Dynamic Actor Level Refinement:
- Goal: Model complex motion and appearance changes specific to moving actors (e.g., brake lights, trajectory shifts).
- Mechanism: Instead of heavy HexPlane representations used in prior work, ArmGS uses a light-weight spatial-temporal encoder. It encodes the actor's position ( $\mu$ ), timestamp ( $t$ ), and spherical harmonics ( $h$ ).
- Refinement: A deformation head ( $D_h$ ) predicts offsets for position ( $\Delta\mu$ ) and spherical harmonics ( $\Delta h$ ), allowing the actor's Gaussians to deform and change appearance over time.

C. Optimization

The model is trained end-to-end using a composite loss function ( $L$ ) comprising:

$L_{rgb}$ : Image reconstruction loss.
$L_{ssim}$ : Structural similarity loss.
$L_{d}$ : Depth loss (using LiDAR data).
$L_{s}$ : Sky mask loss.
$L_{f}$ : Foreground decomposition loss (entropy on dynamic actor alpha values).

3. Key Contributions

First Multi-Granularity Embedding: It is the first method to explicitly embed multi-granularity appearances (local Gaussian, global image, and dynamic actor) to model fine-grained changes in dynamic urban environments.
Efficient Transformation Scheme: The method employs a simple yet effective set of transformation parameters for composite Gaussian refinement, preserving the differentiability of the splatting process without requiring complex neural field structures like HexPlanes.
State-of-the-Art Performance: Extensive experiments demonstrate superior performance over existing NeRF and 3DGS methods in both image reconstruction and novel view synthesis.

4. Experimental Results

The authors evaluated ArmGS on four challenging autonomous driving datasets: Waymo, KITTI, NOTR, and VKITTI2.

Quantitative Performance:
- Waymo: Achieved 38.1 dB PSNR (Reconstruction) and 35.7 dB PSNR (Novel View), significantly outperforming competitors like StreetGS (36.3/34.6) and OmniRe.
- KITTI: Achieved 30.3 dB PSNR, surpassing the second-best method by a significant margin.
- NOTR & VKITTI2: Consistently outperformed state-of-the-art methods (e.g., StreetSurf, EmerNeRF, HUGS) in both static and dynamic splits.
Qualitative Performance:
- ArmGS successfully renders fine-grained details often missed by others, such as traffic light color changes, distant vehicle details, and lane lines.
- It handles diverse weather conditions (fog, rain, sunny) effectively, reducing artifacts like blurring seen in other methods.
Efficiency:
- Unlike NeRF-based methods (often <1 FPS), ArmGS achieves real-time rendering (>30 FPS) due to the efficiency of 3DGS.
Ablation Studies:
- Removing any of the three refinement levels (Local, Global, or Actor) resulted in performance degradation, confirming the necessity of the multi-level scheme.
- The lightweight actor encoder outperformed the heavy HexPlane representation used in 4DGS.

5. Significance

ArmGS represents a significant advancement in autonomous driving simulation. By bridging the gap between the speed of 3DGS and the need for high-fidelity, dynamic appearance modeling, it enables:

Safer Validation: The ability to generate photorealistic, dynamic corner cases (e.g., specific lighting conditions or object behaviors) that are difficult to collect in the real world.
Real-Time Closed-Loop Testing: The real-time rendering capability allows for efficient downstream closed-loop evaluation of autonomous driving systems.
Generalizability: The approach works effectively on both real-world data (Waymo, KITTI) and synthetic data (VKITTI2), making it a versatile tool for the autonomous driving industry.