Technical Summary: Event Enhanced Gaussian Splatting (E2GS)

Problem Statement

Recent advancements in 3D scene reconstruction and novel view synthesis have been driven by Neural Radiance Fields (NeRF) and, more recently, 3D Gaussian Splatting. While 3D Gaussian Splatting offers rapid training and rendering capabilities compared to NeRF, these methods generally assume ideal input conditions. They struggle significantly when presented with motion-blurred images, a common artifact caused by object motion, camera shake, or lens defocusing.

Conversely, event cameras (Dynamic Vision Sensors) offer high dynamic range, no motion blur, and low latency by asynchronously capturing pixel brightness changes. Previous event-based 3D reconstruction methods, such as E2NeRF, successfully utilized event data alongside blurry RGB images to reconstruct sharp scenes. However, these approaches rely on the NeRF paradigm, which necessitates extensive ray-sampling strategies, resulting in prohibitively long training times and slow rendering speeds. There remains a need for a method that combines the efficiency of Gaussian Splatting with the motion-blur resilience of event data.

Methodology

The authors propose Event Enhanced Gaussian Splatting (E2GS), a framework that integrates event stream data into the 3D Gaussian Splatting pipeline to handle motion-blurred inputs. The method operates through three primary stages:

1. Preprocessing and Initialization

• Event-Driven Deblurring: Given a set of blurry images and their corresponding event streams, the method divides the exposure time of each image into $N$ equal timestamps. Using the Event-based Double Integral (EDI) model, it estimates a sequence of sharp intermediate images ($I_i$) by accumulating event data. The blurry image is modeled as the average of these intermediate frames.
• Initialization: The estimated sequence of deblurred images is fed into a Structure-from-Motion (SfM) pipeline (COLMAP) to generate the initial 3D Gaussian coordinates and camera poses. This step is crucial, as standard SfM often fails when initialized directly with blurry images.

2. Training Framework

E2GS optimizes the 3D Gaussian parameters using two distinct loss functions to supervise the reconstruction:

• Image Rendering Loss ($\mathcal{L}_{blur}$): To adapt Gaussian Splatting for blurry inputs, the method renders $N$ intermediate views corresponding to the timestamps and averages them to simulate the blurry ground truth. The loss is a combination of L1 loss and D-SSIM (Dissimilarity Structural Similarity) comparing the simulated average against the input blurry image.
• Event Rendering Loss ($\mathcal{L}_{event}$): To leverage the high temporal resolution of event data, the method randomly selects two frames from the rendered sequence, converts them to log-domain intensity, and calculates the expected number of events based on the intensity difference and a threshold $C$. This estimated event bin is compared against the ground truth event bin using Mean Squared Error (MSE). This loss enforces consistency with the continuous temporal process of blurring, rather than just the static average.

The total loss function is a weighted sum: $\mathcal{L} = \mathcal{L}_{blur} + w_{event}\mathcal{L}_{event}$.

Key Contributions

1. First Integration of Events into Gaussian Splatting: E2GS is the first approach to incorporate event data into the Gaussian Splatting framework, moving beyond the volume rendering paradigm of NeRF.
2. Efficient Deblurring and Synthesis: The method effectively utilizes both blurry RGB images and event streams to achieve high-quality image deblurring and novel view synthesis.
3. Significant Speed Improvements: By replacing ray-sampling with differentiable rasterization, E2GS drastically reduces computational costs compared to NeRF-based event methods.

Experimental Results

The authors evaluated E2GS on both synthetic datasets (simulated camera shake in Blender) and real-world datasets (captured with DAVIS346 event cameras).

• Quantitative Performance:
  • Image Deblurring: On synthetic data, E2GS achieved a PSNR of 30.84, outperforming standard Gaussian Splatting (22.92) and matching or slightly exceeding E2NeRF (29.77). On real-world data, E2GS achieved the lowest BRISQUE scores (indicating higher naturalness), averaging 32.50 compared to E2NeRF's 36.59.
  • Novel View Synthesis: E2GS demonstrated superior or competitive results in PSNR, SSIM, and LPIPS metrics against both standard GS and E2NeRF on synthetic data. On real-world novel view synthesis, E2GS again achieved the best BRISQUE scores (average 32.77).
• Efficiency:
  • Training: E2GS reduced training time by a factor of 60 compared to E2NeRF (50 minutes vs. 2 days).
  • Rendering: E2GS achieved a rendering speed of 140 FPS, representing a 3500x speedup over E2NeRF (0.04 FPS).

Significance and Claims

The paper claims that E2GS successfully bridges the gap between the high-quality reconstruction of event-based NeRF methods and the computational efficiency of Gaussian Splatting. The authors assert that their method generates visually appealing renderings while offering significantly faster training and rendering speeds, making real-time or near-real-time 3D reconstruction from motion-blurred, event-augmented data feasible.

The authors conclude that while their method handles static scenes effectively, future research directions include addressing dynamic scenes with fast-moving subjects (e.g., sports), which remain challenging for current RGB-only or static-scene-focused approaches.