💻 computer science

AsyncEvGS: Asynchronous Event-Assisted Gaussian Splatting for Handheld Motion-Blurred Scenes

This paper introduces AsyncEvGS, a novel framework combining a flexible high-resolution asynchronous RGB-Event dual-camera system with a cross-domain pose estimation and structure-driven optimization strategy to achieve state-of-the-art 3D reconstruction of handheld scenes under severe motion blur.

Original authors: Jun Dai, Renbiao Jin, Bo Xu, Yutian Chen, Linning Xu, Mulin Yu, Tianfan Xue, Shi Guo

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jun Dai, Renbiao Jin, Bo Xu, Yutian Chen, Linning Xu, Mulin Yu, Tianfan Xue, Shi Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a beautiful 3D photo of a room using your smartphone. But there's a catch: you are moving your hand very fast while taking the picture. The result? A blurry mess.

For years, computers have struggled to turn these blurry photos into sharp 3D models. It's like trying to solve a puzzle when half the pieces are smudged. Existing methods either give up or produce ghostly, distorted shapes.

This paper, AsyncEvGS, introduces a clever new way to fix this problem by pairing your standard phone camera with a special "event camera." Here is how it works, explained simply:

1. The Problem: The "Blurry Hand" Dilemma

When you move your phone quickly, the camera's shutter stays open too long, capturing a smear of light instead of a clear picture.

Standard 3D cameras (like on your phone) are great at seeing colors but terrible at fast movement. They get blurry.
Event cameras are like super-fast eyes. Instead of taking full pictures, they only notice changes in light (like a pixel turning from dark to bright). They are so fast they never get blurry, even when you are shaking your hand. However, they only see in black and white and don't know what color things are.

2. The Old Way vs. The New Way

The Old Way (The Strict Synchronization):
Previous attempts to combine these two cameras required them to be perfectly synchronized, like two dancers who must step on the exact same beat at the exact same millisecond. This required expensive, industrial-grade equipment. You couldn't just strap an event camera to your iPhone and go. Also, most event cameras were low-resolution (like an old, grainy TV), so the final 3D model looked pixelated.

The New Way (The Flexible Duo):
The authors built a system that doesn't care if the two cameras are perfectly in sync. It's like a dance where the partners don't need to step on the beat together; they just need to know the general rhythm.

They paired a high-definition phone camera (1280x720) with a high-definition event camera.
Because they aren't perfectly synced, the computer has to work harder to figure out where the cameras were looking.

3. The Magic Ingredients

A. The "Smart Translator" (VGGT)

Since the two cameras are out of sync and see the world differently (one sees color blur, the other sees sharp black-and-white changes), a standard map-making tool (called COLMAP) gets confused and fails.

The Solution: The authors use a powerful AI model called VGGT. Think of VGGT as a super-smart translator that can look at the blurry color photos and the sharp black-and-white event data, ignore the confusion, and say, "Ah, I know exactly where the camera was for both of these!" This gives the system a solid starting point to build the 3D model.

B. The "Structure Detective" (Event Structure Loss)

Usually, when computers try to use event data, they get confused because the data is relative (it only knows change, not absolute brightness).

The Solution: The authors created a new rule for the computer. Instead of asking "Is this pixel bright?", they ask, "Does the shape of the edge match?"
Imagine trying to trace a drawing. If the lines are shaky, you can't tell what it is. But if you focus only on the structure of the lines (the edges), you can still recognize the shape even if the shading is wrong. This "Structure Loss" helps the computer grab the sharp edges from the event camera and paste them onto the 3D model, fixing the blur.

C. The "Color Guardian" (Consistency Regularizers)

There's a risk that when the computer tries to fix the blur, it might invent weird colors or make the image look like a melting painting.

The Solution: They added "guardrails."
- Guardrail 1: Make sure the sharp images next to each other look similar (so the 3D model doesn't wiggle).
- Guardrail 2: Make sure the black-and-white event view matches the colors learned from the blurry phone photos. This ensures the final 3D model is sharp and colorful.

4. The Result

The team tested this on a new dataset they created (called AsyncEv-Deblur) where they waved their cameras around wildly.

Without their method: The 3D model was a blurry, distorted mess.
With their method: The 3D model was sharp, clear, and colorful. You could read text on a sign or see the logo on a bus, even though the original photos were taken while moving fast.

Summary Analogy

Imagine trying to reconstruct a statue from a blurry photo and a sketch made by someone who only sees movement.

Old methods tried to force the sketcher and the photographer to work in perfect lockstep, which was impossible with normal gear.
This paper says: "Let them work at their own pace." Use a super-smart AI to figure out where they were standing, use the sketcher to fix the shaky lines (edges), and use the photographer to fix the colors. The result is a perfect statue, even if the original inputs were messy.

The paper claims this is the first practical way to do high-quality 3D reconstruction with handheld devices that aren't perfectly synced, achieving the best results seen so far.

Technical Summary: AsyncEvGS

Problem Statement

While 3D Gaussian Splatting (3DGS) and Neural Radiance Fields (NeRF) achieve photorealistic novel view synthesis, they rely heavily on high-quality, sharp input images. In real-world handheld 3D capture, severe motion blur is common, rendering standard methods ineffective and leading to geometric distortions and blurry reconstructions. Existing computational deblurring approaches struggle with the ill-posed nature of the problem.

Event cameras offer a solution by providing high-temporal-resolution, asynchronous motion cues that are robust to motion blur. However, current event-assisted 3D reconstruction methods face two critical barriers to practical handheld use:

Low Sensor Resolution: Most methods rely on low-resolution event sensors (e.g., 346 × 260), which fundamentally limits reconstruction fidelity compared to modern multi-megapixel RGB cameras.
Strict Synchronization Requirements: High-resolution solutions often mandate rigid, hardware-level temporal synchronization between RGB and event cameras, making them incompatible with common handheld devices (e.g., smartphones) that lack such synchronization capabilities.

Methodology

The authors propose AsyncEvGS, a framework designed for high-fidelity 3D reconstruction using a flexible, asynchronous, high-resolution RGB–Event dual-camera system. The pipeline consists of three main components:

1. Hardware System

The authors introduce a novel dual-camera rig pairing a high-resolution (1280 × 720) Prophesee EVK-3 HD event camera with a standard RGB camera (e.g., iPhone 13). Crucially, this system operates without hardware-level temporal synchronization, allowing for "in-the-wild" capture on commodity devices.

2. Robust Initialization via Cross-Domain Pose Estimation

Traditional Structure-from-Motion (SfM) tools like COLMAP fail when jointly registering motion-blurred RGB frames and event-derived grayscale images due to domain gaps and blur.

Event-to-Intensity Reconstruction: The raw event stream is converted into a sequence of sharp grayscale images using E2VID.
VGGT Initialization: Instead of relying on COLMAP, the authors leverage the Visual Geometry Transformer (VGGT) to process both the blurred RGB frames and the sharp event-reconstructed grayscale frames. This provides a robust, dense point cloud and accurate camera pose initialization for both modalities, overcoming the failure modes of traditional SfM in motion-blurred scenarios.

3. Optimization Framework

The 3DGS representation is optimized using a multi-objective loss function designed to handle the asynchronous nature of the data and the ill-posedness of deblurring:

Deblurring Loss ( $L_{blur}$ ): Based on Deblurring 3DGS, this synthesizes a blurred image by averaging $N$ sharp neighboring views rendered by an MLP and compares it to the captured blurred RGB input.
Event Structure Loss ( $L_{struct}$ ): To address the ill-posedness of classical event losses (which only constrain intensity differences), the authors propose a structure-driven loss. It uses a structure extractor and a confidence map derived from multi-scale high-frequency consistency. The loss is computed using the Structural Similarity (SSIM) index between the rendered image and the event-reconstructed target, weighted by confidence. This focuses on structural correctness rather than absolute intensity, mitigating brightness inconsistencies and noise.
Consistency Regularizers:
- RGB Consistency ( $L_{reg-r}$ ): Prevents the deblurring module from converging to degenerate solutions where errors in individual views cancel out. It enforces that the rendered image remains close to both the neighboring sharp frames and the observed blurred input.
- Event Color Consistency ( $L_{reg-e}$ ): Since event views lack color, a coarse 3DGS model trained on RGB images serves as a reference. This regularizer ensures the event-view renderings do not drift in color, distilling RGB color information into the event views.

Key Contributions

Novel High-Resolution Asynchronous System: The first practical pipeline for high-fidelity 3D reconstruction using a flexible 1280 × 720 RGB-Event rig, overcoming the resolution limits of prior low-resolution event sensors and the synchronization constraints of industrial setups.
Robust Cross-Domain Algorithmic Framework:
- A pose estimation pipeline using VGGT to handle the domain gap between blurred RGB and event-derived images, enabling stable initialization where COLMAP fails.
- A tailored optimization framework featuring a novel Event Structure Loss to robustly leverage high-frequency event details and Consistency Regularizers to stabilize deblurring and color estimation.
AsyncEv-Deblur Dataset: A new high-resolution RGB-Event dataset captured with the proposed asynchronous system, featuring diverse scenes with severe motion blur and corresponding ground truth.

Experimental Results

The method was evaluated on the new AsyncEv-Deblur dataset and the synthetic Ev-DeblurBlender dataset.

Performance: The approach achieves state-of-the-art performance across all metrics (PSNR, SSIM, LPIPS), significantly outperforming RGB-only methods (e.g., 3DGS, BAGS, DeblurringGS) and existing event-assisted baselines (e.g., LSENeRF adapted to the new format).
Initialization: Ablation studies show that VGGT initialization successfully registers 100% of frames in challenging motion-blurred scenarios, whereas COLMAP fails to register a significant portion of blurred RGB frames.
Component Analysis: Removing the Event Structure Loss ( $L_{struct}$ ) or the consistency regularizers leads to a substantial drop in performance, confirming their necessity for recovering fine details and preventing color degradation.
Comparison to 2D Pipelines: The end-to-end approach significantly outperforms pipelines that first apply 2D deblurring networks followed by standard 3DGS, highlighting the benefits of joint optimization and multi-view consistency.

Significance

The paper claims that AsyncEvGS effectively bridges the gap between high-resolution 3D capture and the robustness required for motion-blurred scenarios. By decoupling the need for hardware synchronization and utilizing high-resolution sensors, the method makes event-assisted 3D reconstruction feasible for common handheld devices like smartphones. The work demonstrates that leveraging the complementary strengths of RGB (color) and Event (high-frequency motion) modalities through a specialized optimization framework can substantially improve reconstruction robustness and fidelity under severe motion blur.