💻 computer science

LongE2V: Long-Horizon Event-based Video Reconstruction, Prediction, and Frame Interpolation with Video Diffusion Models

LongE2V is a novel framework that leverages fine-tuned video diffusion models with specialized mechanisms like autoregressive unrolling and adaptive context switching to achieve high-quality, temporally coherent event-based video reconstruction, prediction, and frame interpolation with superior generalization.

Original authors: Cheng-De Fan, Chun-Wei Tuan Mu, Chen-Wei Chang, Chin-Yang Lin, Kun-Ru Wu, Yu-Chee Tseng, Yu-Lun Liu

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Cheng-De Fan, Chun-Wei Tuan Mu, Chen-Wei Chang, Chin-Yang Lin, Kun-Ru Wu, Yu-Chee Tseng, Yu-Lun Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-fast, super-sensitive camera that doesn't take normal photos. Instead of capturing a full picture every second, it only snaps tiny, invisible sparks whenever something in the scene moves or changes brightness. It's like a camera that only sees the "action" but forgets the "colors" and "shapes" in between. This is an event camera.

The problem? Trying to turn those scattered sparks back into a smooth, colorful movie is like trying to rebuild a shattered vase using only a few crumbs of clay. Old methods tried to guess the missing pieces, but they usually ended up with blurry, muddy videos. Other fancy new methods tried to guess the whole future, but they quickly got confused, drifting off into a weird, colorful nightmare as the video got longer.

Enter LongE2V, a new approach by researchers at National Yang Ming Chiao Tung University. Think of LongE2V as a master chef who has already memorized millions of recipes (a pre-trained video model) and is now learning to cook using only those tiny sparks as ingredients.

The Three Magic Tricks

LongE2V doesn't just do one thing; it tackles three different cooking challenges with the same kitchen:

Reconstruction (The "Restore" Trick): You give it a stream of sparks with no starting picture. It has to guess what the scene looked like from scratch. Old chefs (like E2VID) would just guess the "average" color, resulting in a blurry mess. LongE2V, however, uses its memory of how real movies look to paint sharp, high-definition textures, recovering details that seemed lost forever.
Prediction (The "Future" Trick): You give it one starting photo and a stream of sparks, and ask, "What happens next?" If you ask a standard AI to predict a long movie, it usually starts making things up, drifting away from reality until the colors are wrong and the shapes are melting. LongE2V uses a special "step-by-step" strategy. It checks its own work constantly, like a driver checking the rearview mirror, to make sure it doesn't drift off the road. It can generate long sequences without losing its mind.
Frame Interpolation (The "Fill-in" Trick): You give it a start photo and an end photo, and ask it to fill in the middle. Imagine a car zooming past; you have the photo of it entering the frame and leaving, but you need the blurry middle. Old methods would just smear the car, creating a ghostly double-image. LongE2V acts like a time-traveling editor, looking at the sparks moving forward and backward, then snapping them perfectly together to create a smooth, ghost-free motion.

How It Avoids the "Drift"

The biggest enemy in making long videos is drift. Imagine you are trying to draw a long line by looking at your own drawing of the previous step. If you make a tiny mistake in step one, your brain tries to correct it in step two, but that correction makes a new mistake in step three. Soon, your line is a zigzag mess.

LongE2V solves this with two clever tricks:

Autoregressive Unrolling: Instead of just learning from perfect examples, the model practices by trying to predict its own mistakes. It learns to fix the errors it makes while generating, so it gets better at staying on track over long periods.
Adaptive Context Switching: Sometimes, the model gets too attached to its old memories (the start of the video) and ignores the new sparks. LongE2V has a "reality check." It calculates how much the current scene still cares about the past. If the connection gets too weak, it smartly switches its focus to the immediate past, preventing the video from drifting into nonsense.

The "Zero-Shot" Surprise

Here is the most playful part: The researchers trained LongE2V only on reconstruction and prediction tasks. They never explicitly taught it how to do frame interpolation. Yet, when they handed it a start and end frame, it figured out how to fill in the middle on its own, without any extra training. It's like teaching a dog to fetch a ball, and then watching it figure out how to catch a frisbee just by watching you throw it. This is called zero-shot adaptation, and it means the model is incredibly flexible.

What It Can't Do (The Limits)

The paper is honest about where the magic stops. If the input sparks are too sparse (like trying to rebuild a house with only two bricks), the video quality drops. Also, if the camera has "hot pixels" (tiny, noisy dots that are always on), the model might accidentally preserve those noise dots in the final video, making them look like permanent scars on the image.

The Proof

The team tested LongE2V on real-world datasets like ECD, MVSEC, and HQF. The results were measured using strict math scores (PSNR, SSIM, and LPIPS). In the Reconstruction task, LongE2V beat the best previous methods on all three datasets, achieving a top LPIPS score of 0.139 on the ECD dataset (where lower is better). In Prediction, it crushed the competition, scoring 24.40 PSNR on ECD compared to the next best's 20.33.

For Frame Interpolation, they tested it on the BS-ERGB and HQF datasets. Even though it wasn't trained for this, it beat specialized "interpolation-only" experts, achieving an LPIPS of 0.124 on BS-ERGB, proving it handles complex motion and fine details (like readable text) much better than the old ways.

In short, LongE2V is a versatile, high-fidelity engine that turns the chaotic sparks of event cameras into smooth, stable, and surprisingly long movies, all while keeping its cool and not drifting away into the void.

Technical Summary: LongE2V

Problem Statement

Recovering high-quality video from sparse event streams is an inherently ill-posed problem. Event cameras offer microsecond resolution and high dynamic range but produce intensity-free, asynchronous data incompatible with standard vision algorithms. Existing solutions face three primary challenges:

Reconstruction: Regression-based methods (e.g., E2VID) suffer from "regression-to-the-mean," resulting in blurry textures and loss of high-frequency details.
Prediction: Long-horizon video prediction using generative models often leads to severe error accumulation and temporal drift, causing color shifts and structural instability over time.
Frame Interpolation: Existing interpolation networks struggle with complex, fast dynamics, producing ghosting artifacts and failing to maintain structural integrity.
Furthermore, most state-of-the-art (SOTA) approaches are task-specific, requiring separate architectures for reconstruction, prediction, and interpolation, which limits flexibility and efficiency.

Methodology

The authors propose LongE2V, a unified framework that leverages pre-trained Video Diffusion Models (specifically CogVideoX I2V) to jointly handle event-based video reconstruction, prediction, and frame interpolation. The core architecture treats these tasks as conditional generation problems driven by event voxels.

Key Technical Components

Event Representation and Augmentation:
- Asynchronous event streams are discretized into voxel grids ( $V \in \mathbb{R}^{B \times H \times W}$ ) with $B=3$ temporal bins.
- Event Voxel Density Augmentation: To ensure robustness across varying sensor resolutions, the training process includes random resizing of event voxels (preserving aspect ratio) and normalization based on non-zero statistics. This prevents density mismatches between training and inference.
Autoregressive Unrolling and Adaptive Context Switching:
- To address the domain gap between training (using Ground Truth context) and inference (using predicted context), the authors employ an Autoregressive Unrolling strategy. The model is initially trained with GT context, then iteratively fine-tuned using its own predictions as context.
- Adaptive Context Switching: To mitigate error accumulation in long sequences, the model dynamically evaluates the relevance of the current context. It computes the average attention weight ( $\mu_{attn}$ ) of current tokens attending to context tokens. If relevance drops below a threshold ( $\tau=0.05$ ), the system triggers a "Context Switch," discarding the current generation, updating the context to the immediate predecessor, and regenerating the chunk. This prevents the propagation of drift.
Reencoding Alignment and Cross Residual Correction (for Interpolation):
- Reencoding Alignment: Standard temporal flipping in latent space does not commute with 3D VAE operations, causing misalignment. The authors propose decoding predicted latents to pixel space, flipping them temporally, and re-encoding them to ensure precise bidirectional consistency.
- Cross Residual Correction: To compensate for information loss during the decode-flip-encode loop, the method calculates the residual between original and re-encoded latents. These residuals are injected cross-wise (forward residual into the backward branch and vice versa) to restore high-frequency details and promote temporal consensus.
Unified Training Pipeline:
- The backbone is a frozen 3D VAE and a Diffusion Transformer (DiT).
- The first projection layer is expanded to accommodate event voxel channels and fully fine-tuned, while the DiT blocks are fine-tuned using LoRA.
- The model supports zero-shot adaptation for frame interpolation by utilizing forward and backward event streams as dense motion guides without additional fine-tuning.

Key Contributions

Unified Architecture: LongE2V is the first approach to handle event-based reconstruction, long-horizon prediction, and frame interpolation within a single, fine-tuned video diffusion model, achieving superior data efficiency.
Stability Mechanisms: The introduction of Autoregressive Unrolling combined with Adaptive Context Switching effectively mitigates temporal drift in extremely long sequences, a persistent issue in prior diffusion-based prediction.
Interpolation Consistency: The novel Reencoding Alignment and Cross Residual Correction strategies resolve temporal misalignments in the 3D VAE latent space, eliminating ghosting artifacts in frame interpolation.
Robustness: Event Voxel Density Augmentation ensures the model generalizes across different sensor resolutions and scene depths.

Experimental Results

The method was evaluated on real-world benchmarks (ECD, MVSEC, HQF) and the BS-ERGB dataset.

Reconstruction: LongE2V outperforms SOTA regression methods (E2VID, FireNet, HyperE2VID) across all datasets. It achieves superior perceptual quality (lowest LPIPS) and recovers high-frequency textures that regression methods blur.
Prediction: In long-term prediction tasks, LongE2V significantly outperforms VDM-EVFI. It demonstrates minimal drift and maintains structural fidelity over thousands of frames, whereas baselines suffer from severe color drift and noise accumulation.
Frame Interpolation: Evaluated in a zero-shot setting (without fine-tuning for interpolation), LongE2V surpasses supervised SOTA baselines (CBMNet-Large, TLXNet+). It produces sharper results with less ghosting, particularly in scenes with large motion and fine text.
Subject Consistency: Using VBench, the reconstruction and prediction variants achieved significantly higher subject consistency scores (0.7204 and 0.7187, respectively) compared to baselines, confirming the preservation of identity and structure.

Significance and Claims

The paper claims that LongE2V effectively bridges neuromorphic sensing and human-interpretable vision by leveraging the strong priors of foundation video models. Its significance lies in:

Solving the "Long-Horizon" Problem: By introducing adaptive mechanisms to handle error accumulation, it enables stable video generation over extended durations, a capability previously lacking in event-based diffusion models.
Task Agnosticism: It demonstrates that a single model can flexibly adapt to reconstruction, prediction, and interpolation tasks, reducing the need for specialized architectures.
Zero-Shot Generalization: The ability to perform frame interpolation without task-specific fine-tuning highlights the robustness of the learned event-to-video priors.

The authors acknowledge limitations, noting that performance degrades with highly sparse or low-quality event streams and that the model can be sensitive to noise (e.g., "hot pixels"). Future work is directed toward accelerating inference and exploring more efficient memory mechanisms.