🔬 optics

Engram-E2VID: Reference-Based Event-to-Video Reconstruction via Generative Activation of Appearance Engrams

Engram-E2VID is a reference-based event-to-video reconstruction framework that leverages a one-step diffusion backbone to structurally guide the generative activation of appearance engrams encoded from a reference frame, enabling high-fidelity recovery of target RGB frames from sparse event streams even under complex motion and long temporal intervals.

Original authors: Feiyu Ji, Xiang Li, Hao Ma, Tianxiang Huang, Qingxin Lu, Mengqi Ji, Lei Han, Xiaokang Yang, Xiaoyun Yuan

Published 2026-08-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Feiyu Ji, Xiang Li, Hao Ma, Tianxiang Huang, Qingxin Lu, Mengqi Ji, Lei Han, Xiaokang Yang, Xiaoyun Yuan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to recreate a movie scene, but you only have two very strange tools. The first is a single, perfect photograph of the scene before the action starts. The second is a chaotic stream of tiny, invisible sparks that fly around whenever something moves or changes brightness. These sparks, called "events," are incredibly fast and detailed about movement, but they are completely blind to color, texture, or what things actually look like. They are like a ghostly whisper of motion without a body.

Scientists have long tried to use these sparks to rebuild the missing parts of a movie, but it's like trying to paint a portrait using only a map of where the wind blew. The old methods often get lost when things move too far or too fast, resulting in blurry, ghostly, or completely wrong pictures. This is the puzzle that a team of researchers at Shanghai Jiao Tong University and other institutions decided to solve. They asked: "How can we take that chaotic stream of motion sparks and combine it with our starting photo to rebuild a crystal-clear, new frame of the video, even if a lot of time has passed?"

The answer they found is a new system called Engram-E2VID. Think of it as a magical construction crew that doesn't just copy-paste the old photo. Instead, it builds a "scaffold" or a skeleton of the new scene using the motion sparks. This scaffold tells the system where things are moving and how the structure is changing. Then, the system reaches into its memory bank—the original photo—and pulls out the specific "appearance engrams" (which are like detailed memory fragments of how the objects look).

Here is the clever part: instead of trying to stretch the old photo to fit the new position (which usually makes it look warped and blurry), the system uses a smart "activation" process. It asks the scaffold: "Hey, I need the texture of a red ball here, and the fur of a cat there." The scaffold points to the right spots, and the system grabs the perfect memory fragments to fill them in. If there are parts of the scene that were completely hidden or new, a powerful AI "imagination" (called a diffusion model) steps in to fill in the blanks realistically.

The results are impressive. When tested on three different real-world datasets, this new method didn't just work; it worked much better than the previous best attempts. In simple terms, it made the reconstructed images significantly sharper and more accurate. Specifically, it improved the picture quality score (PSNR) by up to 3.29 dB and reduced visual fuzziness (LPIPS) by up to 0.08 compared to the strongest existing method that used the same inputs.

Perhaps the most exciting finding is how well it handles time. Usually, the longer you wait between the starting photo and the new frame you want to create, the worse the picture gets. But Engram-E2VID degrades much more slowly. Even when the time gap was large, it kept the details sharp and the structures correct, whereas other methods started to blur and hallucinate weird shapes. The researchers suggest that by separating the "structure" (the scaffold) from the "appearance" (the engrams) and letting them talk to each other in a smart way, they can reconstruct scenes that are complex, moving fast, or far away from the original photo, all without needing a second photo to help them out. It's a bit like being able to remember exactly what a friend's face looks like, even if they have moved to a completely different room and are running around, just by knowing the layout of the room and the sound of their footsteps.

Technical Summary: Engram-E2VID

Problem Statement
Reference-based event-to-video reconstruction aims to recover target RGB frames using a reference frame and the asynchronous event stream captured over the interval between the reference and target timestamps. While event cameras offer microsecond temporal resolution and high dynamic range, they encode only relative log-intensity changes rather than absolute appearance, color, or texture. This makes faithful reconstruction inherently ill-posed.

The central challenge lies in associating the sparse, asynchronous event-derived structures at the target time with the relevant appearance information from the reference frame. Existing methods typically rely on estimating motion fields to warp the reference frame. However, under complex motion, occlusion, or long temporal intervals, simple warping fails to reconstruct newly revealed or substantially altered regions. Furthermore, recent diffusion-based approaches that use events as auxiliary conditioning often lack an explicit mechanism to associate event-derived target structures with reference appearance cues, leading to structural inconsistencies or hallucinated content.

Methodology
The authors propose Engram-E2VID, a structure-guided framework that reconstructs target frames through the "generative activation of appearance engrams." The method operates within a one-step diffusion backbone and consists of two primary stages:

Structural Scaffold Estimation:
The asynchronous event stream between the reference ( $t_0$ ) and target ( $t_k$ ) timestamps is converted into a spatiotemporal event voxel representation ( $V$ ). A lightweight dual-encoder U-Net processes both the event voxel and the reference frame to estimate a "motion-structure scaffold" ( $\tilde{I}^s_{t_k}$ ). This scaffold is not required to be photorealistic; its primary function is to expose motion boundaries and scene structures to the diffusion model.
Appearance Engram Activation:
The scaffold and the reference frame are encoded into latent representations ( $z_s$ and $z_r$ ) using a frozen VAE encoder.
- Engrams: The reference latent tokens ( $z_r$ ) serve as "appearance engrams," retaining rich visual information.
- Scaffold Tokens: The scaffold latent tokens ( $z_s$ ) encode target-time structural information.
- Latent Interaction Attention (LIA): A novel interaction interface is introduced where scaffold and reference latents are stacked and rearranged into a joint token sequence. Within the transformer blocks of the diffusion backbone, self-attention layers allow structural tokens to interact with and activate relevant appearance engrams. This association occurs in token space rather than through direct pixel-wise transfer, allowing the model to access reference appearance despite significant spatial displacement.
The framework employs a one-step diffusion process (adapted from SD-Turbo with LoRA) where the scaffold-derived structural tokens act as queries to retrieve relevant engrams. A skip connection with zero-initialized convolutions preserves structural information from the scaffold to prevent drift. For high-resolution inference, the process is applied in a tile-wise manner with overlapping windows.

Key Contributions

Target-Time Motion-Structure Scaffold: The authors introduce a scaffold that integrates event data with reference context to provide explicit structural guidance for reconstruction, rather than relying solely on motion warping.
Structure-to-Engram Interaction Interface: They establish a mechanism where scaffold-derived structural tokens activate relevant appearance engrams within a one-step diffusion backbone. This token-space association avoids rigid pixel-wise correspondence, enabling robust reconstruction under large displacements.
Performance and Robustness: Extensive experiments demonstrate that the method achieves superior visual quality and structural fidelity compared to state-of-the-art baselines, with a notably slower degradation rate as the reconstruction interval increases.

Experimental Results
The method was evaluated on three synchronized event-frame datasets: BS-ERGB, ERF-X170FPS, and HQ-EVFI.

Quantitative Performance: Engram-E2VID consistently outperformed the strongest same-input baseline (RE-VDM). Improvements included up to 3.29 dB increase in PSNR and up to 0.08 reduction in LPIPS across the benchmarks.
Long-Interval Robustness: As the temporal interval (skip) increased, Engram-E2VID exhibited significantly slower performance degradation compared to warping-based methods (e.g., VFPSIE) and other diffusion-based approaches. For instance, on the BS-ERGB dataset, the PSNR drop from skip 1 to 7 was only 1.85 dB for Engram-E2VID, compared to 4.30 dB for VFPSIE.
Qualitative Results: Visual comparisons showed sharper object boundaries, fewer visual artifacts, and better preservation of scene structure and appearance details, particularly in regions with complex motion or occlusion.

Significance and Claims
The paper claims that Engram-E2VID addresses the fundamental limitation of existing reference-based methods by decoupling structural guidance from appearance retrieval. By organizing reference appearance as "engrams" and allowing target structures to "activate" them via attention, the framework enables faithful reconstruction without relying on direct pixel correspondence. The authors assert that this approach effectively leverages the diffusion prior to resolve ambiguities in uncertain or newly revealed regions while maintaining structural consistency. The results suggest that this token-space association is particularly advantageous for long temporal intervals where traditional motion estimation fails. The paper concludes that the method achieves high-quality reconstruction without explicit temporal consistency regularization, maintaining temporal coherence across independently reconstructed frames.

Technical Summary: Engram-E2VID

More like this