💻 computer science

ENCORE: Event-Assisted Complementary Motion Refinement for Learned Video Compression

The paper proposes ENCORE, a learned video compression framework that leverages event camera data to refine RGB-based motion estimation through complementary representation decomposition, redundancy-aware calibration, and energy-aware routing, achieving significant bitrate savings and quality improvements across various datasets.

Original authors: Shuhan Ye, Hongbin Yu, Chenqi Kong, Pingchuan Ma, Chong Wang, Jun Wan, Qixin Zhang

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shuhan Ye, Hongbin Yu, Chenqi Kong, Pingchuan Ma, Chong Wang, Jun Wan, Qixin Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to send a live video of a soccer match to your friends on a phone with a slow internet connection. To make the video fit, you have to compress it, which means throwing away some information. The trickiest part of video compression is handling movement. If a player runs across the screen, the computer doesn't need to send a whole new picture of the player; it just needs to say, "The player moved two steps to the left." This is called motion estimation. However, standard cameras take pictures at a fixed speed, like a flipbook. If the player moves too fast, or if it's foggy, or if the camera shakes, those flipbook pages can get blurry or confusing. The computer gets stuck guessing where the player actually went, leading to a pixelated mess or a file that is still too big.

This is where a special kind of camera called an "event camera" comes in. Unlike a regular camera that takes a full photo every second, an event camera is like a super-sensitive security guard who only shouts out when something changes. If a pixel gets brighter or darker, the event camera instantly reports it. It doesn't care about the color of the grass or the texture of the jersey; it only cares about the motion and the changes in light. It's incredibly fast and works great even in the dark or when things are moving super quickly. The big question scientists have been asking is: Can we use this "shouting guard" to help the "flipbook" camera make better guesses about movement, so we can send clearer videos using less data?

This paper, titled ENCORE, says "Yes, but with a very specific strategy." The researchers built a new system that uses event camera data to help a standard video compression algorithm, but with a twist: the final video is still just a normal video. The event data is never sent to the viewer; it's only used behind the scenes to fix the "guessing game" of movement.

The authors found that simply mixing the two types of data together (like dumping the event data into the video file) actually makes things worse. It's like trying to read a book while someone is shouting random numbers in your ear; the extra noise just confuses you. Instead, they created a smart filter system with three distinct steps to make the event data useful:

The Translator (CMR): First, the system separates the "common sense" movement that both cameras see from the "special info" that only the event camera sees. It realizes that sometimes both cameras agree on where a ball is going, but sometimes the event camera sees a blur that the regular camera missed.
The Editor (SERIC): Next, it acts like a strict editor. It looks at the event camera's "shouts" and asks, "Is this new information, or is it just noise?" If the event camera is shouting about something the regular camera already figured out perfectly, the editor silences it. If the event camera spots a fast movement the regular camera is blind to, the editor highlights it.
The Traffic Cop (EAR): Finally, a routing system decides exactly where and how much to use this new information. If the regular camera is doing a great job tracking a slow-moving tree, the traffic cop tells the event data to stay quiet. But if a car zooms by and the regular camera is confused, the traffic cop opens the gate and lets the event data guide the correction.

The results are impressive. When they tested this system on three different datasets involving fast motion and tricky lighting, it consistently saved a lot of data while keeping the video quality high. On one specific test set called BS-ERGB, the system managed to reduce the file size by 20.80% (measured as PSNR-RGB BD-rate savings) and 22.14% (measured as MS-SSIM-RGB BD-rate savings) compared to the best standard method, without losing any picture quality. Even on other datasets with different sensors and high-speed footage, the system showed clear improvements.

The paper explicitly argues against the idea that you should just mash all the data together or try to send the event data along with the video. They show that doing so introduces noise and actually hurts performance. Instead, they prove that using events strictly as a "helper" to refine the motion guess for the main video is the winning strategy. While the system does require a tiny bit more computing power (about 7.3% more calculations per frame), the trade-off is worth it for the massive savings in data transmission. In short, ENCORE teaches us that sometimes, to see the future clearly, you don't need more pictures; you just need to listen to the things that change.

Technical Summary: ENCORE – Event-Assisted Complementary Motion Refinement for Learned Video Compression

1. Problem Definition

Learned video compression (LVC) relies heavily on accurate temporal modeling to exploit redundancy between adjacent frames. However, existing codecs infer motion solely from discretely sampled RGB frames. This approach becomes unreliable in scenarios involving fast motion, motion blur, occlusion, weak textures, low illumination, or abrupt brightness changes. Inaccurate temporal modeling degrades prediction quality, increases uncertainty in motion and residual representations, and ultimately raises bitrates.

While multimodal compression exists, most methods either jointly encode multiple modalities (e.g., RGB-D, stereo) or use auxiliary modalities for semantic guidance (e.g., text-to-image). There is a lack of methods that utilize an auxiliary temporal stream to specifically refine motion estimation for RGB video compression without reconstructing the auxiliary signal itself. Event cameras, which asynchronously capture fine-grained intensity changes with high temporal resolution and dynamic range, offer complementary evidence to RGB frames regarding inter-frame dynamics. However, naive fusion of event data often introduces noise and redundancy, failing to improve compression performance.

2. Methodology

The authors propose ENCORE (Event-Assisted Complementary Motion Refinement), a framework that integrates event streams as an auxiliary modality to refine motion estimation within a standard learned video compression pipeline. The system retains RGB as the sole coding and reconstruction target, using events only to assist the encoder's motion modeling.

The framework is built upon the HyTIP backbone and inserts three specialized modules into the encoder-side motion estimation interface:

A. Complementary Motion Representation (CMR)

The CMR module decomposes aligned RGB and event features into three distinct components to prevent the dilution of complementary cues:

Common Motion ( $F^c_t$ ): Structures supported by both modalities (e.g., dominant displacement patterns).
RGB-Specific Motion ( $F^{r,s}_t$ ): Information unique to the dense appearance of RGB.
Event-Specific Motion ( $F^{e,s}_t$ ): Dynamics unique to the event stream (e.g., rapid brightness transitions).

This decomposition is enforced via a reconstruction loss (to preserve original information) and a separation loss (to ensure orthogonality between the branches), preventing the common branch from duplicating modality-specific information.

B. Spatial Energy and Redundancy-Informed Calibration (SERIC)

Not all event responses are useful; some are noise, redundant with RGB, or weak. SERIC identifies and calibrates the most valuable event evidence:

Activity Map ( $E_t$ ): Measures the local energy (squared magnitude) of the event-specific feature to identify active regions.
Novelty Map ( $N_t$ ): Computes the discrepancy between the event feature and the RGB feature (projected into event space) to identify evidence not explained by RGB.
Utility Estimation: A learnable function combines $E_t$ and $N_t$ to generate a utility map ( $U_t$ ). This map suppresses weak or redundant responses while retaining regions where events provide novel, high-energy dynamic evidence.
Correction Prediction: The calibrated event feature is combined with the common motion to predict a candidate flow correction ( $\Delta v_t$ ).

C. Energy-Aware Routing (EAR)

The final stage determines where and how strongly the candidate correction should refine the initial RGB flow.

The routing network takes local activity, novelty, the magnitude of the initial RGB flow, and the magnitude of the correction as input.
It outputs a spatial routing map ( $G_t \in [0, 1]$ ) that acts as a gate.
Refined Flow: The final motion vector is computed as $v^{evt}_t = v^r_t + G_t \odot \Delta v_t$ .
This mechanism ensures that in regions where the RGB flow is already reliable, the correction is suppressed (approaching zero), preventing unnecessary perturbations. In regions with strong complementary dynamics, the correction is applied.

D. Optimization

The system is trained end-to-end to minimize the rate-distortion cost of the reconstructed RGB video ( $L_{RD}$ ). Auxiliary losses are used during training to guide the decomposition and alignment but are discarded at inference:

Reconstruction Losses ( $L_r, L_s$ ): Enforce the quality of the CMR decomposition.
Motion Alignment Loss ( $L_m$ ): Distills reliable RGB displacement into the event stem using a reliability mask derived from photometric error.
Warping Loss ( $L_w$ ): Supervises the refined flow by minimizing the warping error of the previous frame.

3. Key Contributions

The paper claims three primary contributions:

First Application: To the authors' knowledge, this is the first work to use event data as an auxiliary modality specifically for RGB video compression, improving temporal prediction while keeping RGB as the exclusive coding target.
Progressive Refinement Framework: The development of a novel architecture (CMR, SERIC, EAR) that explicitly separates common and modality-specific motion, identifies active event evidence unexplained by RGB, and spatially routes the resulting flow correction.
Comprehensive Evaluation: Extensive experiments across three RGB-event datasets (BS-ERGB, HQ-EVFI, CED) and two Group of Pictures (GOP) settings, demonstrating consistent coding gains with a marginal increase in computational complexity (7.3% increase in GFLOPs per P-frame).

4. Experimental Results

The authors evaluated ENCORE against an RGB-only HyTIP baseline and a naive RGB+Event concatenation approach.

Datasets: BS-ERGB (primary), HQ-EVFI (high-frame-rate), and CED (different sensor).
Short GOP (GOP=8): On BS-ERGB, ENCORE achieved a 19.31% PSNR-RGB BD-rate saving and 16.73% MS-SSIM-RGB saving. The naive concatenation approach performed worse than the baseline, highlighting the necessity of the proposed refinement modules.
Long GOP (GOP=32): On BS-ERGB, ENCORE achieved up to 20.80% PSNR-RGB and 22.14% MS-SSIM-RGB BD-rate savings.
Generalization: Consistent gains were observed on HQ-EVFI and CED, demonstrating robustness across different capture conditions and sensor types.
Complexity: The framework adds only a 7.3% increase in total GFLOPs per P-frame compared to the backbone.

5. Significance and Claims

The paper positions ENCORE as a solution to the limitations of RGB-only temporal modeling in challenging video compression scenarios. Its significance lies in:

Efficiency: It demonstrates that auxiliary sensors can significantly reduce bitrate without the overhead of reconstructing the auxiliary signal or jointly encoding multiple modalities.
Robustness: By explicitly calibrating and routing event evidence, the method avoids the pitfalls of naive feature fusion, effectively handling noise and redundancy inherent in event streams.
Practicality: The framework integrates seamlessly into existing learned compression pipelines (specifically HyTIP) with minimal architectural changes and computational overhead, making it a viable candidate for future video compression standards where event cameras might be available.

The authors remain modest, noting that while the results are consistent and significant, the method is specifically designed for scenarios where event data is synchronized and available at the encoder, and the auxiliary head is discarded at inference.