Technical Summary: Moment-Reenacting: Inverse Motion Degradation with Cross-shutter Guidance

1. Problem Statement

The paper addresses the fundamental challenge of motion degradation in computational imaging, which manifests as motion blur in Global Shutter (GS) images or Rolling Shutter (RS) distortion (e.g., jello effects) in RS images under fast motion or low-light conditions.

While prior research has treated blur decomposition (recovering a sequence of latent frames from a single blurred image) and RS temporal super-resolution (reconstructing high-frame-rate videos from RS images) as separate tasks, the authors argue that this separation fails to exploit their intrinsic complementarity. Both tasks are highly ill-posed due to specific ambiguities:

• Blur Decomposition: Suffers from the loss of temporal ordering and motion ambiguity (e.g., multiple motion states can produce identical blurred averages).
• RS Temporal Super-resolution: Suffers from initial-state ambiguity (e.g., different starting poses can result in identical RS distortions) and a lack of complete global content.

Furthermore, existing methods rely heavily on synthetic data, leading to poor generalization in real-world scenarios due to unnatural artifacts in simulation.

2. Methodology

2.1. Dual-Shutter Imaging Setup

The core physical innovation is a dual Blur-RS setting that captures synchronized GS-blur and RS-distorted image pairs.

• Triaxial Imaging System: The authors constructed a hardware prototype using beam splitters to direct light to three cameras: an RS camera, a GS camera, and a High-Speed (HS) camera. This system captures spatially and temporally aligned Blur-RS pairs alongside high-speed ground truth (GT) frames.
• Stereo Extension: To address practical constraints (e.g., mobile devices where coaxial alignment is impossible), the framework is extended to a stereo Blur-RS configuration with a narrow baseline, allowing for performance-cost trade-offs.

2.2. Network Architecture

The proposed neural network operates in two sequential stages: Motion Interpretation and Frame Reconstruction.

A. Motion Interpretation (MI)

This stage disentangles motion into context-aware and temporally-sensitive representations using a Dual-Stream Motion Interpretation Module.

• Dual-Stream Design: The network processes the Blur and RS views in parallel branches.
  • The RS Branch focuses on fine-grained local details and resolving motion direction ambiguities.
  • The Blur Branch leverages global temporal integration to provide robust context for motion magnitude and alleviate initial-state ambiguity.
• Shutter Alignment and Aggregation (SAA): Instead of simple concatenation, this module uses bidirectional flow fields to warp features between the two streams. It employs Spatial Transformer Networks (STNs) to estimate coarse global transformations, ensuring features are aligned before aggregation. This facilitates mutual boosting between the branches.
• Temporal Positional Encoding (TPE): Specifically tailored for the RS branch, this encoding explicitly models the row-by-row readout time of the RS sensor, helping the network resolve temporal ordering uncertainties.
• Privileged Distillation: A teacher-student framework is used where a teacher module (with access to GT) guides the student. To prevent overfitting to privileged information, a region-adaptive distillation strategy is employed, using masks to focus supervision on highly dynamic areas, object boundaries, and low-confidence estimates.

B. Frame Reconstruction

• Self-Prompted Reconstruction: The frame generation module (GenNet) uses a motion-residue prompt. Instead of relying on external depth maps, it computes the difference between intermediate flows toward the Blur and RS views. This difference map highlights regions with complex motion or occlusion, acting as a self-prompt to guide the refinement of warped latent frames in critical areas.

2.3. Learning Objectives

The training combines:

1. Reconstruction Loss: L1/L2 loss between the recovered frames and GT.
2. Distillation Loss: Weighted by the adaptive masks to regularize the student's motion estimation.

3. Key Contributions

1. New Problem Setting: The paper introduces the dual Blur-RS setting as a unified framework to simultaneously address blur decomposition and RS temporal super-resolution, demonstrating that combining these modalities resolves the inherent ambiguities of each when used alone.
2. Real-World Dataset (realBR): The authors constructed the first real-world dataset containing spatially and temporally aligned GS-blur, RS-distorted, and high-speed ground-truth frames. This dataset (84 street scenes) supersedes traditional numerical simulations for training and evaluation.
3. Novel Algorithm: A dual-stream neural network featuring:
   • A Shutter Alignment and Aggregation (SAA) module for cross-view feature interaction.
   • Temporal Positional Encoding for RS views.
   • A Self-Prompted Frame Reconstruction mechanism based on motion residues.
   • Region-Adaptive Distillation to improve motion estimation accuracy.
4. Stereo Extension: The framework is extended to a stereo Blur-RS configuration, proving robustness to spatial misalignment and offering a practical solution for devices with baseline-separated cameras.

4. Experimental Results

The method was evaluated on the realBR dataset, the synthetic GOPRO-BR dataset, and a third-party testset with different sensors and capture parameters.

• Quantitative Performance: The proposed method ("Ours") significantly outperforms State-of-the-Art (SOTA) methods in both blur decomposition and RS temporal super-resolution tasks.
  • On realBR, it achieves ~31.31 PSNR (for 9 frames), surpassing the next best dual-view method (IFEDBR) by ~0.5 dB and single-modality methods by a large margin.
  • On GOPRO-BR, it achieves ~35.63 PSNR, outperforming all baselines including those adapted for dual inputs.
• Temporal Consistency: The method demonstrates superior temporal coherence (lower tOF and tLP metrics) compared to SOTA, effectively reducing flickering and discontinuities.
• Robustness:
  • Misalignment: The model maintains high performance even with spatial shifts (up to 8 pixels) and temporal misalignments between the Blur and RS inputs.
  • Low-light: It handles noisy RS inputs in low-light conditions better than competing methods.
  • Occlusion/Blur: It remains effective when the RS input itself contains motion blur or occlusions.
• Generalization: Models trained on realBR generalize well to synthetic and third-party data. Conversely, models trained solely on synthetic data (GOPRO-BR) fail to perform well on real-world data, highlighting the necessity of real-world training data.
• Stereo Performance: The stereo extension (narrow baseline) achieves competitive results, outperforming single-modality stereo baselines and existing cross-sensor methods, albeit with a slight performance drop compared to the perfectly aligned dual setup.

5. Significance and Claims

The paper claims to establish a new paradigm for realistic high-speed video reconstruction under complex motion degradations. Its significance lies in:

• Unifying Complementary Modalities: It demonstrates that GS blur and RS distortion are not just separate problems but complementary sources of information that, when jointly processed, resolve the ill-posed nature of motion inversion.
• Bridging the Synthetic-Real Gap: By providing a rigorous real-world dataset and a hardware setup, the work moves the field away from reliance on synthetic data, which often leads to poor generalization.
• Practical Applicability: The extension to stereo configurations and the demonstration of robustness to misalignment and noise suggest the approach is viable for real-world deployment, including mobile devices.
• Downstream Impact: The reconstructed latent frames significantly improve downstream tasks such as depth estimation, semantic segmentation, and visual SLAM, which are often degraded by motion artifacts in raw inputs.

The authors conclude that their unified framework, supported by novel hardware and algorithms, effectively "reenacts the imaging moment" regardless of motion-induced degradations, setting a new standard for high-speed video reconstruction.