Video Reconstruction using Diffusion-based Image-to-Video Generation with Trajectory Guidance

Technical Summary: Video Reconstruction using Diffusion-based Image-to-Video Generation with Trajectory Guidance

Problem Statement
This paper addresses the challenge of reconstructing missing or dropped frames in top-down drone footage of Autonomous Surface Vehicles (ASVs) performing maritime maneuvers. The specific context involves low-texture, open-water environments where objects (small vessels) are small relative to the field of view. In such scenarios, visual signals alone are often insufficient to determine accurate object motion, leading to failures in traditional interpolation methods that rely on predictable, smooth motion or purely data-driven priors. Furthermore, standard deep learning approaches often produce blurry outputs or fail to synthesize plausible textures when conditioned on sparse observations. The core difficulty lies in bridging the gap between sparse, non-visual telemetry data (GPS logs) and the generation of temporally coherent, high-fidelity video sequences without requiring domain-specific model fine-tuning.

Methodology
The authors propose a pipeline that converts raw GPS telemetry and a single reference video frame into a trajectory-guided video sequence using a pre-trained image-to-video diffusion model, specifically SG-I2V (built upon Stable Video Diffusion). The pipeline operates through four distinct stages:

1. Input Construction & Bounding Box Initialization:
   Given a reference frame, the user manually initializes bounding boxes for the vessels. While automated detection (e.g., YOLO) is possible, the authors opt for manual initialization to ensure reliable identity-consistent assignment of GPS trajectories to specific vessels, avoiding re-identification errors in close-proximity scenarios. Additionally, four corner bounding boxes are placed to encode global camera motion (held constant in this specific experiment as the drone was stationary).

2. GPS-to-Pixel Projection:
   A custom mapping function converts geographic coordinates into image-space motion cues. This process involves:

   • Geographic-to-Metric Transform: Converting GPS fixes (longitude, latitude) into local East-North metric offsets using an equirectangular approximation.
   • Rotation: Aligning the metric frame with the image axes using an estimated camera yaw (derived from vessel Course Over Ground and visual inspection).
   • Scale Estimation: Calculating a pixel-per-meter scale ($s$) by comparing the metric distance between vessels (derived from GPS) against their pixel distance in the reference frame.
   • Per-Frame Projection: Interpolating GPS logs to video timestamps and projecting them into pixel space to generate trajectory arrays ($T_i$) for each vessel.

3. Trajectory-Conditioned Generation:
   The constructed payload—consisting of the reference image and per-object trajectory arrays—is fed into the SG-I2V model. The model generates the subsequent 14 frames conditioned on both the visual content and the embedded trajectory guidance. Notably, the authors apply no domain-specific fine-tuning to the pre-trained model, relying instead on a systematic ablation study of hyperparameters (latent optimization iterations, heatmap spread, FFT frequency blend) to optimize visual quality and temporal coherence.

4. Evaluation Baselines:
   The proposed method is benchmarked against:

   • Optical Flow Extrapolation: Warping the reference frame forward using Farneback optical flow.
   • RIFE (Real-Time Intermediate Flow Estimation): A deep learning interpolation method that conditions on both temporal endpoints (which is an advantage not available in true missing-frame scenarios).

Key Contributions
The paper outlines three primary contributions:

1. GPS-to-Image Projection Scheme: A method to convert sparse tracking data into spatial motion cues that are directly consumable by off-the-shelf generative models.
2. Zero-Fine-Tuning Pipeline: A complete reconstruction workflow that achieves physically plausible synthesis without retraining the underlying diffusion model on maritime data.
3. Systematic Benchmarking: A quantitative evaluation on real maritime footage demonstrating the advantages of generative trajectory-guided synthesis over traditional interpolation and extrapolation methods, particularly under challenging small-object conditions.

Results
The evaluation compares the methods using perceptual (LPIPS), temporal (smoothness), no-reference quality (BRISQUE), and trajectory accuracy metrics against ground truth:

• Perceptual Quality: SG-I2V produced the most naturally appearing frames among synthesis methods, with a BRISQUE score of 25.52, closest to the ground truth score of 23.64. In contrast, RIFE scored 42.46, indicating over-smoothed outputs.
• Temporal Consistency: SG-I2V achieved a temporal smoothness of 1.14 px/frame, closely matching the ground truth of 1.42. Optical flow (0.39) and RIFE (0.59) underestimated motion magnitude.
• Trajectory Adherence: SG-I2V demonstrated the strongest adherence to GPS trajectories with an error of 9.31 pixels. This is significantly lower than the ground truth video's own alignment error of 28.70 pixels (attributed to the approximate manual temporal alignment between footage and logs), indicating the model successfully incorporated GPS-derived motion cues.
• LPIPS: While Optical Flow achieved the lowest LPIPS (0.110), the authors note this reflects the preservation of original image statistics via warping rather than genuine perceptual quality generation.

Significance and Claims
The paper claims that trajectory-guided diffusion synthesis is a viable approach for maritime video reconstruction, specifically in low-texture, small-object conditions where visual signals are ambiguous. The significance lies in the ability to steer video synthesis toward physically plausible outcomes using auxiliary sensor data (GPS) without the need for expensive, domain-specific model retraining.

The authors modestly acknowledge limitations, noting that the evaluation is restricted to a single two-second clip, which limits the generalizability of the quantitative findings. They do not claim universal superiority over all video reconstruction tasks but highlight the specific efficacy of their approach in scenarios where object motion is known via telemetry but visually difficult to track. Future work is proposed to extend the pipeline to longer sequences, automate the synchronization and bounding box initialization, and incorporate Speed-Over-Ground (SOG) data to encode displacement magnitude more accurately.