💻 computer science

RGLStab: A Robust Online Video Stabilization Framework Based on Grid Flow and LSTM

This paper proposes RGLStab, a robust online video stabilization framework that combines structured grid optical flow with a unidirectional LSTM network to achieve real-time, high-fidelity stabilization for aircraft camera footage by effectively addressing feature tracking failures and dynamic foreground interference while suppressing local distortions.

Original authors: Yimin Zhi, Junfeng Jing, Meng Jiang, Yongzhe Wang, Xinyao Li

Published 2026-10-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yimin Zhi, Junfeng Jing, Meng Jiang, Yongzhe Wang, Xinyao Li

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

When a camera is mounted on a flying machine, the video it captures is rarely still. The air itself is a turbulent river, pushing and pulling the aircraft, while the spinning rotors and rapid maneuvers add their own violent shakiness. This jitter turns a clear view of the world into a chaotic blur, making it difficult for a human operator to track a target or for a computer to recognize what it is seeing. For decades, engineers have tried to fix this by smoothing out the video after it has been recorded, a process that requires waiting for the entire clip to finish before starting work. However, for a drone flying a mission in real time, waiting is not an option; the video must be stabilized the instant it is captured, with no delay and no look-ahead into the future. The challenge lies in doing this quickly enough to keep up with the flight while ensuring the picture does not warp or stretch into something unrecognizable.

Researchers at Xi'an Polytechnic University have developed a new system called RGLStab to solve this specific problem. Their approach treats the video stabilization task as a two-part puzzle: first, figuring out exactly how the camera is moving, and second, predicting a smooth path for the image to follow without ever peeking at the next frame. Traditional methods often rely on tracking specific dots or corners in the image, but when a drone spins quickly or the light changes, these dots disappear or move unpredictably, causing the system to fail. Other modern methods use complex, heavy computer networks that can handle the motion but are too slow for real-time use or tend to twist the image in unnatural ways, making buildings lean or trees bend like rubber.

To overcome these hurdles, the team designed a framework that looks at the movement of the entire image grid rather than just a few scattered points. They use a sophisticated tool to measure how every pixel shifts between frames, creating a dense map of motion. Instead of trying to process this massive amount of data all at once, which would slow everything down, they sample this motion at regular intervals, like taking a snapshot of the flow at specific grid points. They then use a mathematical filter to separate the true movement of the camera from the chaotic noise caused by wind or vibration, extracting a clean, four-part description of the camera's motion: how far it moved left or right, how far it moved up or down, how much it rotated, and how much it zoomed in or out.

Once the system knows the camera's raw, shaky movement, it must decide how to smooth it out. Most smoothing algorithms look at the future to know how to handle the present, but a real-time system cannot do that. The researchers solved this by training a specialized type of computer memory, known as a long short-term memory network, to learn the physical laws of how a camera stabilizer works. This network acts like a seasoned pilot who has flown thousands of hours; it remembers the history of the flight and the current moment to predict the next smooth movement without needing to see what comes next. It learns to ignore the sudden, high-frequency bumps caused by the engine or the wind while keeping the natural, slower movements of the flight path intact.

A critical part of their success was teaching the computer what mistakes to avoid. In video stabilization, a small error in calculating a rotation can cause the edges of the image to swing wildly, creating large black borders or cutting off the view. The researchers created a specific set of rules for the computer to follow during its learning process, punishing it heavily if it tried to rotate the image too much, while being more lenient with simple side-to-side shifts. This ensures that the final video stays straight and upright, preserving the full field of view without the need to crop out the edges. The result is a video that looks steady and natural, as if the camera were mounted on a perfectly smooth gimbal, even when the drone is performing sharp turns or flying through rough air.

The team tested their system on a collection of challenging video clips featuring rapid rotations, intense zooming, and crowded scenes. They found that their method could process video at a speed of 24.51 frames per second, which is fast enough to keep up with live flight operations. In terms of quality, the stabilized video was nearly as smooth as videos processed by offline methods that have the luxury of time and future information, but without the delay. Crucially, the system avoided the common pitfalls of other real-time solutions: it did not introduce the "jello" effect where vertical lines wobble, nor did it distort the shapes of buildings or landscapes. By balancing the need for speed with the need for geometric accuracy, the researchers have provided a practical solution that allows drones to see the world clearly, even when the air around them is anything but calm.

Technical Summary: RGLStab – A Robust Online Video Stabilization Framework

1. Problem Statement

Video sequences captured by airborne cameras (e.g., drones) are frequently compromised by external environmental disturbances, mechanical rotor vibrations, and rapid maneuvers. These factors introduce nonlinear visual oscillations that degrade operator observation and hinder downstream computer vision tasks.

Existing stabilization technologies face significant trade-offs:

Traditional Feature-Based Methods: Rely on sparse, handcrafted local features. They often fail under rapid rotation, translation, or dynamic blurring due to insufficient feature points and unstable matching, leading to trajectory noise or compensation deviations.
Deep Learning-Based Methods: While dense optical flow and grid deformation methods offer better resilience to complex jitter, they suffer from high computational overheads (inference latency) and often introduce local geometric distortions (non-rigid artifacts) when applying localized nonlinear deformations to eliminate parallax.
Offline vs. Online: Most high-quality methods are offline, relying on future frames (non-causal) which introduces unacceptable latency for real-time applications. Online methods that strictly adhere to causality often struggle to balance real-time performance with geometric fidelity and robustness against feature loss.

2. Methodology

The proposed RGLStab framework addresses these challenges through a three-stage pipeline designed for strict online, causal processing without reliance on future frames.

2.1 Motion Estimation: Grid Flow and RANSAC

Instead of relying on sparse feature points or computationally expensive dense pixel-level processing for the entire frame, RGLStab employs a hybrid approach:

Dense Flow Extraction: It utilizes the RAFT (Recurrent All-Pairs Field Transforms) architecture to generate high-quality, pixel-level dense optical flow fields, ensuring robustness even in low-texture or rapidly moving scenarios.
Grid Sampling: To reduce computational complexity, the high-dimensional flow field is downsampled via structured grid sampling, extracting motion vectors only from grid vertices.
Geometric RANSAC: The sampled motion vectors are processed using the RANSAC algorithm to filter dynamic outliers and estimate a global four-degree-of-freedom (4-DOF) similarity transformation matrix. This matrix includes translation ( $t_x, t_y$ ), rotation ( $\theta$ ), and uniform scaling ( $s$ ). This step enforces rigid geometric constraints, preventing the local non-rigid distortions common in grid-deformation methods.

2.2 Trajectory Smoothing: Causal Unidirectional LSTM

To achieve real-time smoothing without future frame dependencies, the framework employs a fully causal temporal model based on a unidirectional Long Short-Term Memory (LSTM) network.

Input: The network receives the current 4-DOF physical state vector ( $x_t = [t_x, t_y, \theta, s]^T$ ) and historical hidden states.
Mechanism: Through gating mechanisms (forget, input, and output gates), the LSTM learns the kinematic laws of the physical stabilizer. It effectively separates high-frequency mechanical tremors from the intended camera trajectory using only past and current observations.
Output: The model predicts a smooth pose sequence, ensuring zero-latency inference relative to the input stream.

2.3 Decoupled Hybrid Loss Function

To guide the LSTM in learning optimal stabilization without sacrificing field of view (FOV), a decoupled weighted loss function is introduced. Unlike standard losses that treat all motion components equally, this function applies specific penalties:

Translation ( $\lambda_{trans} = 1.0$ ): Anchors the global trajectory.
Rotation ( $\lambda_{rot} = 2.0$ ): Assigns a higher penalty to rotation errors. Since rotational jitter causes significant leverage effects at image edges (leading to black borders and FOV loss), this high weight forces the network to minimize rotational deviations.
Scaling ( $\lambda_{scale} = 1.0$ ): Prevents unnatural zoom artifacts.
Norm Selection: The L1 norm is used instead of L2 to provide steady linear gradients and mitigate instability caused by sudden mechanical impacts.

2.4 Frame Synthesis

The final stage reconstructs the video frame using the global 4-DOF similarity transformation. By treating the image as an incompressible rigid body and applying a unified linear algebraic mapping, the system avoids the "Jello effect" and local distortions associated with non-rigid grid warping. This ensures high geometric fidelity and maximizes the retained effective FOV.

3. Key Contributions

RGLStab Framework: An online video stabilization framework integrating structured grid-based optical flow (RAFT + Grid Sampling) with temporal networks (LSTM) to balance computational efficiency and robustness.
Robust Motion Extraction Strategy: A novel strategy combining RAFT dense flow grid sampling with RANSAC to derive a global 4-DOF similarity matrix. This effectively suppresses local distortions and handles complex maneuvers where sparse feature tracking fails.
Causal Temporal Smoothing: A fully causal LSTM model that performs trajectory smoothing based solely on historical memory and current observations, achieving low-latency prediction without future frame dependence.
Decoupled Optimization: A mixed loss function with weighted penalties (specifically high weight on rotation) that effectively eliminates high-frequency tremors while preserving geometric proportions and maximizing the usable field of view.

4. Experimental Results

The framework was evaluated on the NUS Stabilization Benchmark Dataset across five scenarios: Regular, Fast Rotation, Intense Zooming, Crowd, and Parallax.

Quantitative Performance:
- Stability: RGLStab achieved a Stability Score of 0.8768 (average), rivaling offline methods and outperforming other online baselines like Meshflow (0.8617) and StabNet (0.8338).
- Distortion: It achieved a Distortion score of 0.9788, significantly outperforming Meshflow (0.8447) and maintaining high geometric fidelity comparable to offline methods like PWStableNet (0.9803).
- Cropping Ratio: The method retained a high Field of View with an average cropping ratio of 0.9423, superior to most online methods and indicating minimal loss of effective image area due to black borders.
Computational Efficiency:
- RGLStab achieved an online inference frame rate of 24.51 FPS on an NVIDIA RTX 2080Ti.
- This outperforms other online deep learning methods like NNDVS (14.63 FPS) and StabNet (1.08 FPS), while maintaining higher geometric fidelity than ultra-lightweight sparse methods like SuperPoint (30.67 FPS but lower stability in complex scenarios).
Qualitative Analysis: Visual comparisons demonstrated that RGLStab effectively suppresses local nonlinear deformations (Jello effect) and avoids the aggressive cropping and black borders seen in competing online methods (StabNet, SuperPoint) during fast rotations.

5. Significance and Claims

The paper claims that RGLStab provides a highly practical and engineering-oriented solution for real-time-sensitive tasks on airborne platforms. Its significance lies in:

Breaking the Trade-off: It successfully balances real-time performance (low latency) with geometric fidelity, a challenge where existing online methods often fail.
Robustness in Extreme Conditions: By moving away from sparse feature matching to grid-sampled dense flow, it overcomes tracking failures during rapid rotation and dynamic blurring.
Causal Efficiency: It achieves stabilization quality comparable to some offline global algorithms while strictly adhering to causality, making it suitable for mission-critical applications like aerial patrol and disaster reconnaissance where future frame data is unavailable.
Geometric Integrity: The use of rigid 4-DOF constraints and rotation-penalized loss functions ensures that the stabilized video maintains natural geometric proportions and maximizes the effective field of view, avoiding the artifacts common in non-rigid deformation approaches.

The authors acknowledge limitations, noting that the 4-DOF similarity model cannot fully correct for rolling shutter distortion or complex perspective warping caused by significant depth disparities during aggressive 3D maneuvers, representing an engineering trade-off for real-time efficiency.