💻 computer science

Rolling Shutter Relative Pose Estimation Made Practical

This paper introduces a practical rolling shutter relative pose estimation method that utilizes affine correspondences and novel RS-corrected constraints to solve for pose and motion from just seven correspondences in 1.2ms, achieving state-of-the-art accuracy on both rolling and global shutter datasets.

Original authors: Daniel Barath

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daniel Barath

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are taking a photo with your smartphone. Most modern cameras don't take a picture all at once like a flash going off. Instead, they scan the image from top to bottom, line by line, very quickly. This is called a Rolling Shutter.

If you are standing still, this works fine. But if you are moving the camera quickly (like on a drone or a running drone), the top of the photo sees the world from one angle, and by the time the camera scans the bottom, you've moved, so the bottom sees the world from a slightly different angle. This causes the image to look "jello-like" or warped.

The Problem: The "Jello" Math is Too Hard

To build maps, navigate drones, or create Augmented Reality, computers need to figure out exactly how the camera moved between two photos. This is called Relative Pose Estimation.

For standard cameras (Global Shutter), this math is easy and fast. But for Rolling Shutter cameras, the math gets messy because the "rules" of geometry change for every single line of the image.

The previous best method to solve this "jello" math was incredibly inefficient. It was like trying to find a specific needle in a haystack, but the haystack was the size of a mountain.

The Old Way: To solve the math, the computer needed to look at 20 matching points between two images.
The Consequence: Because it needed so many points, the computer had to try millions of random combinations to find the right answer. This took too long and was often too slow for real-world use.

The Solution: Adding "Shape" to the Points

The authors of this paper found a clever shortcut. Instead of just looking at where a point is (like a dot), they looked at the shape of the patch around that point.

Think of it this way:

Old Method (Point Correspondence): You see a red dot in the left photo and a red dot in the right photo. You match them. (1 piece of info).
New Method (Affine Correspondence): You see a red dot, but you also notice that the patch around it is stretched, squished, or tilted. You match the dot and the shape of the stretch. (3 pieces of info).

By using these "shape-aware" matches (called Affine Correspondences), the computer gets much more information from each match.

The Breakthrough: Doing More with Less

Because each new "shape-aware" match gives three times more information, the authors could build a new math solver that only needs 7 matches instead of 20.

The Analogy: Imagine you are trying to guess a secret code.
- The old method asked you to guess 20 numbers to get the code right. You had to try millions of combinations.
- The new method asks you to guess only 7 numbers, but each number is a "super-number" that tells you three things at once. You only need to try a few hundred combinations.

What They Achieved

Speed: By reducing the number of matches needed from 20 to 7, the computer doesn't have to try millions of guesses. It drops from taking forever to taking just 1.2 milliseconds to solve the math.
Accuracy: On real-world data (like the TUM dataset), their method was the most accurate at figuring out the camera's rotation and position.
Velocity: A unique bonus is that their method is very good at estimating how fast the camera is moving (translational velocity). Previous methods were terrible at this because the math got "confused" between where the camera was and how fast it was going. The new method clears up this confusion.
Versatility: Even when they tested it on standard cameras that don't have this "jello" effect (Global Shutter), the method still worked perfectly, proving it's robust.

Summary

The paper introduces a new way to calculate camera movement in "jello" images. By using extra information about the shape of image patches, they reduced the heavy math workload from a "mountain" to a "hill." This makes it practical to use these calculations in real-time applications like drones and smartphones, solving a problem that was previously too slow and expensive to use reliably.

Technical Summary: Rolling Shutter Relative Pose Estimation Made Practical

Problem Statement
Rolling shutter (RS) cameras are ubiquitous in consumer devices, yet robust relative pose estimation for these sensors has remained impractical. Standard multi-view geometry methods assume a global shutter (GS) model where the essential matrix is constant. In RS cameras, the row-by-row readout invalidates this assumption, making the essential matrix row-dependent. While Dai et al. [17] derived a minimal solver for RS relative pose, it requires a minimum of 20 point correspondences (PCs). Due to the exponential dependence of RANSAC iteration counts on sample size ( $\eta^{-k}$ ), a sample size of 20 renders robust estimation prohibitively expensive (requiring millions of iterations at 50% inlier ratios). Existing methods either ignore RS effects (introducing systematic bias) or are too computationally heavy for practical use.

Methodology
The paper introduces Affine Correspondences (ACs) to the RS two-view geometry problem to reduce the required sample size. An AC augments a point match with a local $2 \times 2$ affine transformation, providing three equations per correspondence (one epipolar, two affine) rather than just one.

RS-Corrected Affine Constraints:
The authors derive novel constraints that account for the coupling between point perturbations and the row-dependent essential matrix. Unlike the GS formulation, shifting a point in an RS image changes its readout row, altering the essential matrix. The derivation includes correction terms involving the partial derivatives of the essential matrix with respect to the row coordinates ( $\partial \tilde{E} / \partial \tau_k$ ). Each AC yields three equations: the standard epipolar constraint plus two new affine constraints modified by these RS correction terms.
Linearization and Null-Space Projection:
The RS two-view problem involves 17 unknowns: 5 for relative pose ( $R, t$ ) and 12 for per-camera angular and translational velocities ( $\omega_k, v_k$ ). The authors observe that RS motion parameters are physically small (typically $\|\omega\| < 0.15$ rad). They linearize the constraints around the GS solution ( $\theta = 0$ ), treating the RS parameters as linear variables.
With 7 ACs, the system provides 21 linear equations for the 12 RS parameters. The solver eliminates these 12 unknowns via null-space projection. By projecting the residual vector onto the left null space of the Jacobian, the system reduces to 9 polynomial equations involving only the 5 pose parameters.
Algebraic Solver:
The pose parameters are parameterized using Cayley rotation and a scaled translation vector. The resulting system is reduced to 10 polynomial equations of degree 4. The algebraic degree of the system is 20. The authors solve this using action matrices (computational algebraic geometry), extracting all 20 solutions in approximately 1.2 ms. The RS parameters are then recovered via least squares.
RANSAC Integration:
The solver is integrated into a RANSAC pipeline (SupeRANSAC). The reduction of the minimal sample size from 20 PCs to 7 ACs drastically reduces the expected number of RANSAC iterations (e.g., from $>10^6$ to $\sim600$ at 50% inliers). A non-minimal refinement step using Levenberg-Marquardt optimizes all 17 parameters jointly, utilizing both epipolar and affine residuals.

Key Contributions

Novel Constraints: Derivation of RS-corrected affine constraints that capture the coupling between point perturbations and row-dependent essential matrices, a coupling absent in GS formulations.
7-AC Minimal Solver: Development of a linearized algebraic solver that estimates pose and RS motion from only 7 ACs. This reduces the RANSAC sample size from 20 to 7, making robust estimation computationally feasible.
Translational Velocity Estimation: Demonstration that the proposed method provides accurate translational velocity estimates, which are poorly conditioned in point-only methods due to $v$ - $t$ coupling.
Generalization: Validation that the solver generalizes to global shutter settings without performance degradation, correctly recovering near-zero RS parameters.

Experimental Results

Synthetic Data: The solver achieves pose accuracy competitive with the 44-PC solver (which uses significantly more data) while using 6 $\times$ fewer correspondences. Crucially, it achieves $\sim3\times$ lower error in translational velocity estimation compared to point-based baselines.
TUM-RS Benchmark: On real-world RS data, the method achieves the best pose accuracy among all tested methods (including GS baselines and RS-20PC/44PC solvers). For example, with AffNet features, it achieves an AUC@5 $^\circ$ of 0.502, outperforming the GS 5-point solver (0.428) and the RS-44PC solver (0.267). It also provides the most accurate RS parameter estimates.
EuRoC MAV (Global Shutter): When tested on global shutter data, the solver performs comparably to standard GS solvers and correctly estimates near-zero RS parameters, confirming it does not introduce artifacts in the absence of RS distortion.
Runtime: While the algebraic solver takes $\sim1.2$ ms (slower than microsecond GS solvers), the drastic reduction in RANSAC iterations results in a total pipeline runtime (3.7s–7.5s) that is practical and significantly faster than the 20/44-PC RS solvers (12s–46s).

Significance
The paper claims to make RS-aware relative pose estimation practical for the first time. By leveraging affine correspondences, it bridges the gap between the theoretical requirements of RS geometry and the computational constraints of robust estimation. The method enables accurate reconstruction and motion estimation on consumer devices with rolling shutters, addressing a long-standing limitation where RS solvers were either too slow (due to large sample sizes) or too inaccurate (due to ignoring RS effects). Furthermore, it uniquely provides reliable estimates of translational velocity, a parameter previously difficult to recover from point correspondences alone.