🔢 mathematics

Single-View Rolling-Shutter SfM

This paper addresses the challenge of rolling-shutter structure-from-motion by characterizing single-view geometry to systematically derive minimal reconstruction problems for recovering motion and scene parameters from a single image, while evaluating their feasibility and limitations through proof-of-concept solvers.

Original authors: Sofía Errázuriz Muñoz, Kim Kiehn, Petr Hruby, Kathlén Kohn

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sofía Errázuriz Muñoz, Kim Kiehn, Petr Hruby, Kathlén Kohn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are taking a photo with your smartphone while running down the street. If you use a standard camera, the whole picture is captured at the exact same instant. But most modern phones use Rolling Shutter cameras. Instead of freezing the whole scene at once, they scan the image like a curtain falling, line by line, from top to bottom.

If you are moving while this "curtain" falls, things get weird. A straight pole might look bent like a banana. A spinning fan might look like a wobbly jelly. A person running might look like they have three legs because their body was captured at three different moments as the shutter passed over them.

This paper is about fixing the math behind these weird, distorted images so we can figure out exactly how the camera was moving and what the 3D world looked like, even if we only have one single photo.

Here is the breakdown of their work using simple analogies:

1. The Problem: The "Moving Curtain"

Think of a Rolling Shutter camera like a scanner at a library.

Global Shutter (Old cameras): The scanner flashes a light over the whole book instantly. You get a perfect snapshot.
Rolling Shutter (Phones): The scanner moves down the page slowly. If the book is being flipped while the scanner moves, the text gets stretched, squished, or duplicated.

The authors ask: If we see a distorted, wobbly line or a duplicated point in a single photo, can we work backward to figure out how fast the camera was moving and where the objects actually were?

2. The Solution: Finding the "Fingerprint" of Motion

The team realized that these distortions aren't random chaos; they follow strict mathematical rules. They treated the camera's motion like a polynomial recipe (a specific type of math formula).

The "Order" of the Camera: They proved that a world point (like a streetlight) doesn't just appear once in a distorted photo. Depending on how fast the camera is spinning or moving, that streetlight might appear twice, three times, or more in the same image. They calculated exactly how many times it should appear based on the camera's speed. It's like knowing that if you spin a fan fast enough, you'll see 3 blades instead of 1.
The "Bent" Lines: They discovered that a straight line in the real world (like a building edge) always turns into a specific type of curved line in the photo. They mapped out exactly what kind of curve it becomes based on the camera's motion. It's like knowing that if you drag a stick through water at a certain speed, the wake it leaves will always be a specific shape.

3. The "Minimal Problems": The Puzzle Pieces

In computer vision, a "Minimal Problem" is like a puzzle: What is the absolute minimum amount of information I need to solve the mystery?

The authors created a catalog of these puzzles. They asked:

"If I see one straight line that looks like a curve, can I figure out the camera's speed?" (Sometimes yes, sometimes no).
"If I see two points that appear twice, can I solve it?"
"If I see three lines, do I have enough info?"

They systematically listed every possible combination of points and lines that allows a computer to solve the 3D scene from a single photo. They even calculated how many different answers (solutions) the computer might find for each puzzle.

4. The Analogy: The "Ghostly" Detective

Imagine you walk into a room and see a ghostly trail of a person who moved quickly. You can't see the person, only the blur.

The Old Way: You'd need a video (multiple photos) to see the person move and guess their speed.
The New Way (This Paper): The authors say, "Wait! The shape of that ghostly trail tells us exactly how fast they were running and which direction they were going, even if we only have one photo."

They built "solvers" (mathematical detectives) that look at the shape of the blur (the curve) or the number of ghosts (duplicated points) and instantly calculate the camera's movement.

5. Real-World Testing

They didn't just do math on paper. They tested their "detectives" on:

Synthetic Data: Computer-generated worlds where they knew the exact answer.
Real Videos: Footage from iPhones and other cameras.

The Results:

When the motion was simple and the scene had clear lines (like a hallway or a street), their method worked surprisingly well.
It was a bit sensitive to "noise" (like grainy photos), but it proved that single-image 3D reconstruction is possible for rolling shutter cameras.

Why Does This Matter?

This is a building block for future technology.

Self-driving cars: They need to understand their speed and the road layout instantly, even if their cameras are rolling shutters.
Augmented Reality (AR): When you point your phone at a room to place a virtual chair, the phone needs to know exactly how it's moving to keep the chair from sliding around.
Robotics: Robots navigating through warehouses need to build a 3D map of the world from a single glance.

In a nutshell: This paper took the messy, distorted images from our phones and figured out the secret mathematical code to turn those "wobbly" photos back into accurate 3D maps, using just a single snapshot. They turned the "bug" of the rolling shutter into a "feature" for measuring motion.

1. Problem Statement

Rolling-Shutter (RS) cameras are ubiquitous in consumer devices (smartphones, drones) due to their low cost and high frame rates. Unlike Global Shutter (GS) cameras, RS sensors capture images line-by-line over time. If the camera moves during capture, this induces geometric distortions:

Multiple Projections: A single 3D world point may appear multiple times in the same image.
Non-linear Curves: Straight 3D world lines are projected as non-linear curves in the image.

These distortions make standard Structure-from-Motion (SfM) and camera pose estimation algorithms inaccurate. While multi-view RS solvers exist, they often rely on restrictive assumptions (e.g., planar scenes, IMU data, or consecutive frames). Single-view RS SfM (reconstructing motion and scene from a single image) remains largely unsolved for general motion models. This paper aims to systematically explore the theoretical limits and derive minimal solvers for single-view RS SfM using world points and lines.

2. Methodology and Camera Model

The authors propose a rigorous algebraic geometry approach to characterize RS imaging.

Camera Model:
- The camera center $C(x)$ and orientation $R(x)$ vary polynomially with the scanline position $x$ .
- Rotation: Modeled using the Cayley transform (rational parameterization of $SO(3)$) to ensure the motion model remains polynomial, which is crucial for algebraic solvers.
- Degrees: $d$ is the degree of the polynomial for camera center motion; $\delta$ is the degree for the quaternion parameters of rotation.
Geometric Characterization:
- Order of the Camera: The paper proves that a generic RS camera of degree $(d, \delta)$ projects a world point $1 + d + 2\delta$ times. This "order" is a fundamental invariant.
- Image of Lines: The projection of a 3D line is a rational irreducible curve of degree $D = 1 + d + 2\delta$ . These curves pass through the point at infinity on the y-axis $D-1$ times.
- Constraints: The authors derive that these image curves lie in a specific projective subspace $\mathcal{H}_D$ and satisfy specific linear and non-linear constraints depending on the motion parameters.

3. Key Contributions

The paper makes three primary contributions:

A. Theoretical Characterization of RS Geometry

Theorem 1 & 2: Formally prove the order of the camera and the degree/type of curves formed by world lines. This establishes the algebraic foundation for all subsequent solvers.
Ambiguity Analysis: Identifies inherent ambiguities, such as the "line direction ambiguity" (motion along the direction of a parallel line cannot be recovered from that line's image alone).

B. Systematic Derivation of Minimal Problems

The authors systematically enumerate minimal reconstruction problems (problems where the number of constraints equals the degrees of freedom, yielding a finite number of solutions). They categorize these based on the type of primitive (points vs. lines) and motion model:

Line-based SfM (Section 4):
- Pure Rotation ( $d=0, \delta>0$ ): Derives solvers for pure rotational motion. For $\delta=1$ , a single line image curve uniquely determines the camera pose (analogous to the essential matrix).
- Pure Translation ( $d>0, \delta=0$ ): Shows that for $d=1$ , line images are in one-to-one correspondence with world lines. For $d>1$ , the set of image curves forms a subvariety that determines the camera.
- Combined Motion ( $d>0, \delta>0$ ): Derives minimal problems for general motion, listing solution counts (degrees) for various configurations (e.g., $d=1, \delta=1$ yields 20 solutions).
Point-based SfM (Section 5):
- Exploits the fact that points appear multiple times.
- Derives balanced problems where $p$ points are observed $k$ times.
- Highlights that for $d \ge 2$ , the multiple projections of points satisfy linear constraints (Lemma 1), reducing the effective number of constraints.

C. Practical Solvers and Evaluation

Solver Implementation: Implemented minimal solvers using homotopy continuation (via the MiNuS library).
Noise Robustness: Demonstrated that while single-line solvers are sensitive to noise, using multiple lines with fewer points per line (e.g., 3 lines with 3 points each) significantly improves robustness.
Real-World Validation: Tested on:
- Synthetic Data: Validated noiseless stability and noise robustness.
- Real Datasets: Used the iPhone 3GS sequence (pure rotation) and a sequence with parallel/coplanar lines (pure translation).
- Results: The proposed solvers outperform existing approximation-based methods (like LAAA) and achieve acceptable accuracy for scenes with simple motion and regular structures.

4. Key Results

Minimal Problems Table: The paper provides a comprehensive table (Table 1 in the paper) listing all discovered minimal problems, their degrees (number of solutions), and required inputs (e.g., number of lines and points per line).
Degrees of Freedom:
- For pure rotation ( $\delta=1$ ), a single line curve with 5 points yields 10 solutions.
- For pure translation ( $d=1$ ), 2 parallel lines with 4 and 3 points respectively yield 2 solutions.
Performance:
- Noiseless: Solvers are stable.
- Noisy: Solvers with $d=1, \delta=0$ achieve velocity errors $<20^\circ$ in ~23% of cases and $<40^\circ$ in ~45% of cases.
- Real-world: On the iPhone 3GS dataset, the $\delta_1(4,3)$ solver achieved axis errors $<40^\circ$ in 33% of cases, outperforming the LAAA baseline.

5. Significance and Future Work

Foundational Theory: This work provides the first complete algebraic characterization of single-view RS geometry for arbitrary polynomial motion models, moving beyond approximations.
Algorithmic Blueprint: By deriving minimal problems, the authors provide the building blocks for robust, general-purpose RS SfM pipelines (e.g., within RANSAC frameworks).
Limitations & Future Directions:
- Current solvers rely on polynomial motion; future work could explore rational motion models.
- The paper focuses on single-view; extending this to multi-view RS SfM is a natural next step.
- Combining points and lines in a single view remains an open problem.
- Improving numerical stability and runtime for real-time applications is necessary.

In summary, this paper bridges the gap between theoretical algebraic geometry and practical computer vision, offering a systematic framework to solve the challenging problem of single-view structure and motion estimation for rolling-shutter cameras.