💻 computer science

Scanline-Aware Animatable Gaussian Avatars from Rolling-Shutter Videos

This paper introduces RS-Avatar, a method that reconstructs sharp, animatable 3D Gaussian avatars from rolling-shutter videos by replacing standard motion-blur rendering with a scanline-aware compositing operator that correctly accounts for the sequential readout of different body parts within a single frame.

Original authors: Youxiang Wang

Published 2026-08-19

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Youxiang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of digital imaging, there is a quiet assumption that every photograph captures a single, frozen moment in time. When a camera takes a picture, it is generally believed that the entire scene—from the top of a person's head to their feet—is recorded at the exact same instant. This belief underpins much of modern computer vision, the field dedicated to teaching machines to see and understand the world. It allows researchers to build complex 3D models of people, known as avatars, which can be animated and viewed from any angle. These digital doubles are becoming essential for everything from virtual telepresence and film production to sports analysis and artificial intelligence. However, this assumption is almost never true in the real world. The vast majority of cameras, from the smartphones in our pockets to the high-speed sensors in professional studios, do not capture an image all at once. Instead, they scan the image line by line, top to bottom, in a fraction of a second. This method, called a rolling shutter, means that by the time the camera finishes reading the bottom of the frame, the top has already been recorded a few milliseconds earlier. For a person moving their arms or walking, those few milliseconds are enough for their body to shift position significantly, creating a subtle but pervasive distortion where different parts of the body are captured at different times.

For years, researchers trying to build these 3D avatars from video have largely ignored this timing mismatch, treating every frame as if it were a perfect snapshot. The result has been a digital ghost: a 3D model that looks correct when viewed from the same angles as the original video, but falls apart when the viewer tries to look at it from a new angle or move the avatar into a new pose. The distortion gets baked into the model's shape, causing limbs to appear wobbly, sheared, or unnaturally thick. A researcher has now addressed this problem directly, not by trying to fix the video before building the model, but by teaching the model itself how to understand the camera's scanning behavior. They developed a new system that reconstructs sharp, undistorted 3D avatars directly from rolling-shutter video, acknowledging that every single line of pixels in an image represents a slightly different moment in time.

The core of this new approach, which the researcher calls RS-Avatar, relies on a simple but powerful shift in perspective. Instead of asking the computer to guess what the person looked like at a single moment, the system accepts that the person was moving throughout the entire duration of the scan. The software builds a model of the body's motion that is precise enough to know where every part of the body was at any given fraction of a second. When it creates the final image, it does not average the different positions together, which would result in a blur. Instead, it acts like a meticulous editor, selecting the exact position of the body for the top line of pixels, a slightly different position for the next line, and so on, matching the specific time each line was captured by the camera. This process is distinct from how motion blur works; while a blur is like mixing all the colors of a moving object into a single smear, a rolling shutter is more like a precise slicing of time, where each slice holds a clear, sharp image of the body at a specific instant.

To test their idea, the researcher created a new benchmark using data from a standard motion-capture studio. They took high-quality recordings of people moving and artificially simulated the rolling-shutter effect, creating a dataset where the true, sharp answer was known. They then compared their new method against several existing approaches. One common strategy is to try to "fix" the video first, using software to straighten the distorted lines before building the 3D model. Another strategy is to use a model designed for motion blur, which assumes the camera averaged the motion over time. The results were clear and surprising. The method that tried to fix the video first improved the picture slightly, but the resulting 3D models were still flawed because the correction software could not guarantee that the different camera angles agreed with each other in three-dimensional space. The motion-blur model performed even worse, actually producing lower-quality results than simply ignoring the shutter problem entirely. This happened because the math used to handle blur is fundamentally different from the math needed for rolling shutter; the blur model tries to guess the direction of movement, while the rolling shutter already provides that direction through the order in which the lines are scanned.

The new system, which models the shutter directly, outperformed all other methods. It produced avatars that were significantly sharper and more accurate, preserving fine details like the taper of an arm and the gap between a hand and a hip that other methods had smoothed over or distorted. The researcher found that they did not need to make the time slices finer or more numerous to get better results; the existing motion model was already detailed enough to capture the movement within a single frame. The key was simply changing how the computer combined the different moments of the body's motion into the final image. By treating the camera's scan as a schedule of events rather than a single snapshot, the system could reconstruct a clean, undistorted 3D body that could be animated and viewed from any angle without the wobble or shear that plagued previous attempts.

This work demonstrates that the way a camera captures an image is not just a technical detail to be corrected later, but a fundamental part of the physics of seeing. By integrating the camera's behavior directly into the 3D reconstruction process, the researcher has shown that it is possible to recover a clear, accurate digital representation of a moving person from imperfect video. The findings suggest that the tools used to fix one type of image distortion, such as blur, cannot simply be swapped in to fix another, like rolling shutter, because the nature of the error is different. While the current system works best with multiple cameras and well-lit conditions, and while it relies on a mathematical model of the body's joints, it opens a new path for creating realistic digital humans from the everyday video footage that surrounds us. The researcher leaves open the question of whether this dense sampling of time could eventually be used to create video that runs faster than the camera captured it, but for now, the achievement is a clearer, truer picture of the moving human form.

Technical Summary: Scanline-Aware Animatable Gaussian Avatars from Rolling-Shutter Videos

1. Problem Statement

The reconstruction of animatable 3D human avatars from multi-view video typically relies on the assumption that every pixel in a frame captures the body at a single, instantaneous moment. However, the vast majority of CMOS cameras (from smartphones to capture studio rigs) utilize Rolling Shutter (RS) sensors, which read out image rows sequentially.

In an RS frame, the time difference between the first and last row can be tens of milliseconds. For a moving, articulated human body, this means the head and feet (or different limbs) are captured at significantly different poses within a single frame.

The Failure of Current Methods: Feeding RS video directly into state-of-the-art avatar reconstruction pipelines (e.g., Gaussian Splatting or NeRF) forces the optimizer to explain scanline-dependent geometry with a single pose per frame. This "bakes" the distortion into the canonical representation, resulting in shear and wobble artifacts under novel views and poses.
Limitations of Existing Solutions:
- Ignoring the shutter: Leads to corrupted canonical models and violates multi-view consistency because different cameras read out rows at different times.
- Two-stage correction: Applying 2D RS correction networks before avatar training fails because they operate per-view, cannot enforce 3D consistency, and hallucinate content in disoccluded regions.
- Existing RS-aware rendering: Current RS-aware neural rendering methods (e.g., RS-NeRF) assume a static scene with a moving camera. They cannot handle the inverse problem of a static camera rig observing a non-rigidly deforming human.
- Motion Blur models: While recent work reconstructs avatars from blurry video, the mathematical operator for blur (temporal averaging) is structurally different from RS (temporal selection). A blur model fails on RS data because it averages renderings that the sensor never actually averaged.

2. Methodology: RS-Avatar

The author proposes RS-Avatar, a framework that reconstructs sharp, undistorted, animatable 3D Gaussian avatars directly from RS video by integrating the shutter model into the image formation process.

Core Insight:
The formulation is minimal. A motion-aware avatar already renders the body at several sub-frame instants to handle motion blur. The only required change to adapt this for RS is the combination operator:

Blur Model: Averages $T$ sub-frame renderings uniformly ( $\frac{1}{T} \sum$ ).
RS Model: Composites the renderings scanline by scanline. Each row (or band of rows) is sampled from the specific sub-frame rendering corresponding to the time that row was read out.

Technical Components:

3D-Aware RS Image Formation:
- The latent sharp image is modeled as the rendering of the avatar under a specific body state at time $t$ .
- The observed RS frame is a composition of these latent images, where row $v$ samples the image at time $t_{c,n}(v) = n\Delta + \delta_c + \frac{v}{H}\rho_c$ .
- The model discretizes the frame into $T$ contiguous row bands. The synthesized frame $\hat{I}^{RS}$ is computed as a masked sum: $\sum_{i=1}^T M_i \odot R(W(G, S_{t_i}); \Pi_c)$ , where $M_i$ is the binary mask for band $i$ .
Sub-Frame Motion Model:
- To supply the $T$ body states required per frame, the method uses a B-spline interpolation of per-frame SMPL control points.
- A small network predicts a residual deformation on top of the interpolated pose to account for clothing and soft-tissue dynamics.
- Inter-frame Regularization: A geodesic penalty enforces continuity between the end of one frame's trajectory and the start of the next. Crucially, unlike in blur models, this prior does not need to disambiguate the direction of motion (as the scan order provides the "arrow of time"), reducing its burden to enforcing continuity alone.
Optimization:
- The system jointly optimizes canonical Gaussians, per-frame spline control points, deformation networks, shape, and skinning weights against raw RS observations.
- The temporal discretization ( $T$ ) does not need to be extremely fine (e.g., matching the number of rows) because the intra-frame trajectory is a low-order polynomial; coarse discretization already oversamples the motion curve.

3. Key Contributions

Problem Definition: The paper introduces the specific challenge of animatable avatar reconstruction from rolling-shutter video, identifying that standard "instantaneous" assumptions corrupt the canonical model.
Minimal Formulation: It presents a 3D-aware RS image formation model for Gaussian avatars that differs from the motion-blur model only in its combination operator (selection vs. averaging).
Theoretical Insight: The author demonstrates that corruption models are not interchangeable even when the underlying machinery (splines, renderers) is shared. They show that a motion-aware blur model, when applied to RS data, performs worse than a shutter-oblivious baseline because the averaging operator creates a mismatch with the selection-based ground truth.
Benchmark (RS-ZJU): They contribute a new benchmark built by scanline-compositing temporally interpolated ZJU-MoCap sequences, providing ground truth sharp frames for evaluation.

4. Results

Evaluated on the RS-ZJU benchmark (6 subjects, 18 held-out cameras):

Performance: RS-Avatar achieves a PSNR of 24.68, outperforming the shutter-oblivious baseline (GauHuman, 23.69) by 0.99 dB and the best two-stage correction method (CVR + GH, 23.99) by 0.69 dB.
Qualitative Improvements: The method successfully recovers sharp limb details (e.g., forearm taper, gaps between hand and hip) that are blurred or thickened by other methods.
Failure of Blur Transfer: The motion-aware blur model (MAD-Avatar) scored 22.99, falling 0.70 dB below the shutter-oblivious baseline. This confirms that the blur operator's averaging logic is fundamentally incompatible with RS data, despite using the same sub-frame rendering machinery.
Ablation: Removing the RS model (treating it as global shutter) drops performance by ~1.57 dB. Removing the intra-frame spline (using one pose per frame) causes a catastrophic drop of 3.25 dB, highlighting that the sub-frame motion model is essential even before the RS operator is applied.

5. Significance and Claims

The paper argues that the rolling shutter should be treated as an intrinsic part of the image formation model rather than a preprocessing artifact. By shifting the assumption from "a frame is a snapshot" to "a frame is a schedule," the method avoids the geometric inconsistencies that plague current avatar reconstruction.

The author emphasizes a broader lesson for the field: Corruption models are not interchangeable. While blur and rolling shutter both involve sub-frame motion, their operators (averaging vs. selection) have opposite ambiguities. A prior designed to resolve the directionality ambiguity in blur is unnecessary (and potentially harmful) for rolling shutter, where the scan order explicitly defines the time arrow.

The work is modest about its limitations, noting that it assumes sRGB well-exposed captures, relies on SMPL for sub-frame motion (struggling with loose garments or handheld objects), and currently operates on synthetic RS data (though the readout schedule is exact by construction). It leaves open the question of whether the dense temporal sampling of RS can be leveraged for temporal super-resolution.