Evaluating the Effect of Compression on Video Temporal Consistency Using Objective Quality Metrics

This paper systematically evaluates how video compression impacts temporal consistency across multiple codecs and content types, revealing that temporal degradation follows a non-linear pattern and is disproportionately severe in sequences with unpredictable dynamics, thereby challenging the assumption that motion volume alone dictates encoding difficulty.

The Gist

Imagine you are trying to send a flipbook animation to a friend over a slow internet connection. To make the file smaller, you have to "compress" it—basically, you tell the computer to be smart about what details to keep and what to throw away. Usually, the computer assumes that if an object moves, the next picture will look very similar to the last one, so it only sends the changes. This is how video compression works.

This paper is like a detective story investigating what happens when that "smart assumption" breaks down.

The Main Mystery: The "Predictability Trap"

The researchers tested four different video compression tools (think of them as different brands of video editors: H.264, HEVC, VP9, and AV1) on many different types of videos. They wanted to see how well these tools kept the video looking smooth and consistent from one frame to the next.

They discovered a strange phenomenon they call the "Predictability Anomaly."

Here is the analogy:

• Scenario A (The Train): Imagine a video of a train moving smoothly down a track. Even if the train is moving very fast, the computer can easily guess what the next frame will look like because the motion is predictable.
• Scenario B (The Crowd): Now imagine a video of a chaotic crowd or splashing water. The movement is wild and irregular. Even if the total amount of movement is less than the train, the computer cannot guess what happens next.

The Surprise: The researchers found that the computer handles the fast, predictable train (Scenario A) much better than the chaotic crowd (Scenario B). In fact, the chaotic crowd causes the video to glitch, flicker, and look unstable much faster than the fast train does.

The "VMAF Paradox": The Camera That Lies

The paper highlights a major problem with how we currently measure video quality. There is a popular tool called VMAF that acts like a judge, giving videos a score based on how sharp and clear they look.

The researchers found a "Paradox":
When the computer struggles with the chaotic crowd (Scenario B), it gives up on trying to predict the motion. Instead, it stops guessing and just takes a perfect, high-quality photo of every single moment (these are called "I-frames").

• The Result: Because every single frame is a sharp, perfect photo, the VMAF judge gives the video an unrealistically high score. It thinks the video looks great.
• The Reality: If you watch the video, it looks terrible. The images are sharp, but they "jump" or "flicker" because the connection between the frames is broken. It's like looking at a flipbook where every drawing is a masterpiece, but the animation is jerky and broken.

The paper calls this the "VMAF Paradox": The video looks perfect on paper (high score) but feels broken to the human eye (low stability).

The "Smoking Gun"

The researchers proved this by looking at how much the video improved when they gave the computer more data (higher bitrate).

• For the predictable train, doubling the data made the video much smoother and more stable.
• For the chaotic crowd, even giving the computer four times as much data didn't fix the flickering. The computer just kept taking perfect, isolated photos instead of learning how to connect them.

The Takeaway

The paper concludes that predictability matters more than speed.

• Old Assumption: "Fast motion is hard to compress."
• New Discovery: "Unpredictable, chaotic motion is the real nightmare for compression."

The current tools are "cheating" by focusing on making individual frames look sharp, which tricks our quality meters, but they are failing to keep the motion smooth. The paper suggests that future video technology needs to stop just looking at single frames and start paying attention to how the video flows from one moment to the next, especially for chaotic scenes like crowds or water.

Technical Summary

Technical Summary: Evaluating the Effect of Compression on Video Temporal Consistency Using Objective Quality Metrics

Problem Statement
While modern video compression algorithms effectively reduce bitrate by removing spatial and temporal redundancies, aggressive quantization often compromises temporal coherence. This manifests as artifacts such as flicker, motion inconsistency, and unstable textures ("floating" textures). Although spatial quality degradation is well-documented, the relationship between compression intensity and temporal stability remains insufficiently characterized. Traditional frame-by-frame metrics (e.g., PSNR, SSIM) fail to capture the dynamic lack of coherence that severely impacts human perceptual quality, particularly in motion-heavy content. This paper addresses the gap in understanding how varying compression levels influence temporal properties across different codec generations and content types.

Methodology
The study systematically analyzed the progression of frame-to-frame coherence errors using four video codecs representing different generations: H.264, HEVC, VP9, and AV1. VVC was excluded to ensure fair comparison regarding bitrate constraints and Rate-Distortion Optimization (RDO) behavior.

• Datasets: The evaluation utilized 44 aggregated sequences from the Ultra Video Group (UVG), HEVC Class B, and BVI-HD datasets, covering a wide variety of textures and motion patterns (1080p, 24–120fps).
• Complexity Grouping: Sequences were categorized into four quartiles based on Temporal Information (TI), defined as the maximum standard deviation of space-time differences between consecutive frames:
  • TI Group 1 (Static): Stationary backgrounds.
  • TI Group 2 (Predictable Motion): Low-to-medium motion with high temporal structure.
  • TI Group 3 (Unpredictable Dynamics): Medium-to-high irregular motion (e.g., water, crowds) lacking clear motion vector structure.
  • TI Group 4 (Global Motion): High-magnitude but structured motion (e.g., consistent camera pans).
• Compression Settings: Videos were encoded at seven bitrate levels (scaling factors 1 through 8) using FFmpeg libraries with parameters optimized for perceptual quality.
• Evaluation Metrics: The study employed a comprehensive suite of metrics:
  • Spatial: PSNR, SSIM, VMAF.
  • Temporal: ST-RRED (fluctuation of information content), MOVIE Index (distortions along motion trajectories), T-SSIM, and T-PSNR.
  • Generative: Fréchet Video Distance (FVD).
  • Efficiency: Bjøntegaard Delta (BD-Rate) curves.

Key Results and Findings
The analysis confirmed a clear generational performance hierarchy, with AV1 consistently yielding the highest Rate-Distortion (R-D) performance, followed closely by HEVC. However, the study identified a critical non-linear relationship between content complexity and stability, termed the "Predictability Anomaly."

1. The Predictability Anomaly: Sequences with unpredictable or irregular dynamics (TI Group 3) experienced disproportionately higher instability than sequences with high-magnitude but predictable motion (TI Group 4).
2. Fallback to Intra-Coding: Encoders struggled to establish reliable motion vectors for TI Group 3, forcing a "reset" of the prediction loop. This resulted in higher I-frame frequencies (0.61% for Group 2 and 0.57% for Group 3) compared to Group 4 (0.50%), despite Group 4 having higher raw motion magnitude.
3. The VMAF Paradox: At higher bitrates (≥4000 kbps), VMAF scores for TI Group 3 surpassed all other groups. This occurred because the encoder's reliance on frequent I-frames produced a sequence of high-fidelity spatial snapshots. Since VMAF is heavily weighted toward spatial features (Visual Information Fidelity and Detail Loss Measure), it rewarded this spatial sharpness despite the temporal inefficiency.
4. Metric Decoupling: Temporal-aware metrics (ST-RRED, MOVIE Index) revealed that while the "I-frame heavy" strategy improved spatial quality, it failed to resolve—and potentially exacerbated—temporal jitter and "breathing" artifacts. TI Group 3 maintained high error variance and low relative stability recovery even with significant bitrate increases, whereas TI Group 4 showed significant stability improvements.

Significance and Contributions
The paper's primary contribution is the identification of the "Predictability Anomaly," which challenges the conventional assumption that motion volume alone dictates encoding difficulty. The findings demonstrate that motion regularity is a more significant determinant of temporal stability than raw motion magnitude.

The study highlights a critical limitation in current evaluation pipelines: encoders can effectively "cheat" spatial-heavy metrics (like VMAF) by sacrificing temporal efficiency to achieve frame-by-frame sharpness through I-frame resets. This results in artificially high objective scores that do not reflect the underlying lack of motion-trajectory integrity. Consequently, the paper argues for the necessity of temporal-aware metrics in compression pipelines to accurately assess quality, particularly for content with unpredictable dynamics. Future work proposed by the authors includes analyzing specific spatial features disrupting motion estimation, examining internal codec statistics, and conducting subjective validation to measure the impact on Quality of Experience.