RobustSpring: Benchmarking Robustness to Image Corruptions for Optical Flow, Scene Flow and Stereo

The paper introduces RobustSpring, a comprehensive benchmark and dataset that evaluates the robustness of optical flow, scene flow, and stereo vision models against 20 types of image corruptions, addressing the gap in existing benchmarks that primarily focus on accuracy rather than resilience to real-world perturbations.

The Gist

Imagine you have built a very smart robot driver. You've trained it on perfect, sunny days with crystal-clear roads. It's a champion at navigating those conditions. But what happens when it starts raining? Or when the camera lens gets foggy? Or when a bird flies in front of the lens and leaves a smudge?

Most current tests for these robots only check how well they drive on perfect days. They don't ask, "Can you still drive safely when the world gets messy?"

This paper introduces RobustSpring, a new "stress test" for the eyes of robots (specifically for tasks like optical flow, scene flow, and stereo vision). Here is the breakdown in simple terms:

1. The Problem: The "Glass House" Effect

Think of current AI models like a glass house. They look beautiful and strong when the sun is shining (high accuracy on clean data). But as soon as a storm hits (real-world noise, rain, blur), the glass might shatter.

For years, researchers have been obsessed with making the glass house bigger and shinier (more accurate). But they haven't tested if it can survive a hurricane. The authors realized that a model can be incredibly accurate on clean images but completely fail when the image is slightly corrupted by rain or snow.

2. The Solution: The "Weather Simulator"

The authors took a high-quality video dataset called Spring (which is like a pristine, computer-generated movie of a city) and decided to ruin it on purpose.

They created 20 different types of "ruining" effects, including:

• Blurry Vision: Like a camera lens that is out of focus or moving too fast.
• Bad Weather: Rain, snow, fog, and frost.
• Digital Glitches: Pixelation (like a low-res video), JPEG compression artifacts, and random static noise.
• Color Shifts: Making the image too bright, too dark, or too colorful.

The Magic Trick: The authors didn't just slap a filter on a single photo. They made sure the "ruin" made sense in 3D space and over time.

• Example: If it's raining, the raindrops fall consistently in the video. If you look at the scene with two eyes (stereo vision), the rain looks the same to both eyes. If you move the camera, the rain moves with the perspective. This makes the test incredibly realistic.

3. The New Scorecard: "Stability" vs. "Accuracy"

Usually, we grade a robot driver by how close its path is to the perfect line (Accuracy).
RobustSpring introduces a new grade: Stability.

• The Old Way: "How close was your guess to the truth?"
• The RobustSpring Way: "If I shake the camera or pour water on the lens, did your guess change wildly, or did you stay calm?"

They use a metric based on Lipschitz continuity (a fancy math term that basically means "if the input changes a little, the output shouldn't change a lot").

• High Stability: The robot sees a rainy road and still knows where the lane is.
• Low Stability: The robot sees a rainy road and thinks the lane is moving sideways or disappears entirely.

4. The Results: The "Glass House" Cracks

The authors tested 17 of the smartest, most popular AI models on this new stress test. The results were surprising:

• No One is Perfect: Even the best models struggled significantly with certain types of "ruin," especially rain, snow, and noise.
• Accuracy $\neq$ Robustness: Being the "smartest" model on a clean day didn't guarantee it would be the "safest" model on a rainy day. Some models that were very accurate on clean data were actually less stable when things got messy.
• The "Noise" Problem: Many models got confused by random static noise, acting like they were hallucinating.

5. Why This Matters

Imagine you are buying a self-driving car. You wouldn't just want to know how fast it drives on a test track; you'd want to know if it can handle a blizzard.

RobustSpring is the first standardized "blizzard test" for computer vision. It forces researchers to stop just chasing higher scores on clean data and start building models that are resilient. It's about moving from "smart in a lab" to "safe in the real world."

Summary Analogy

• Old Benchmarks: Testing a swimmer in a calm, indoor pool.
• RobustSpring: Testing that same swimmer in the ocean during a storm with waves, jellyfish, and cold water.
• The Goal: We don't just want swimmers who are fast in a pool; we want swimmers who won't panic when the ocean gets rough.

Technical Summary

1. Problem Statement

Standard benchmarks for optical flow, scene flow, and stereo vision (e.g., Spring, KITTI, Sintel) primarily focus on accuracy (how well a model predicts ground truth under ideal or specific conditions). However, they largely neglect robustness to real-world image corruptions such as noise, blur, weather, and compression artifacts.

• The Gap: High accuracy does not guarantee robustness; in fact, models optimized for accuracy can be brittle to perturbations.
• The Limitation of Existing Work: While robustness studies exist for image classification, 3D object detection, and monocular depth, there is no systematic benchmark for dense correspondence tasks (optical flow, scene flow, stereo) that accounts for the unique requirements of these tasks: temporal consistency (video), stereo consistency (left/right views), and depth consistency (3D geometry).
• Consequence: Without systematic evaluation, the reliability of these algorithms in real-world scenarios (e.g., autonomous driving in rain, surgical assistance with glare) remains unquantified.

2. Methodology: The RobustSpring Framework

The authors propose RobustSpring, a comprehensive dataset and benchmark built upon the high-resolution Spring dataset.

A. Dataset Creation

• Base Data: Uses 2,000 test frames from the Spring dataset (stereo video with dense ground truth).
• Corruptions: Applies 20 distinct image corruptions categorized into five groups:
  1. Color: Brightness, Contrast, Saturation.
  2. Blur: Defocus, Gaussian, Glass, Motion, Zoom.
  3. Noise: Gaussian, Impulse, Speckle, Shot.
  4. Quality: Pixelation, JPEG compression, Elastic transform.
  5. Weather: Fog, Spatter, Frost, Snow, Rain.
• Novel Consistency Integration: Unlike prior 2D/3D corruption benchmarks, RobustSpring ensures corruptions are consistent across three dimensions where applicable:
  • Time-Consistent: Corruptions evolve smoothly over frames (e.g., frost patterns persisting over time).
  • Stereo-Consistent: Both left and right views undergo the same transformation strength (e.g., shared brightness adjustment), though noise realizations may differ.
  • Depth-Consistent: Weather effects (rain, snow, fog) are rendered in 3D space, respecting scene geometry and generating view-dependent projections.
• Implementation Details:
  • Depth and extrinsics are estimated (using MS-RAFT+ and COLMAP) to enable 3D rendering without leaking ground truth.
  • Severity: A single severity level per corruption is chosen to balance evaluation resources. Severity is tuned so that the Structural Similarity Index (SSIM) between clean and corrupted images reaches a target threshold (SSIM $\ge$ 0.7 for most, $\ge$ 0.2 for noise).

B. Robustness Metric

The paper introduces a ground-truth-free robustness metric based on Lipschitz continuity.

• Concept: Instead of measuring the error against ground truth (which mixes accuracy and robustness), it measures the stability of the model's prediction.
• Definition: The metric quantifies the distance between the prediction on a clean image $f(I)$ and the prediction on a corrupted image $f(I_c)$.
  $$R_c = M[f(I), f(I_c)]$$
  Where $M$ is a distance metric (e.g., End-Point Error for flow, Disparity error for stereo).
• Rationale: A robust model should produce consistent predictions even when the input is perturbed. Lower $R_c$ values indicate higher stability (robustness). This approach disentangles robustness from accuracy.

C. Benchmark Functionality

• Subsampling: To manage data volume (20,000 corrupted frames), a refined subsampling strategy retains only 0.05% of the data (excluding "Hero" frames) while maintaining statistical equivalence to full-data evaluation.
• Ranking: Models are ranked using three strategies: Average, Median, and the Schulze voting method (pairwise comparison) to handle the 20 different corruption scenarios holistically.
• Integration: RobustSpring is coupled with the existing Spring accuracy benchmark, allowing researchers to view accuracy and robustness on a two-axis plot.

3. Key Contributions

1. Tailored Image Corruptions: The first dataset to apply 20 diverse corruptions to optical flow, scene flow, and stereo tasks with integrated time, stereo, and depth consistency.
2. Corruption Robustness Metric: A novel, ground-truth-free metric based on prediction stability (Lipschitz continuity) that separates robustness from accuracy.
3. Standardized Benchmark: A public framework enabling community-driven comparisons, integrated with the Spring benchmark website.
4. Initial Evaluation: A comprehensive benchmark of 17 state-of-the-art models (9 optical flow, 2 scene flow, 6 stereo) revealing significant sensitivity to corruptions.

4. Experimental Results

The authors evaluated 17 models (including RAFT, GMFlow, MS-RAFT+, FlowFormer, LEAStereo, etc.) without fine-tuning them on RobustSpring data.

• General Sensitivity: All models showed significant performance degradation under corruptions. Weather (Rain, Snow) and Noise (Impulse, Gaussian) caused the largest errors.
• Corruption Specifics:
  • Color changes had the least impact.
  • Noise was particularly damaging to Transformer-based models (e.g., GMFlow, FlowFormer), likely due to their global matching mechanisms.
  • Weather (Rain/Snow) caused massive errors, especially for stereo disparity models.
• Accuracy vs. Robustness:
  • There is a weak, non-linear correlation between accuracy and robustness.
  • Contrary to adversarial robustness (where accuracy and robustness often trade off), here, more accurate models sometimes showed better robustness to weather (by confining errors to particles rather than spreading them to the background).
  • However, no single model excelled in both dimensions across all corruptions.
• Model Performance:
  • SEA-RAFT and GMFlow achieved the best average robustness scores.
  • FlowNet2 was surprisingly robust to noise despite lower overall accuracy.
  • Stereo models (e.g., GANet, LEAStereo) were highly sensitive to pixel-level noise.
• Real-World Transfer: The robustness rankings on RobustSpring correlated well with performance on noisy real-world KITTI data, validating the benchmark's utility for predicting real-world behavior.

5. Significance and Impact

• First-Class Citizen for Robustness: RobustSpring treats robustness as a primary evaluation axis alongside accuracy, addressing a critical gap in dense matching research.
• Real-World Applicability: By simulating realistic, consistent corruptions (e.g., rain in 3D), the benchmark provides a more reliable proxy for how models will perform in autonomous driving, robotics, and medical imaging.
• Methodological Insight: The results highlight that architectural choices (e.g., Transformers vs. Hierarchical CNNs) have distinct trade-offs regarding specific corruption types, guiding future model design.
• Community Resource: By providing a standardized metric and public leaderboard, it fosters the development of models that are not just accurate, but resilient to the unpredictable nature of the real world.

In conclusion, RobustSpring establishes a new standard for evaluating dense correspondence models, proving that current state-of-the-art models are fragile to common image corruptions and providing the tools necessary to build more reliable computer vision systems.