💻 computer science

Failure or Drift? Evaluating Monocular SLAM under Synthetic and Real-World Corruptions

This paper evaluates monocular SLAM systems under synthetic and real-world corruptions, revealing that learned trackers often trade catastrophic failure for severe drift and that the validity of synthetic stress tests depends on their physical fidelity to real-world conditions.

Original authors: Abhay Skaria Thomas, Shashank Agnihotri, Margret Keuper

Published 2026-09-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abhay Skaria Thomas, Shashank Agnihotri, Margret Keuper

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a car driving itself through a city, or a robot navigating a warehouse, relying entirely on a single camera to understand where it is. This technology, known as visual SLAM, is the invisible compass that allows machines to build a map of their surroundings and track their movement in real time. For these systems to work safely, they must function not just in perfect, sunlit conditions, but also when the world turns against them: when rain blurs the lens, fog obscures the road, or the sun dips low and casts long, confusing shadows. Engineers have long tried to test these systems by artificially damaging the video feed with computer-generated noise, blur, or darkness, hoping to predict how the machine would behave in a storm. The core question, however, has remained unanswered: does a test that looks like a storm actually behave like one? If a computer simulation says a robot is safe, is it truly safe, or is it just fooling the test?

A team of researchers at the University of Mannheim and the Max Planck Institute for Informatics set out to answer this by putting three different types of camera-tracking systems through a rigorous gauntlet. They took a standard driving route recorded in Germany and subjected it to two very different kinds of stress. First, they applied simple, flat distortions to the images, such as changing the brightness, adding grainy noise, or blurring the picture, much like applying a filter to a smartphone photo. Second, they applied more complex, physics-based distortions that respected the depth of the scene, such as simulating rain that falls in front of distant objects but not close ones, or fog that thickens in the distance. They then compared how these systems performed against real-world footage taken in actual rain, winter, and evening conditions. The goal was not just to see which system failed, but to understand how it failed.

The results revealed a fundamental split in how these machines break. One of the systems, which relies on finding distinct, sharp corners and edges in the image, behaves like a human who loses their way in thick fog: it simply stops. When the visual features become too hard to see, this system admits defeat and returns no path at all. This is a clear, explicit failure that is easy to detect. The other two systems, which use advanced learning to guess the movement of the scene, behave differently. They do not stop. Even when the image is heavily corrupted, they continue to output a path, but that path slowly drifts away from reality. They might tell the robot it is moving in a straight line when it is actually turning, or that it has traveled a hundred meters when it has only moved a few. This silent drift is far more dangerous because the system appears to be working perfectly while quietly leading the machine off a cliff.

Perhaps the most surprising discovery was that the type of test used completely changed which system appeared to be the best. When the researchers used the simple, flat image distortions, one of the learning-based systems seemed superior, handling the noise better than its competitor. However, when they switched to the more realistic, physics-based distortions that accounted for depth and scene structure, the ranking flipped entirely. The system that looked weaker in the simple tests became the clear winner in the complex ones. This suggests that using simple image filters to test robustness can be misleading; a test that looks like a storm might actually be testing for a completely different problem than a real storm.

The researchers then checked these findings against real-world data from actual rainy days, foggy mornings, and cloudy evenings. They found that the complex, physics-based tests correctly predicted which system would perform better in the rain and winter conditions. The simple tests, however, failed to predict the outcome for evening lighting, incorrectly favoring the system that actually performed worse in the real world. This proves that while synthetic tests are useful tools for diagnosis, they are not perfect crystal balls. A test that looks convincing on a screen does not always translate to the messy reality of the physical world.

The study concludes that we must be careful about how we judge these technologies. A system that never stops producing a path is not necessarily a reliable one; it might just be confidently wrong. True robustness requires distinguishing between a system that admits it cannot see and one that keeps guessing while drifting off course. Furthermore, the choice of test matters immensely. To truly understand if a robot can handle a storm, we must test it with simulations that respect the physics of the storm, not just the look of it. Only by matching the complexity of the test to the complexity of the real world can we trust that the machines we send out into the rain will find their way home.

Technical Summary: Failure or Drift? Evaluating Monocular SLAM under Synthetic and Real-World Corruptions

Problem Statement

Visual Simultaneous Localization and Mapping (SLAM) is a prerequisite for autonomous driving, mobile robotics, and augmented reality. However, standard evaluations predominantly rely on clean benchmark trajectories, failing to capture the adverse conditions (e.g., rain, fog, low illumination, motion blur, sensor noise) encountered during deployment. While synthetic corruptions offer a controlled way to isolate specific degradation factors, their validity as proxies for real-world conditions remains unverified for stateful systems like SLAM. In such systems, errors propagate through time; a corruption that alters visual correspondences may favor different trackers or lead to different failure modes (explicit loss of tracking vs. silent accumulation of drift) compared to natural adverse conditions. This work investigates the external validity of synthetic stress tests for monocular SLAM, asking whether conclusions drawn from controlled image-space or geometry-aware corruptions hold when applied to real-world adverse data.

Methodology

The study employs a controlled evaluation protocol comparing three distinct monocular SLAM paradigms without corruption-specific adaptation:

ORB-SLAM2: A classical, sparse feature-based system with explicit loop closure and relocalization.
DPVO: A learned, patch-based visual odometry system using recurrent updates and differentiable bundle adjustment, lacking classical loop closure.
DROID-SLAM: A dense, learned correspondence system utilizing global multi-frame optimization and dense bundle adjustment.

Datasets and Corruptions:

Baseline: Clean KITTI odometry sequences (00–10).
Synthetic Stress Test: Sequence 00 of KITTI is subjected to 115 corrupted variants across three categories at five severity levels:
- Image-space: 13 transformations (e.g., brightness, contrast, noise, blur, JPEG compression) from the imagecorruptions package.
- Geometry-aware: 7 effects (e.g., fog, rain, low light, focus changes) from 3D Common Corruptions, utilizing depth and scene structure.
- Compound: Combinations of the above (e.g., rain + contrast).
Real-World Reference: Four sequences from the 4Seasons dataset representing cloudy afternoon, rainy afternoon, winter cloudy morning, and cloudy evening conditions. These serve as condition-level matches rather than paired recordings.

Evaluation Protocol:
The authors introduce a failure-aware protocol to distinguish between explicit tracking failure and accumulated drift:

Validity ( $V_r$ ): A binary metric indicating if a non-empty, evaluable trajectory is returned.
Conditional Error: Absolute Pose Error (APE) and Relative Pose Error (RPE) are calculated only for valid trajectories. This prevents methods that return no output from artificially lowering error metrics by avoiding calculation.
Comparison: The study analyzes the "valid-run rate" and the relative ordering of learned trackers (DPVO vs. DROID-SLAM) across different corruption types and severities.

Key Contributions

Controlled Evaluation: A systematic comparison of sparse feature-based, learned patch-based, and dense learned SLAM paradigms across image-space, geometry-aware, compound, and natural adverse conditions.
Failure-Aware Protocol: A methodology that reports trajectory validity separately from pose accuracy, avoiding the conflation of tracking loss with low error metrics.
Fidelity Analysis: Evidence demonstrating that the "fidelity" of the corruption model (image-space vs. geometry-aware vs. compound) fundamentally alters the ranking of learned trackers, and that synthetic conclusions do not always transfer to real-world data.

Results

1. Failure Modes: Explicit Failure vs. Silent Drift

ORB-SLAM2: Frequently exhibits explicit failure, returning no evaluable trajectory under noise, snow, or severe blur. When valid, it often achieves the lowest median APE on clean data but suffers from route-dependent instability.
Learned Trackers (DPVO & DROID-SLAM): Remain active in 100% of synthetic attempts. Instead of failing explicitly, they accumulate sustained and sometimes severe drift. Under geometry-aware and compound corruptions, their aligned APE frequently exceeds 100 meters. This "silent degradation" is operationally dangerous as it may pass through autonomy stacks without triggering relocalization or fallback mechanisms.

2. Corruption Fidelity Changes Tracker Selection

The apparent robustness of a system depends heavily on the type of corruption applied:

Image-Space Corruptions: Mostly favor DPVO. In 30 out of 43 paired settings, DPVO achieves lower APE. Benchmarks restricted to simple global transformations (brightness, contrast) would incorrectly select DPVO as the superior system.
Geometry-Aware & Compound Corruptions: Strongly favor DROID-SLAM. It achieves lower APE in all 33 geometry-aware settings and 14 of 15 compound settings. The dense, global optimization of DROID-SLAM appears more robust to structured degradations like rain and fog than the local window optimization of DPVO.

3. Synthetic-to-Real Transferability

The study finds that synthetic conclusions do not uniformly transfer to real-world conditions:

Rain and Winter (Fog): The ordering observed in compound synthetic proxies (DROID-SLAM > DPVO) is preserved in real rainy and winter sequences.
Evening Illumination: The synthetic proxy (brightness + contrast) favors DROID-SLAM, but the real-world cloudy evening sequence favors DPVO.
Conclusion: Simple image-space proxies (like global brightness) fail to capture complex real-world factors such as automatic exposure, spatially varying illumination, and camera-specific noise, leading to incorrect architectural recommendations.

Significance and Claims

The paper argues that robustness evaluation for SLAM must move beyond single-trajectory error metrics on clean data. The primary significance lies in exposing the dichotomy between availability and trustworthiness: a system may remain active (available) while its trajectory becomes globally incorrect (untrustworthy).

The authors claim that:

Synthetic corruptions are valuable diagnostic tools but not calibrated forecasts of deployment error.
The choice of corruption model dictates the conclusion: image-space tests favor local, patch-based methods, while geometry-aware tests favor dense, globally optimized methods.
A "proxy ladder" is necessary for evaluation: starting with image-space for breadth, moving to geometry-aware/compound for mechanism fidelity, and validating with natural adverse conditions.
Future robustness metrics must distinguish between trajectory validity, map validity, and system validity, particularly as dense reconstruction systems (e.g., Gaussian Splatting SLAM) begin to couple tracking quality directly with map state.

The work concludes that while synthetic stress tests can reveal architectural weaknesses that persist in natural data (as seen with rain and fog), they can also mislead (as seen with illumination), necessitating a more nuanced evaluation framework that accounts for the specific failure mechanisms of different SLAM paradigms.