🤖 machine learning

Unifying Runtime Monitoring Approaches for Safety-Critical Machine Learning: Application to Vision-Based Landing

This paper proposes a unified framework that categorizes runtime monitoring approaches for safety-critical machine learning into Operational Design Domain, Out-of-Distribution, and Out-of-Model-Scope types, demonstrating their complementary benefits through a vision-based aircraft landing experiment.

Original authors: Mathieu Dario, Florent Chenevier, Kévin Delmas, Joris Guerin, Jérémie Guiochet

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mathieu Dario, Florent Chenevier, Kévin Delmas, Joris Guerin, Jérémie Guiochet

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have hired a very talented, but slightly nervous, robot pilot to land an airplane. This robot is an expert at recognizing runways in pictures, but it has a few quirks: it gets confused if the weather is weird, it sometimes hallucinates runways where there are none, and it can be tricked by tricky lighting.

The paper you're asking about is like a safety manual for hiring a team of security guards to watch over this robot pilot. The authors noticed that different groups of experts (some who study computer code, others who study airplane safety) were inventing their own separate security guards, often talking past each other. They decided to build a unified "Security Team" framework to organize these guards into three distinct roles, ensuring no safety gaps are left open.

Here is how their "Security Team" works, using simple analogies:

1. The Gatekeeper (ODD Monitor)

The Concept: Operational Design Domain (ODD)
The Analogy: Imagine the robot pilot is only trained to land on runways during a sunny day. If a storm rolls in, or if the plane is flying upside down, the robot shouldn't even try to land.
What this guard does: This guard stands at the front door. Before the robot even looks at the picture, this guard checks the "ID card" of the situation.

"Is it sunny?"
"Is the plane at the right altitude?"
"Is there only one runway visible?"
If the answer is "No" (e.g., it's foggy, or the plane is too high), the Gatekeeper says, "Stop! This isn't a job for our robot." They reject the input immediately. They don't care what the robot thinks; they just know the rules of the game.

2. The Pattern Detective (OOD Monitor)

The Concept: Out-of-Distribution (OOD)
The Analogy: The robot was trained on millions of photos of runways. It knows what a "normal" runway photo looks like. But what if someone hands it a photo of a runway covered in strange, glowing neon paint, or a photo that is super blurry because of a broken camera lens? The robot has never seen this before.
What this guard does: This guard looks at the photo itself, not the robot's answer. They are like a detective checking for "weirdness."

"Does this photo look like the millions of photos we trained on?"
"Is the brightness weird? Is the texture strange?"
If the photo is too different from what the robot learned (even if the weather is technically "okay"), the Detective says, "I don't recognize this pattern. It's too risky. Let's not let the robot guess." They reject the input because the data itself is suspicious.

3. The Performance Coach (OMS Monitor)

The Concept: Out-of-Model-Scope (OMS)
The Analogy: Sometimes, the photo looks normal, and the weather is perfect, but the robot still makes a silly mistake. Maybe it gets confused by a weird shadow, or it gets tricked by a cleverly placed sticker on the runway (an "adversarial attack").
What this guard does: This guard stands behind the robot. They watch the robot's brain (its internal thoughts) and its final answer.

"The robot says it sees a runway, but its internal confidence is shaky."
"The robot is guessing wildly."
"The robot's internal logic is acting strange."
If the robot is struggling or acting weird, the Coach says, "I don't trust this specific answer. Even though the photo looked fine, the robot is failing right now." They catch errors that the first two guards missed.

The Experiment: Putting the Team to Work

The authors tested this "Security Team" on a simulated airplane landing task.

The Result: When they used just one guard, they missed some dangers. But when they used all three guards working together in a line (Gatekeeper $\rightarrow$ Detective $\rightarrow$ Coach), they caught almost all the mistakes.
The Catch: Being super safe has a cost. Because the guards are so careful, they sometimes say "No" to perfectly good flights just to be safe. This is called the "Availability Cost." The paper shows that while you get much safer, you might have to cancel more flights than you'd like.

The Big Takeaway

The paper argues that we shouldn't just throw random safety tools at AI problems. Instead, we should clearly define who does what:

Gatekeepers check the rules of the world.
Detectives check if the data looks normal.
Coaches check if the AI is thinking correctly.

By separating these roles, engineers can build better, safer AI systems for things like landing planes, knowing exactly which type of danger each guard is designed to stop. The paper proves that these three approaches are complementary—they fill each other's blind spots, making the whole system much more reliable than any single guard could be alone.

1. Problem Statement

The integration of Machine Learning (ML), particularly Deep Neural Networks (DNNs), into safety-critical systems (e.g., aviation, autonomous driving) introduces significant reliability challenges. DNNs lack robustness, often fail under unforeseen inputs, and exhibit excessive confidence in incorrect predictions. While runtime safety monitors exist to mitigate these risks, current research is fragmented across different communities (dependability vs. machine learning).

The Gap: Existing monitoring methods operate in silos with different terminologies and scopes. Some focus on system-level operational constraints, others on statistical data distribution shifts, and others on model internal states.
The Consequence: This lack of unification leads to partial safety coverage, overlapping methodologies, and difficulty in comparing or combining different monitors using common metrics. There is no unified framework to systematically categorize threats and select appropriate monitoring strategies.

2. Methodology: The SwMF Framework

The authors propose a unified framework called SwMF (Safety-critical ML Monitoring Framework). This framework redefines the ML input space and categorizes monitoring approaches based on the specific "data threats" they address.

A. Decomposition of the Input Space

The authors project three key concepts onto the ML input space (e.g., images) to define distinct domains and threats:

Operational Design Domain (ODD): The set of conditions under which the system is designed to operate (e.g., weather, altitude, runway visibility).
- Threat: ODD Violations. Inputs outside the intended operating conditions.
In-Distribution Domain ( $D_{ID}$ ): Data drawn from the same distribution as the training data.
- Threat: Out-of-Distribution (OOD) Threats. Inputs that are within the ODD but statistically different from training data (e.g., covariate shifts like lighting changes, or semantic shifts like new object classes).
Model Scope ( $D_f$ ): The subset of inputs where the specific ML model $f$ $f$ produces correct predictions.
- Threat: In-Distribution (ID) Errors. Inputs that are within $D_{ID}$ but still cause the model to fail (e.g., adversarial attacks, subtle physical faults, or model generalization limits).

B. Categorization of Monitors

Based on these threats, the framework defines three complementary monitor categories:

ODD Monitors: Rule-based monitors that verify system variables (e.g., GPS, IMU, weather sensors) against predefined constraints. They reject inputs before they reach the ML model if operating conditions are invalid.
OOD Monitors: Data-driven monitors that detect statistical anomalies in input data (e.g., brightness, entropy, edge density) compared to the training distribution. They reject inputs that deviate from the training distribution but are still within the ODD.
OMS (Out-of-Model-Scope) Monitors: Post-hoc monitors that analyze the ML model's internal features (e.g., logits, neuron activations) or outputs to detect anomalous behavior. They specifically target inputs where the model is likely to fail, even if the input looks "normal."

C. Experimental Setup

Application: Vision-Based Landing (VBL), specifically the sub-task of Runway Detection (VBRD).
Dataset: The LARD dataset (synthetic images generated via Google Earth Studio).
Model: YOLOv5-n (a single-stage object detector).
Architecture: A serial monitoring pipeline where ODD monitors run first, followed by OOD monitors, and finally OMS monitors after the model prediction.
Evaluation Metrics: Safety-oriented metrics proposed by Guerin et al.:
- Safety Gain (SG): Proportion of dangerous situations correctly filtered.
- Residual Hazard (RH): Proportion of dangerous situations missed.
- Availability Cost (AC): Proportion of safe situations wrongly rejected (false alarms).

3. Key Contributions

Unified Terminology: The paper clarifies the definitions of ODD, OOD, and OMS, resolving the ambiguity where these terms are often used interchangeably in literature. It explicitly distinguishes between data distribution shifts (OOD) and model failure regions (OMS).
Threat-Based Classification: It introduces a formal mapping between specific data threats (ODD violations, OOD shifts, ID errors) and the appropriate monitoring category, providing a systematic design guide for safety engineers.
Complementarity Demonstration: Through experiments, the authors prove that these three monitor types address different failure modes and are inherently complementary, rather than redundant.
Unified Evaluation: The framework enables the comparison of diverse monitors using a single set of safety metrics (SG, RH, AC).

4. Experimental Results

The study was conducted in two experiments:

Experiment 1: Nominal Test Set (Synthetic Data)

Baseline: The YOLOv5-n model had a Residual Hazard (RH) of ~20% (meaning 20% of images caused errors).
Individual Performance:
- ODD monitors detected ~4.6% of errors (SG).
- OOD monitors detected ~4.1% of errors.
- OMS monitors detected ~4.0% of errors.
Combined Performance: When all three monitors were applied sequentially, the Safety Gain increased to 9.7%, and the Residual Hazard dropped to 10.5%.
Trade-off: The combined approach increased the Availability Cost (AC) to ~31%, meaning roughly 30% of safe images were rejected. This highlights the "safety vs. availability" trade-off.

Experiment 2: Corrupted Data (Robustness Test)

Setup: Images were corrupted with brightness changes, blur, fog, and noise.
OOD Monitor: Highly effective at rejecting corrupted images (high SG), acting as a "precautionary" filter. However, this resulted in high Availability Costs (AC) because it rejected data even when the model might have handled it correctly.
OMS Monitor: Less effective on corrupted data compared to OOD. It performed best on "clean" but difficult inputs (ID errors).
Conclusion: The results validate the serial architecture. OOD monitors should filter out obvious distribution shifts first, allowing OMS monitors to focus on subtle model failures within the remaining "safe" distribution.

5. Significance and Conclusion

Systematic Safety Design: The SwMF framework provides a structured approach for designing runtime monitoring systems, ensuring that all major threat vectors (operational, statistical, and model-specific) are covered.
Bridging Communities: It successfully bridges the gap between the dependability community (system-level safety) and the ML community (data distribution and model behavior).
Practical Implications: The study confirms that while combining monitors significantly improves safety (reducing residual hazard), it inevitably increases the rate of false alarms (availability cost). Future work must focus on optimizing this trade-off and improving the data selection for training these monitors to better represent operational threats.

In summary, the paper argues that a unified, multi-layered monitoring strategy is essential for the safe deployment of ML in critical aviation systems, moving beyond isolated solutions to a holistic safety architecture.