Defining Operational Conditions for Safety-Critical AI-Based Systems from Data

This paper proposes a novel, automated Safety-by-Design method that uses a multi-dimensional kernel-based representation to derive the Operational Design Domain (ODD) from collected data, thereby addressing certification challenges for safety-critical AI systems as validated by Monte Carlo simulations and a real-world aviation collision-avoidance use case.

The Gist

Imagine you are teaching a robot to fly a plane. You want the robot to be safe, so you need to tell it exactly where and when it is allowed to fly. In the world of AI safety, this "allowed zone" is called the Operational Design Domain (ODD).

Traditionally, experts would sit down with a whiteboard and try to draw this zone by hand, writing down rules like "don't fly in rain" or "don't fly above 30,000 feet." But the real world is messy. Weather, traffic, and wind interact in complex ways that are impossible to list perfectly on a whiteboard. This often leads to safety gaps where the robot thinks it's safe, but it's actually in a dangerous situation it wasn't told about.

This paper proposes a new way to draw that safety zone: let the data draw it for you.

Here is a simple breakdown of how they did it, using everyday analogies:

1. The Problem: The "Blank Map"

Imagine you have a map of a city, but the streets are hidden in fog. You know the city exists, but you don't know exactly where the safe roads are and where the cliffs are.

• Old Way: Experts guess where the roads are based on their experience. They might miss a hidden cliff.
• New Way: You drop thousands of glowing marbles (data points) onto the map. Where the marbles land, you know it's safe. Where they don't land, you assume it might be dangerous.

2. The Solution: The "Glowing Net"

The authors created a method to turn those scattered data points into a smooth, continuous safety map. They call this a Kernel-Based Representation.

Think of each data point (a safe flight condition) as a campfire.

• The Fire: Right at the campfire, it's very warm (very safe).
• The Heat: As you walk away from the fire, the heat fades. It doesn't just stop abruptly; it gets cooler and cooler until it's barely noticeable.
• The Net: The AI system creates a giant, invisible "heat map" by combining the warmth of all these campfires.
  • If you are standing where the heat is strong, you are inside the safety zone.
  • If you are in a cold spot between fires, you are outside the safety zone.

This is better than drawing a hard box around the campfires because it accounts for the "gray areas" in between.

3. The "Safety Net" for Mistakes

What if you accidentally drop a marble in a place that is actually dangerous (like a cliff edge)? The system needs to know not to put a fire there.

• The authors added a rule: If a "dangerous" data point gets too much heat from the nearby campfires, the system automatically dims the fires around it until the dangerous spot is cold again.
• This ensures the safety zone never accidentally covers a known danger.

4. Why This Matters for Certification

To get a plane or a car approved for use, regulators need to know the rules are solid.

• Deterministic: The paper claims that if you run this process twice with the same data, you get the exact same safety map every time. It's not a "black box" guess; it's a mathematical calculation.
• Order-Independent: It doesn't matter if you feed the data into the computer in the morning or the afternoon, or in a different order. The result is always the same.
• Conservative: If the system isn't sure if a spot is safe (because there are no data points there), it assumes it's unsafe. This is a "better safe than sorry" approach, which is crucial for safety-critical systems.

5. The Proof: The "Flight Simulator" Test

The authors tested this method in two ways:

1. Math Simulation: They created a fake, perfect safety zone on a computer and then tried to rebuild it using only scattered data points. Their "glowing net" method recreated the original zone with over 98% accuracy.
2. Real-World Aviation: They applied it to a real aviation problem: Collision Avoidance. They used data from a system designed to stop planes from hitting each other. The method successfully mapped out the safe operating conditions for this complex system, proving it works even with real, messy data.

Summary

This paper presents a tool (called autoSAFE) that takes raw data from a safety-critical system and automatically draws a precise, mathematically proven "safety zone" around it. Instead of guessing the rules, it learns the boundaries from the data itself, ensuring that the AI only operates where it has been proven to be safe. This makes it much easier to certify AI systems for things like flying planes or driving cars.

Technical Summary

Technical Summary: Defining Operational Conditions for Safety-Critical AI-Based Systems from Data

1. Problem Statement

The rapid deployment of Artificial Intelligence (AI) in safety-critical domains (e.g., aviation, automotive) necessitates rigorous safety assurance and certification. A central requirement for certifying AI-based systems is the definition of the Operational Design Domain (ODD)—the specific set of environmental and operational conditions under which the system is intended to function safely.

Traditionally, ODDs are defined manually by domain experts early in the development lifecycle. However, for complex real-world systems, this approach faces significant challenges:

• Complexity: Defining interrelations (ontology) between parameters (e.g., how weather affects landing speed) is difficult and often incomplete.
• Static Nature: Expert-defined ODDs may not capture implicit parameter dependencies or adapt to new data.
• Certification Gaps: Current data-driven approaches often lack deterministic, order-independent representations or formal frameworks for ODD similarity, which are prerequisites for formal certification.

Existing mathematical representations (e.g., convex polytopes) fail to model non-linear ODD ontologies, while neural network-based approaches introduce order-dependence and uncertainty. There is a need for a method to derive ODDs directly from data that is deterministic, order-independent, bounded, and interpretable.

2. Methodology

The paper proposes a Safety-by-Design method to define ODDs a posteriori from collected data using a multi-dimensional kernel-based representation. The core methodology involves the following steps:

2.1 Mathematical Formalization

The authors formalize the ODD as a mathematical structure $O = (X, R^O, f^O, \Omega^O)$, where $X$ is the taxonomy (parameter space), $R^O$ is the ontology (constraints), and $f^O$ is the interpretation function. Crucially, they define ODD similarity ($O_1 \sim O_2$) not by semantic equivalence, but by data-centric equivalence: two ODDs are similar if they generate the same dataset $Y$.

2.2 Kernel-Based Affinity Representation

Instead of manually constructing boundaries, the method constructs the ODD directly from data samples:

• Anchor Points: In-distribution (ID) samples ($D_{ID}$) serve as anchor points ($A$). Out-of-distribution (OOD) samples ($D_{OOD}$) are explicitly excluded.
• Local Affinity: For each anchor point $x_i$, a local affinity function $\alpha_i(x)$ is defined using a positive-definite kernel (specifically the Radial Basis Function (RBF) kernel).
• Global Affinity: The global ODD membership is determined by the superposition of local affinities:
  $$\alpha(x) = 1 - \prod_{i} (1 - \alpha_i(x))$$
  A sample $x$ belongs to the ODD if $\alpha(x) \geq \zeta$, where $\zeta$ is a predefined threshold.

2.3 Automated Parameterization

To ensure the process is fully automated and avoids manual tuning:

• Diagonal Covariance: The kernel covariance matrix $\Sigma$ is assumed to be diagonal, based on the assumption of local independence and uniform anchor point distribution.
• Distance-Dependent Scaling: The diagonal entries of $\Sigma$ are defined as a function of the distance to the nearest neighbor ($d^*_i$):
  $$\sigma^{(i)}_{kk} = (\kappa - \lambda) \exp(-\eta d^*_i) + \lambda$$
  This reduces the number of hyperparameters from $N \times n^2$ to just two ($\kappa, \eta$) plus a lower bound $\lambda$.

2.4 OOD Consistency Constraint

To ensure safety, the method enforces that OOD samples must not be classified as part of the ODD. If an OOD sample $x$ violates the threshold ($\alpha(x) > \xi$), the algorithm iteratively scales down the covariance matrix of the most influential kernel for that point. This process is order-independent and proven to terminate in finite steps, ensuring $\alpha(x) \leq \xi$ for all OOD samples.

3. Key Contributions

1. Deterministic, Order-Independent Framework: The paper presents the first fully automated pipeline to derive ODDs from data using a kernel-based representation that is uniquely determined by the data and invariant to sample ordering.
2. Formal Definition of ODD Similarity: A data-centric definition of ODD similarity is introduced, enabling the comparison of data-driven ODDs against ground truth or proxies without requiring semantic alignment.
3. Safety-by-Design Implementation: The resulting ODD representation is bounded, conservative (under-approximates the true ODD in sparse regimes), and interpretable, satisfying key requirements for future AI certification.
4. Handling Sparse Data: The method is designed to function effectively even with sparse data regimes, making it applicable to early development stages.
5. Open-Source Tool (autoSAFE): The authors developed and open-sourced a tool (autoSAFE) that implements this framework, supporting various data formats (CSV, JSON/ASAM OpenLABEL) and efficient nearest-neighbor searches.

4. Validation and Results

The methodology was validated through two primary experiments:

4.1 Monte Carlo Simulation

• Setup: A synthetic 2D ODD was generated with a linear inequality constraint. Anchor points were sampled from the true ODD, and validation samples were generated across a wider hyperrectangle.
• Findings: The data-driven ODD was compared against the true underlying ODD and the convex hull of anchor points.
  • Precision and recall curves showed a strong correlation between the data-driven ODD and the convex hull ($R^2 = 0.9855$ for precision, $R^2 = 0.9987$ for recall).
  • This suggests that the convex hull can serve as a reliable proxy for tuning affinity thresholds when the ground-truth ODD is unknown.
  • Results held for dimensions up to 10 and complex relationship functions.

4.2 Real-World Aviation Use Case (VCAS)

• Context: The method was applied to the Vertical Collision Avoidance System (VCAS), a component of the next-generation ACAS X system.
• Data: The dataset included 622,110 anchor points representing state vectors (relative altitude, vertical rates, time to CPA, previous advisory).
• Findings:
  • The data-driven ODD was compared against the known ground-truth ODD and the convex hull.
  • High coefficients of determination were observed ($R^2 = 0.991$ for precision, $R^2 = 0.999$ for recall).
  • The results confirmed that the kernel-based approach performs well in a realistic, high-dimensional safety-critical scenario, despite the lower density of anchor points compared to the Monte Carlo simulation.

5. Significance and Claims

The paper claims that while a "fully truthful" reconstruction of an ODD is not always possible from data alone, the proposed kernel-based approach sufficiently approximates the underlying ODD for certification purposes.

• Certification Readiness: The method provides a mathematically rigorous, deterministic, and explainable basis for defining operational boundaries. This supports regulatory frameworks (e.g., EASA) that require continuous safety assessment and clear operational limits.
• Runtime Monitoring: Unlike rigid geometric boundaries, the continuous affinity function $\alpha(x)$ allows for graded warning zones. As the system state drifts from anchor points, the affinity score decays smoothly, enabling early detection of out-of-distribution conditions before a hard safety boundary is crossed.
• Conservatism: The approach is inherently conservative. In sparse data regimes, the derived ODD under-approximates the true safe region, ensuring the system is only deployed in areas supported by sufficient training data. This aligns with the Safety-by-Design principle.
• Limitations: The authors acknowledge that the method relies on kernel parameters ($\kappa, \eta$) and that the diagonal covariance assumption simplifies cross-dimensional dependencies. They note that the resulting ODD may be overly conservative in sparse regions, potentially excluding valid operational conditions, but argue this is a necessary trade-off for safety assurance.

In conclusion, the work establishes a pathway for data-driven ODD construction that is compatible with formal safety arguments, bridging the gap between empirical data and the rigorous requirements of AI certification in safety-critical systems.