🤖 machine learning

Mechanistic Interpretability for Learning Assurance of a Vision-Based Landing System

This paper proposes and demonstrates a novel assurance framework for vision-based aircraft landing systems that satisfies EASA guidance by using mechanistic interpretability to separate content from style in a model's situation representation, thereby enabling reliable prediction reliance on task-relevant features and the development of runtime out-of-model-scope detection.

Original authors: Romeo Valentin, Olivia Beyer Bruvik, Marc R. Schlichting, Mykel J. Kochenderfer

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Romeo Valentin, Olivia Beyer Bruvik, Marc R. Schlichting, Mykel J. Kochenderfer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a robot pilot that learns to land airplanes just by looking at pictures of runways. In the past, engineers could explain exactly how every part of a machine worked, like a mechanic tracing a wire to a switch. But with modern "neural network" AI, the system learns from data in a black box; we know what goes in (a picture) and what comes out (a landing command), but we don't know how it decided that.

This is a problem for aviation safety. Regulators (like EASA in Europe) say: "We can't just trust the AI because it got good test scores. We need to see its internal 'thought process' to make sure it's thinking about the right things."

This paper proposes a new way to peek inside that black box and prove the AI is thinking correctly. Here is how they did it, explained simply:

1. The Problem: The AI Might Be Cheating

Imagine you are teaching a student to identify a cat. If you only show them photos of cats sitting on red carpets, the student might learn to recognize "red carpets" instead of "cats." If you then show them a cat on a blue rug, they might fail.

In aviation, an AI might learn to land by recognizing specific colors or lighting conditions (like "sunny days look like this") rather than the actual shape of the runway. This is called relying on style (the look) instead of content (the structure). If the AI relies on style, it might crash when the weather changes.

2. The Solution: Disentangling the "Thoughts"

The researchers treated the AI's internal brain activity like a messy pile of ingredients. They wanted to separate the essential ingredients (the runway shape, the edges, the markings) from the flavorings (the color of the sky, the grain of the photo, the simulator graphics).

They used a mathematical tool called K-SVD to break the AI's internal "thoughts" (called embeddings) into small, distinct building blocks called atoms.

Contentful Atoms: These are like the blueprint of the runway. They light up when the AI sees a runway edge or a threshold marking, no matter if the photo is from a video game, a satellite, or a real camera.
Stylistic Atoms: These are like the filter on a photo. They light up only when the AI sees "Google Earth green grass" or "motion blur."

3. The Test: Did the AI Listen to the Right Atoms?

Once they separated the atoms, they asked a simple question: When the AI makes a decision to land, which atoms is it listening to?

They found that the AI's "decision-making head" (the part that calculates the landing spot) placed almost 82% of its trust on the Contentful Atoms. It mostly ignored the Stylistic Atoms.

The Analogy: It's like a chef who tastes a soup. If the chef says, "I'm adding salt because the soup tastes bland (content)," that's good. If the chef says, "I'm adding salt because the bowl is blue (style)," that's bad. This paper proved the chef is mostly tasting the soup, not looking at the bowl.

4. The New Safety Net: "Out-of-Model-Scope" (OOMS)

The paper introduces a new safety monitor called Out-of-Model-Scope (OOMS).

Current Safety Nets: Usually, we check if the input is weird (e.g., "Is this a picture of a banana?") or if the output is weird (e.g., "Did the AI say the runway is on the moon?").
The New Monitor (OOMS): This checks the AI's internal thoughts before it makes a decision. It asks: "Does the AI's brain currently contain the 'Runway Blueprint' atoms?"

How it works:
The researchers built a simple alarm system. If the AI looks at an image and its internal "atoms" don't light up with the necessary runway structures (perhaps because the runway is hidden behind a cloud or the image is just sky), the alarm goes off. The system says, "I don't have enough evidence to land," and discards the prediction.

They tested this on a tricky scenario where the runway was completely outside the camera frame. The AI tried to guess anyway, but the OOMS monitor caught it because the "runway atoms" were missing. The monitor was 96% accurate at spotting these dangerous situations.

Summary

This paper doesn't just say "the AI works." It provides a mechanistic explanation of why it works.

They broke the AI's brain into "Runway Facts" and "Photo Filters."
They proved the AI relies mostly on "Runway Facts."
They built a new safety switch that turns off the AI if it ever stops looking at "Runway Facts."

This gives regulators a concrete way to verify that a self-learning airplane system is actually understanding the runway, not just memorizing the look of the pictures.

Technical Summary: Mechanistic Interpretability for Learning Assurance of a Vision-Based Landing System

Problem Statement
The integration of data-driven neural networks into safety-critical aviation systems challenges traditional assurance paradigms, which rely on specifying and verifying every component step. While the European Union Aviation Safety Agency (EASA) has issued guidance requiring data-driven systems to build and monitor their own "situation representation," the technical means to provide such evidence for neural networks remain an open problem. Current assurance often relies on testing against held-out data, which fails to distinguish between models that have learned generalizing mechanisms versus those that rely on spurious correlations. Specifically, there is a critical gap in monitoring the model's internal situation representation—the features computed to bridge input and output—rather than just the input conditions (Operational Design Domain, ODD) or the output statistics (Out-of-Distribution, OOD).

Methodology
The authors propose a framework based on mechanistic interpretability to address this gap, applied to a vision-based runway keypoint regression task. The methodology proceeds through the following stages:

Model Architecture: A Vision Transformer (ViT) backbone (DINOv2-S) is adapted for runway keypoint regression using a Soft-Argmax (SAM) operator. Unlike Convolutional Neural Networks (CNNs) that often discard spatial information via pooling, the ViT preserves patch-level spatial structure. The SAM operator maps patch embeddings directly to continuous coordinate estimates without a pooling layer, ensuring the regression head is minimal and the burden of solving the task lies within the patch embeddings themselves.
Sparse Disentanglement (K-SVD): The high-dimensional patch embeddings ( $z_p$ ) are decomposed into interpretable "atoms" using K-SVD sparse dictionary learning. This process assumes the linear representation hypothesis, where semantic concepts correspond to directions in the embedding space. The model learns an overcomplete dictionary $D$ and sparse codes $X$ such that $z_p \approx DX$ .
Content vs. Style Separation: The learned atoms are categorized into two types based on cross-subset invariance across four distinct visual domains in the LARDv2 dataset (X-Plane, Google Earth Studio, ArcGIS, and Bing Maps):
- Contentful Atoms: Activate consistently across all domains, tracking task-relevant geometric structures (e.g., runway edges, thresholds).
- Stylistic Atoms: Activate predominantly in specific domains, tracking incidental appearance factors (e.g., color grading, rendering artifacts, lighting).
- Classification is performed using the Coefficient of Variation (CV) of activation rates across subsets.
Validation: The separation is validated qualitatively via visualization of activating patches and quantitatively by measuring the SAM Reliance Score. This score aggregates the alignment of atoms with the regression head's weights, their activation frequency, and activation magnitude to determine how much the model's predictions rely on contentful versus stylistic atoms.
Out-of-Model-Scope (OOMS) Detection: Building on the content/style split, the authors define OOMS as a state where the input lacks sufficient task-relevant (contentful) features, even if the input appears valid (in-ODD) and the output is statistically typical. An OOMS detector is implemented as a constrained L1 logistic regression classifier over binary atom activations. The classifier is trained to output a probability of being "In-Model-Scope" (IMS), with constraints ensuring that the default state is OOMS and that only active atoms can increase the IMS probability.

Key Contributions

Adaptation of ViT for Landing: The authors demonstrate a simple, efficient adaptation of the Vision Transformer with a Soft-Argmax head for runway keypoint regression, showing it is trainable on the LARDv2 dataset.
Mechanistic Decomposition: They successfully decompose the internal feature space of the ViT into contentful geometric structures and stylistic factors using K-SVD, establishing a foundation for mechanistic interpretability in this domain.
Validation of Representation: Through qualitative visualization and quantitative head-weight analysis, they confirm that the regression head relies primarily (approx. 80%) on contentful atoms, validating the model's situation representation.
OOMS Monitoring: The paper introduces Out-of-Model-Scope (OOMS) detection as a novel runtime assurance artifact. Unlike ODD (input monitoring) or OOD (output monitoring), OOMS directly monitors the model's internal situation representation to verify the presence of task-relevant semantic features.

Results

Regression Performance: The pretrained ViT model achieved a median test-set pixel error of 1.91 px, significantly outperforming a from-scratch baseline (3.19 px).
Content/Style Reliance: Quantitative analysis showed that the regression head places approximately 81.7% of its effective reliance score on contentful atoms for the pretrained model, and 79.8% for the from-scratch model.
OOMS Detection: On an artificial dataset distinguishing between valid runway crops (IMS) and "inverse-BOGO" crops where the runway is absent (OOMS), the constrained logistic regression classifier achieved an AUROC of 0.96.
Atom Selection: The OOMS classifier automatically selected a sparse subset of 34 atoms (out of 512) to achieve high accuracy. Notably, 78% of these selected atoms were classified as contentful, demonstrating that the detector relies on the task-relevant features identified during the interpretability phase.

Significance and Claims
The paper claims to deliver the first concrete piece of representation-level evidence demanded by EASA's learning-assurance guidance. By demonstrating that a minimally assurable model must separate content from style and rely on the former, the work provides a pathway to verify that a model is not exploiting spurious correlations. The proposed OOMS monitor offers a practical building block for future aviation safety cases, complementing existing input and output monitors by directly addressing the requirement to monitor the model's internal situation representation. The authors position mechanistic interpretability not as a full white-box explanation (which remains infeasible for high-dimensional models) but as a necessary condition for assurance: verifying that the model's internal mechanics align with the physical task requirements.