💻 computer science

Explaining Failures of Cyber-Physical Systems with Actual Causality

This paper introduces a novel framework and two practical algorithms for explaining failures in complex, black-box Cyber-Physical Systems by leveraging actual causality, addressing theoretical gaps and demonstrating effectiveness on a neural-network-controlled autonomous car.

Original authors: Khen Elimelech, Tom Yaacov, David A. Kelly, Hana Chockler, Moshe Y. Vardi

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Khen Elimelech, Tom Yaacov, David A. Kelly, Hana Chockler, Moshe Y. Vardi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a self-driving car that suddenly crashes. Because the car's "brain" is a complex, black-box neural network (like a human brain made of code), we can't just look inside and see exactly why it made that mistake. We know that it crashed, but we don't know which specific part of the road or environment caused the disaster. Was it a specific pothole? A sudden shadow? A weirdly placed cone?

This paper is like a detective manual for finding the culprit in these crashes. It introduces a new way to explain failures using a concept called "Actual Causality."

Here is a breakdown of the paper's ideas using simple analogies:

1. The Problem: The "Black Box" Crash

Think of the self-driving car as a magician. You see the trick (the car driving), and sometimes you see the trick fail (the crash). But you can't see the hidden mechanics inside the hat.

The Issue: When the car crashes, it's often because of a mix of many things: the shape of the road, the weather, and the position of obstacles.
The Goal: We need to point a finger at the specific things that caused the crash, so we can fix the car or avoid that situation in the future.

2. The Old Way vs. The New Way

The Old Way (General Causality): This is like asking, "What usually causes car accidents?" It looks at statistics and says, "Rain causes accidents." This is good for predicting the future, but it doesn't help explain this specific crash that happened on a sunny day.
The New Way (Actual Causality): This is like a detective asking, "What specifically caused this crash?" It looks backward at the exact moment of failure. It asks: "If we had removed this one specific rock from the road, would the car still have crashed?" If the answer is "No, it would have been fine," then that rock is the culprit.

3. The Tricky Part: Why Cars Are Harder Than Photo Apps

The authors explain that you can't just use the same "detective tools" used for simple computer programs (like an app that identifies cats in photos).

The Photo App Analogy: If an app thinks a picture of a dog is a cat, you can just look at the pixels. You can ask, "If I turn off these 5 pixels, does it still think it's a cat?" It's a static, one-time snapshot.
The Car Analogy: A car is moving. It sees the road piece by piece.
- The "Blind Spot" Problem: Imagine a car crashes at mile 1. There is a giant boulder at mile 10. The car never saw the boulder at mile 10, so it couldn't have caused the crash. The paper says we must ignore things the car didn't "see" before it crashed.
- The "Type of Crash" Problem: A crash isn't just a "crash." Crashing into a tree is different from crashing into a wall. If we change the environment too much, we might change the type of crash entirely. The paper argues we need to explain the specific way the car failed, not just the fact that it failed.

4. The Solution: Two Detective Algorithms

The paper proposes two methods (algorithms) to find the culprit, depending on how much time and computer power you have.

Method A: The "Exhaustive Search" (The Perfect Detective)

How it works: This detective tries every single possible combination of obstacles. It asks, "What if I remove this rock? What if I remove that cone? What if I remove both?"
Pros: It is guaranteed to find the absolute smallest, most perfect list of causes.
Cons: It is incredibly slow. If there are many obstacles, the number of combinations is so huge it would take a supercomputer years to check them all.

Method B: The "Responsibility-Guided Search" (The Smart Detective)

How it works: This detective is a bit lazy but very smart. Instead of checking everything, it first does a quick "sniff test" to guess which obstacles are the most "responsible" for the crash. It ranks them from "most likely culprit" to "least likely." Then, it only checks the top suspects.
Pros: It is much, much faster. It finds a good answer in seconds.
Cons: Sometimes it might include a few extra suspects in the list that weren't strictly necessary, but it gets very close to the perfect answer.

5. The Test Drive

The authors tested these methods on a simulated self-driving car on a track full of obstacles.

The Result: The "Perfect Detective" (Exhaustive) found the smallest list of causes but took a huge amount of time. The "Smart Detective" (Responsibility-Guided) found the answer much faster.
The Sweet Spot: They found a middle-ground version of the Smart Detective that was fast and almost always found the perfect, smallest list of causes.

Summary

This paper gives us a new mathematical toolkit to explain why complex, AI-driven machines fail. It teaches us how to ignore the things the machine didn't see and how to distinguish between different types of failures. By using these new "detective algorithms," we can move from saying "The car crashed" to saying "The car crashed because of these specific three rocks," which helps engineers build safer, more trustworthy robots and cars.

Technical Summary: Explaining Failures of Cyber-Physical Systems with Actual Causality

Problem Statement

Modern autonomous Cyber-Physical Systems (CPSs), such as self-driving cars, increasingly rely on "black-box" Deep Neural Network (NN) components for perception and control. While these systems must operate robustly in complex, unseen environments, their opacity makes formal verification prior to deployment intractable. Consequently, unexpected failures—where the system violates its specification—are inevitable and can lead to catastrophic outcomes.

Current methods for explaining these failures are insufficient. General or type causality frameworks are forward-looking and probabilistic, designed for prediction rather than explaining specific, rare past events. Furthermore, existing applications of actual causality (a backward-looking framework for specific events) have been limited to simple, depth-2 models like image classifiers. Applying these standard frameworks directly to temporally extended CPSs (which involve continuous dynamics, recursive loops, and variable input sizes) leads to incorrect explanations. Specifically, naive applications fail to account for:

The distinction between global environment parameters and local elements.
The fact that CPSs only observe a subset of the environment before a failure occurs.
The existence of distinct "types" of failures (e.g., collisions at different trajectory points) that a simple binary success/failure indicator cannot distinguish.

Methodology

The paper proposes a novel framework that adapts the theory of actual causality to the specific constraints of CPS failure analysis.

1. Theoretical Extensions to Actual Causality

The authors extend the standard actual causality framework (originally defined for classifiers) to handle CPSs by introducing three critical modifications to the causal model construction:

Input Differentiation (Parameters vs. Elements): The environment is split into parameters (global, essential variables like track shape) and elements (local, additive variables like obstacles). In the causal model, parameters are treated as exogenous constants (frozen in their original state) because changing them alters the fundamental type of failure. Only elements are included as endogenous variables subject to intervention (masking).
Observation Constraints: Unlike classifiers which process the entire input at once, CPSs process the environment incrementally over time. The framework restricts potential explanations to only those environment elements that were observed by the system prior to the failure. Unobserved elements are excluded from the explanation set to prevent logically invalid causality (e.g., blaming an obstacle the car never saw).
Failure Type Abstraction: To avoid the issue where infinitesimal trajectory changes break the "same output" condition required for causality, the authors propose abstracting the continuous trajectory into discrete "failure types." The outcome variable $O$ is defined not merely as a binary crash/no-crash, but as a specific class of failure (e.g., "collision with obstacle $X$ "). This ensures the explanation identifies the specific cause of the observed failure event.

2. Explanation Derivation Algorithms

The paper presents two system-agnostic algorithms to derive sufficient explanations (minimal subsets of observed elements that, if kept at their original values, guarantee the failure occurs):

Exhaustive Search (Algorithm 1): This algorithm performs a brute-force search over all subsets of observed elements. It intervenes on the environment by removing elements and re-simulating to check if the failure type is preserved. While it guarantees an optimal (minimal) explanation, it is computationally intractable for complex scenarios due to exponential growth in simulation runs.
Responsibility-Guided Search (Algorithm 2): To address efficiency, the authors propose a heuristic algorithm based on causal responsibility.
1. Approximation: It samples random interventions to estimate the degree of responsibility for each observed element (based on how often its removal changes the outcome).
2. Heuristic Search: Elements are ranked by responsibility. The algorithm incrementally adds high-responsibility elements to a candidate explanation until the failure type is preserved.
3. Minimization (Optional): An optional post-processing step performs an exhaustive search on the resulting candidate subset to ensure minimality.

Key Contributions

Conceptual Framework: The paper introduces the application of actual causality to CPS failure explanation, identifying and resolving theoretical gaps that arise when moving from static classifiers to dynamic, temporally extended systems.
Correctness Guidelines: It provides specific guidelines for constructing causal models for CPSs, emphasizing the handling of unobserved elements and the abstraction of failure types.
Practical Algorithms: It delivers two algorithms: one for exact, optimal explanation derivation and a second, efficient, responsibility-guided heuristic that balances accuracy with computational cost.
Empirical Validation: The approach is evaluated on a neural-network-controlled autonomous car navigating an obstructed track, demonstrating the trade-offs between explanation optimality and derivation efficiency.

Experimental Results

The authors evaluated the algorithms using the LiteRacer simulator with 32 falsifying traces involving 6, 9, 12, and 15 obstacles.

Accuracy: The Exhaustive Search (ES) consistently produced minimal explanations. The Responsibility-Guided (RG) algorithm without minimization produced larger (less accurate) explanations, with error rates increasing as obstacle count rose. However, the Responsibility-Guided with Minimization (RGM) variant successfully recovered the exact minimal explanation in all tested cases.
Efficiency: ES required a number of simulation runs that grew exponentially with the number of obstacles (e.g., ~1,152 runs for 15 obstacles). In contrast, the RG and RGM variants required drastically fewer runs (e.g., ~200–400 runs), scaling much more moderately.
Trade-off: The results highlight a clear trade-off: RG offers high efficiency but lower accuracy; RGM offers a balanced approach, achieving near-optimal accuracy with significantly reduced computational effort compared to ES.

Significance and Claims

The paper claims to open a novel direction for understanding and handling autonomous CPS failures by rooting the process in the rigorous mathematical theory of actual causality. The authors assert that:

Standard causality frameworks are insufficient for CPSs due to temporal extension and partial observability.
The proposed guidelines are necessary to derive correct explanations; naive application leads to logically invalid results.
The framework provides a "system-agnostic" method, meaning it does not require access to the internal weights of the neural network (purely black-box), making it applicable to proprietary systems.
The work serves as a foundational step toward explainable and trustworthy autonomous systems, specifically by enabling the localization of failure causes and the generation of new failure scenarios for mitigation.

The authors remain modest regarding the current scope, noting that the framework currently handles sufficient explanations (minimal sets causing the failure) and relies on depth-2 causal models. Future work is identified as extending the framework to capture the continuous, temporally extended nature of CPSs more deeply and exploring necessary explanations.