🤖 machine learning

Causal Explanations from the Geometric Properties of ReLU Neural Networks

This paper proposes a method for generating accurate causal explanations for ReLU neural networks by directly extracting decision rules from their geometric structure as piecewise linear functions defined by convex polytopes, thereby avoiding the performance degradation and fidelity issues associated with distilling models for interpretability.

Original authors: Hector Woods, Philippa Ryan, Rob Alexander

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Hector Woods, Philippa Ryan, Rob Alexander

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have built a very smart robot driver. This driver is so good at navigating that it never crashes, but it's a "black box." You can't ask it why it turned left instead of right; it just does. This makes people nervous. If something goes wrong, we don't know if the robot is broken or just making a weird choice.

This paper proposes a way to open that black box, specifically for a type of AI called a ReLU Neural Network. Here is how the authors explain it, using simple ideas and metaphors.

The Core Idea: The "Folded Paper" Map

The authors argue that these specific AI networks aren't actually mysterious blobs of math. Instead, they act like a folded piece of paper or a crumpled map.

The Fold: The AI takes a huge, complex world of inputs (like speed, distance, weather) and folds it up into many small, flat, geometric shapes called polytopes. Think of these as tiny, flat rooms.
The Rule: Inside each of these tiny rooms, the AI is actually very simple. It's just doing basic math (a straight line). It's not "thinking" deeply; it's just following one simple rule for that specific room.
The Problem: Because the paper is folded so many times, there are millions of these rooms. To understand the AI, we usually try to look at the whole crumpled ball, which is impossible.

The Solution: Walking the Neighborhood

Instead of trying to map the entire crumpled ball (which would take forever), the authors suggest a smarter way: Polytope Marching.

Imagine you are standing in one of these tiny rooms (the room the AI is currently in). You want to know:

"Why did you turn left?" (Why did it choose this action?)
"Why didn't you turn right?" (Why didn't it choose the other action?)

1. The "Why" Explanation (The Current Room)

To answer "Why did you turn left?", the algorithm looks at the specific room you are standing in.

It checks the walls of that room.
It removes any walls that don't actually matter (redundant constraints).
The Result: It gives you a short list of rules, like "If speed is under 30 and distance is over 10, turn left." This is a Minimally Complete Explanation. It's the shortest, most accurate list of reasons why the AI made that specific choice.

2. The "Why Not" Explanation (The Neighboring Rooms)

To answer "Why didn't you turn right?", the algorithm plays a game of "neighborhood hopping."

It starts in your current room.
It checks the rooms right next to you (by flipping one "switch" or bit in the math).
It asks: "Is there a room next door where turning right would be the best choice?"
If yes: It tells you exactly which wall you would have to cross to get to that room. "You didn't turn right because you are on the 'slow' side of the speed wall. If you were faster, you would turn right."
If no: It keeps hopping to the next nearest rooms until it finds one where turning right makes sense, or it proves that turning right is impossible no matter what.

Why This is Better Than Other Methods

Usually, to explain an AI, scientists try to build a simplified copy (a "distilled" model) of the original AI.

The Analogy: Imagine trying to explain a complex painting by making a stick-figure sketch of it. The sketch is easy to understand, but it's not the real painting. It might miss important details, and you can't guarantee the sketch behaves exactly like the painting.
This Paper's Approach: Instead of making a sketch, they look directly at the geometry of the real painting. They don't simplify the AI; they just find the specific "room" the AI is in and explain the rules of that room. This means the explanation is 100% accurate to the real AI's behavior.

The Catch (Limitations)

The authors admit there are two main hurdles:

Too Many Dimensions: If the AI is looking at a simple game with 3 numbers, the explanation is easy. But if the AI is looking at a photo with thousands of pixels (dimensions), the list of rules becomes huge and hard for humans to read.
The "Why Not" Search: Finding the "Why not" answer is easy if the answer is right next door. But if the AI would have to make a huge change to do something else, the computer has to search through millions of rooms, which takes a long time.

Summary

The paper shows that we can explain AI decisions by treating the AI's brain as a map of geometric rooms. By walking through these rooms and checking the walls, we can give exact, truthful answers to "Why?" and "Why not?" questions without needing to build a simplified, potentially inaccurate copy of the AI. The goal is to make autonomous systems (like self-driving cars or ships) trustworthy by letting us understand their logic.

Technical Summary: Causal Explanations from the Geometric Properties of ReLU Neural Networks

Problem Statement

Neural networks, particularly those used for learning control policies in autonomous systems via Reinforcement Learning (RL), function as "black boxes." This lack of interpretability hinders safety assurance and user trust. While fields like Explainable Artificial Intelligence (XAI) and Explainable Reinforcement Learning (XRL) aim to provide "why" and "why not" explanations, current state-of-the-art methods often rely on distilled models (simplified surrogates). Although these distilled policies are interpretable, they suffer from significant performance degradation compared to the original model and do not guarantee an accurate reflection of the original model's decision-making processes. Consequently, they cannot be used to formally guarantee the safety of the original system. There is a need for methods that generate exact causal explanations directly from the original neural network without approximation or performance loss.

Methodology

The authors propose a method to generate Minimally Complete Explanations (MCE) for neural networks consisting of linear layers and Rectified Linear Unit (ReLU) activation functions. The approach leverages the geometric property that a ReLU network corresponds to a piecewise linear function divided into regions defined by $n$ -dimensional convex polytopes.

The core methodology involves two primary algorithms:

Geometric Representation:
- The network is viewed as dividing the input space into distinct regions (polytopes). Within each polytope, the network's behavior for each output neuron is a single linear transformation.
- Each neuron's activation (zero or non-zero) is represented by a bit vector. A specific combination of activations defines a unique polytope $P$ in the input space, described by a system of linear inequalities (H-Representation).
"Why" Explanations (Factual):
- Input: An input $x$ and the network $N$ .
- Process: The algorithm performs a forward pass to collect the activation signature (bit vector) for $x$ . It constructs the polytope $P$ corresponding to this signature.
- Refinement: It defines a subset polytope $P_{Output} \subset P$ where the actual output class $c$ is maximal. To achieve minimality, the algorithm solves a series of linear programs to identify and remove redundant constraints from the system of inequalities defining $P_{Output}$ .
- Output: A simplified set of linear inequalities ( $H'$ ) that exactly define the region where the network produces the observed decision.
"Why Not" Explanations (Counterfactual):
- Input: An input $x$ , the actual output $c$ , and a counterfactual output $c'$ .
- Process: The algorithm first checks if the counterfactual class $c'$ is feasible within the current polytope $P$ . If feasible, the explanation is the linear inequality distinguishing the classes within $P$ .
- Search: If $c'$ is not feasible in $P$ , the algorithm employs Adjacent Polytope Marching. It iteratively flips bits in the activation signature (increasing Hamming distance) to explore adjacent polytopes until it finds a region $P'$ where $c'$ is feasible.
- Output: The explanation consists of the specific constraints (the differing bits/inequalities) that distinguish the factual polytope $P$ from the counterfactual polytope $P'$ .
Representations:
- H-Representation: A system of linear inequalities. It is exact and scales linearly with the number of neurons but can be complex for high-dimensional inputs.
- V-Representation: A set of extreme points (vertices) defining the bounds of the polytope. This offers greater brevity (constant size relative to network depth) but introduces exponential complexity relative to input dimensionality.

Key Contributions

Exactness: The method generates explanations that are 100% accurate reflections of the original ReLU network's behavior, avoiding the approximation errors inherent in model distillation.
Direct Extraction: Rules are extracted directly from the geometric properties (polytopal decomposition) of the neural network rather than from a surrogate model.
Algorithmic Framework: The paper introduces specific algorithms for generating both "Why" (factual) and "Why Not" (counterfactual) explanations using polytope marching and constraint reduction.
Dual Representation: The work demonstrates how to provide explanations in both H-Representation (constraints) and V-Representation (bounds), discussing the trade-offs between interpretability and computational complexity for each.

Results and Complexity Analysis

Computational Complexity:
- "Why" Explanations: The time complexity is $O(n)$ , where $n$ is the number of neurons, as it involves solving a linear number of linear programs to remove redundant constraints.
- "Why Not" Explanations: In the best case, complexity is $O(n)$ . In the worst case, it is $O(2^n)$ , equivalent to enumerating all possible polytopes. However, the authors note that in practice, the algorithm likely terminates quickly because only a single feasible polytope for the counterfactual is required to halt the search.
Space Complexity: Both algorithms maintain constant space complexity, storing only the current polytope and bit vectors.
Scalability: The authors acknowledge that while the method is theoretically sound, the H-Representation scales linearly with model size (number of constraints), and the V-Representation scales exponentially with input dimensionality. High-dimensional inputs result in constraints involving all input variables, which may reduce interpretability.

Significance and Claims

The paper claims that its primary significance lies in bridging the gap between safety assurance and interpretability. By providing exact, geometric explanations, the method allows users to verify that a system will not enter an unsafe state and understand the specific decision boundaries that led to a particular action.

The authors modestly frame their work as a demonstration of feasibility rather than a complete solution to all interpretability challenges. They explicitly state:

The explanations are "minimally complete" regarding the network's geometry but may not be "satisfying" (lovely) to end-users due to the complexity and number of constraints, particularly in high-dimensional spaces.
The method is currently limited to networks with linear layers and ReLU activations.
Future work is required to validate the "loveliness" of these explanations with real users and regulators (e.g., in the maritime domain) and to extend the approach to other architectures (like CNNs) and activation functions (like Leaky ReLU).

The work positions itself as a foundational step toward mechanistic interpretability that does not compromise the performance or safety guarantees of the original autonomous system.