🤖 machine learning

Verifying Machine Learning Interpretability Requirements through Provenance

This paper proposes a method to verify the non-functional requirement of interpretability in machine learning models by leveraging machine learning provenance to transform the otherwise immeasurable requirement into quantifiable functional requirements through the systematic recording of model and data lineage.

Original authors: Lynn Vonderhaar, Juan Couder, Daryela Cisneros, Omar Ochoa

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lynn Vonderhaar, Juan Couder, Daryela Cisneros, Omar Ochoa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've hired a brilliant but mysterious chef to cook a special dish for a very important dinner. This chef uses a secret recipe (a Machine Learning model) to make the food. The problem is, the chef won't tell you why the dish tastes the way it does, or even what ingredients went into it. You just have to trust that the food is safe and good.

In the world of Artificial Intelligence, this is a huge problem. We rely on these "black box" models for everything from diagnosing diseases to approving loans, but we often can't explain how they make decisions. This paper proposes a solution: The "Digital Recipe Book" (Provenance).

Here is the breakdown of the paper's idea in simple terms:

1. The Problem: The "Black Box" Mystery

Machine Learning models are like magic boxes. You put data in, and an answer comes out. But inside the box, it's a jumble of math that humans can't easily read.

The Requirement: We need these boxes to be "interpretable," meaning we need to be able to understand why they made a decision.
The Catch: Currently, "interpretability" is a vague goal. It's like saying, "Make this car safe," without defining what "safe" actually means. You can't measure it, so you can't prove you did it.

2. The Solution: The "Digital Recipe Book" (Provenance)

The authors suggest that to prove a model is understandable, we shouldn't just look at the final answer. Instead, we need to save the entire history of how the model was built. They call this Provenance.

Think of Provenance as a super-detailed, unchangeable diary that records every single step of the cooking process:

Where the ingredients (data) came from.
Who chopped them (data cleaning).
What spices were added (hyperparameters).
The exact temperature of the oven (environment settings).
Every time the recipe was tweaked.

By saving this "diary," we create a trail of evidence. If we have the full diary, we can retrace the steps and see exactly how the chef arrived at the final dish.

3. How It Works: Turning "Maybe" into "Proof"

The paper argues that "Interpretability" is too hard to measure directly. So, they break it down into smaller, measurable tasks called Functional Requirements.

Instead of asking, "Is this model interpretable?" (which is hard to answer), they ask:

"Did we save the list of ingredients?" (Yes/No)
"Did we save the steps taken to clean the data?" (Yes/No)
"Did we save the final formula?" (Yes/No)

If the answer to all these small questions is YES, then the big question ("Is it interpretable?") is automatically YES.

4. The Experiments: Two Ways to Use the Diary

The authors tested this idea with two different types of "chefs" (models):

Experiment A: The Transparent Chef (Linear Regression)

The Model: A simple model that draws a straight line through data points.
The Trick: Because the model is simple, if you have the "diary" (the data and the math), you can literally rebuild the model from scratch on a piece of paper and get the exact same result.
The Result: They saved the data and the math in their digital diary. They then used that diary to recreate the model in Excel. It worked perfectly. This proved that if you save the history, you can simulate the behavior.

Experiment B: The Mystery Chef (Counterfactual Explanations)

The Model: A complex model that predicts if someone earns a high income. It's too complicated to rebuild on paper.
The Trick: Instead of rebuilding the whole model, they used the "diary" to ask "What If?" questions.
- Example: "If this person worked 5 more hours a week, would the prediction change?"
The Result: By looking at the saved history of how the model was trained, they could map out exactly which inputs (like age or education) caused the model to say "Yes" or "No." They visualized this as a map, showing exactly where the model draws the line between "High Income" and "Low Income."

5. The Takeaway

The paper concludes that we don't need to invent new magic to make AI understandable. We just need to be better at taking notes.

Current State: We build AI and hope it makes sense.
Proposed State: We build AI while keeping a strict, digital "recipe book" of every single step.

If we save the right notes (Provenance), we can prove that the AI is transparent. We can show regulators, doctors, or customers: "Here is the exact path we took to get this result. You can follow it yourself."

In short: You can't verify a mystery if you don't keep a record of the clues. This paper says, "Start keeping the record, and the mystery solves itself."

1. Problem Statement

The rapid adoption of Machine Learning (ML) in safety-critical domains necessitates the application of rigorous Software Engineering and Requirements Engineering (RE) practices. While literature has defined ML-specific Non-Functional Requirements (NFRs), such as interpretability, a significant gap exists in verification.

The Core Issue: Interpretability is currently defined as an immeasurable NFR. Without a quantifiable metric or a clear verification method, it is impossible to definitively confirm whether an ML model meets its interpretability requirements.
The Challenge: Existing methods often lack a structured way to decompose high-level interpretability goals into verifiable, low-level system requirements.

2. Methodology

The authors propose a novel approach to verify interpretability by decomposing the high-level NFR into ML Provenance requirements, which are then further decomposed into verifiable Functional Requirements (FRs).

A. Conceptual Framework

Provenance Definition: Defined as the origin, history, and transformations of data and models throughout the ML lifecycle. It includes datasets, preprocessing steps, hyperparameters, and code.
Decomposition Strategy:
1. Interpretability NFR $\rightarrow$ Provenance Requirement (The need to save specific historical data).
2. Provenance Requirement $\rightarrow$ Functional Requirements (FRs) (Specific system actions to save, log, and trace specific data artifacts).
3. Verification: By verifying that the FRs (saving specific provenance) are met, the system implicitly verifies the Interpretability NFR.

B. Technical Implementation

Tooling: The authors utilize yProv4ML, a Python library compliant with the W3C PROV-O (Provenance Ontology). This tool structures provenance data using the Activity, Entity, and Agent classes, outputting data in PROV-JSON and graph formats.
Data Categories: The approach tracks two main categories of provenance:
- Data Provenance: Source, collection method, preprocessing (cleaning, normalization), labeling, and splits.
- Model Provenance: Architecture, hyperparameters, optimization algorithms, environment configurations, and source code.

C. Experimental Validation

The authors validated their approach using two experiments representing different categories of interpretability methods (based on a taxonomy of Inherently Interpretable, Surrogate, and Feature Attribution):

Experiment 1: Linear Regression (Inherently Interpretable)
- Goal: Simulate the model's behavior.
- FRs: Save training/testing datasets, preprocessing steps, and the resulting linear function.
- Verification: The saved provenance was used to manually reconstruct the linear regression line in Microsoft Excel, successfully replicating the model's predictions. This proved that saving provenance allows for the simulation of inherently interpretable models.
Experiment 2: Counterfactual Explanations (Feature Attribution)
- Goal: Map model behavior to input space (since the underlying black-box model cannot be directly simulated).
- FRs: Save datasets, preprocessing, feature selection, and the generated counterfactual explanations.
- Verification: The saved provenance allowed developers to visualize how specific input changes (e.g., age, education) affected the model's output (e.g., income prediction), effectively mapping the decision boundary in human-understandable terms.

3. Key Contributions

Requirement Decomposition: A formal method to decompose the abstract NFR of "Interpretability" into concrete, verifiable Provenance Requirements.
Verification Mechanism: A proof-of-concept demonstrating that verifying Functional Requirements related to data/model provenance storage effectively verifies the Interpretability NFR.
Structured Framework: The application of W3C PROV-O and yProv4ML to standardize how ML development artifacts are tracked, moving beyond simple version control to semantic provenance.
Empirical Validation: Two distinct experiments showing that this approach works for both models that can be simulated (Linear Regression) and complex black-box models requiring input-space mapping (Counterfactuals).

4. Results

Success in Simulation: For the Linear Regression model, the saved provenance allowed for the exact reproduction of the model's mathematical function, confirming that the model's behavior was transparent and interpretable.
Success in Mapping: For the Counterfactual experiment, the provenance data successfully documented the relationship between input features and model decisions, allowing stakeholders to understand why a specific prediction was made.
Tool Limitations Identified: The authors noted that current tools (like yProv4ML) sometimes force incorrect categorizations (e.g., saving preprocessing steps as "model parameters" rather than "data provenance"), highlighting a need for more specialized provenance tools.

5. Significance and Future Work

Bridging RE and ML: This work bridges the gap between traditional Requirements Engineering and ML Engineering, providing a concrete path to satisfy regulatory and safety requirements for interpretability.
Quantifying the Unquantifiable: It transforms "interpretability" from a subjective quality into a verifiable set of data artifacts.
Future Directions:
- Expanding the verification to a broader survey of interpretability methods (e.g., LIME, SHAP, Decision Trees).
- Clarifying the distinction between interpretability and explainability and verifying provenance for both.
- Developing a dedicated ML provenance tracking tool that correctly distinguishes between data transformations and model parameters.

In conclusion, the paper argues that transparency through provenance is the key to verifying interpretability. By rigorously saving the "history" of a model's creation and behavior, engineers can provide the necessary evidence to confirm that a model meets its interpretability requirements.