💻 computer science

I-SAFE: Wasserstein Coherence Metrics for Structural Auditing of Scientific AI Models

The paper introduces I-SAFE, a post-hoc auditing framework that utilizes Wasserstein Coherence Metrics to evaluate the structural alignment of scientific AI models with domain knowledge through input perturbations, revealing distributional inconsistencies that standard accuracy-based benchmarks fail to detect.

Original authors: Barbara Tarantino, Gennaro Auricchio, Paolo Giudici

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Barbara Tarantino, Gennaro Auricchio, Paolo Giudici

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a chef to cook a complex dish. You taste the final meal, and it's delicious. You give them a high score. But here's the problem: Did they actually cook the dish using the right ingredients and techniques, or did they just memorize a specific recipe that happens to taste good on this one night?

In the world of "Scientific AI" (computers trying to solve real-world science problems like drug discovery), we often make the same mistake. We look at a model's test score (how well it predicts things) and assume it understands the science. But the paper argues that a high score is fragile. The AI might be "cheating" by noticing tiny, irrelevant patterns in the data (like the font size of the text or a specific background noise) rather than understanding the actual biological structure.

The authors of this paper, Barbara Tarantino, Gennaro Auricchio, and Paolo Giudici, introduce a new way to audit these AI models called I-SAFE.

Here is how I-SAFE works, explained through simple analogies:

1. The Problem: The "Magic 8-Ball" vs. The "Understanding Chef"

Current AI models are like Magic 8-Balls. You shake them (give them data), and they give an answer. If the answer is right 90% of the time, we are happy. But we don't know why they are right.

The Risk: The AI might be right because it memorized the test questions, not because it understands the chemistry. If you change the question slightly (like changing a specific ingredient in a drug), a "smart" model should react logically. A "cheating" model might react randomly or not at all.

2. The Solution: The "Interventional Audit"

Instead of just asking the AI, "What is the answer?", I-SAFE asks, "What happens if we poke the input?"

Think of the AI model as a black box. The scientists have a map of what should be important (called a Structural Prior). In their experiment, this map was a list of specific "binding pockets" on a protein where drugs actually attach.

The Test: They take a drug and a protein. They create two versions of the protein:
1. The "Real" Poke: They mess with the parts the map says are important (the binding pockets).
2. The "Fake" Poke: They mess with the parts the map says are unimportant (random parts of the protein).
The Goal: They want to see if the AI reacts differently to the "Real" poke than the "Fake" poke. If the AI is truly scientific, changing the important parts should cause a logical, organized change in its prediction. Changing the unimportant parts should cause a messier, less organized change.

3. The Three "Rulers" (The Metrics)

The paper introduces three specific ways to measure how "organized" the AI's reaction is. They use fancy math names, but think of them as three different rulers:

Ruler 1: The "Location" Ruler (QBM)
- Analogy: Imagine a line of people ranked by height. If you change the important people, does the whole line shift up or down in a smooth, predictable way?
- What it checks: Did the average prediction move in a logical direction?
Ruler 2: The "Order" Ruler (WCM)
- Analogy: Imagine a race. If you change the shoes of the runners, do the finish times shuffle in a way that makes sense? Or does the winner suddenly become the loser for no reason?
- What it checks: Did the AI keep the ranking of its predictions consistent? (e.g., If Drug A was better than Drug B before, is it still better after the change?)
Ruler 3: The "Shape" Ruler (TI-WCM)
- Analogy: Imagine a balloon. If you squeeze it, does it just get smaller (shift), or does it change its weird, lumpy shape?
- What it checks: Did the pattern of the answers change, or just the numbers? This ruler ignores simple shifts to see if the underlying "shape" of the data stayed coherent.

4. The Experiment: Testing Three "Chefs"

The authors tested this on three different AI models designed to predict how drugs interact with proteins (specifically kinases).

The Setup: They used a standard dataset (Davis) and the "binding pocket" map (KLIFS) as their guide.
The Surprise: All three models had similar "taste scores" (accuracy) on the test. They all seemed to be good chefs.
The Audit Result:
- Model A & B (DeepDTA, DeepConvDTI): When the scientists poked the "important" parts, these models reacted almost the same as when they poked the "unimportant" parts. They were not organized. They were likely just guessing or relying on shortcuts.
- Model C (TAPB): This model reacted differently! When the "important" parts were changed, its answers shifted in a very organized, logical way. When the "unimportant" parts were changed, the answers were messy.
- Conclusion: Even though all three models got similar grades on the final exam, only TAPB actually seemed to understand the structure of the problem.

5. Why This Matters

The paper concludes that accuracy is not enough. You can have a model that gets 90% right but is scientifically useless because it's "cheating."

I-SAFE is like a lie detector test for AI. It doesn't care if the model is right or wrong on a single guess; it cares if the model's brain is wired correctly. It checks if the model's logic holds up when you poke it in the places that science says matter.

In short:

Old Way: "Did you get the right answer?" (Yes/No)
I-SAFE Way: "If I change this specific part of the problem, does your answer change in a way that makes scientific sense?" (Coherent/Incoherent)

This framework allows scientists to audit "black box" AI models without needing to see their internal code, ensuring that the AI is actually learning the science and not just memorizing the test.

Technical Summary: I-SAFE – Interventional Secure, Accurate, Fair and Explainable

1. Problem Statement

In scientific machine learning, deep learning models are frequently evaluated based on held-out predictive performance (e.g., accuracy, AUROC). However, high benchmark performance is a fragile indicator of scientifically meaningful behavior. Models may achieve strong predictive scores by exploiting "shortcut" features, dataset-specific regularities, or distributional biases (such as target bias or scaffold effects) rather than learning the underlying structural properties of the domain.

Standard evaluation methods fail to distinguish between a model that has learned the relevant scientific structure and one that has merely memorized statistical correlations. Furthermore, existing post-hoc auditing methods often rely on scalar summaries (e.g., average sensitivity or single-point attribution), which may miss how the full output distribution reorganizes when scientifically relevant input structures are perturbed. There is a need for a framework that moves from observational evaluation to interventional evaluation, assessing not just if a model predicts correctly, but how its predictions change under controlled, structurally guided perturbations.

2. Methodology: The I-SAFE Framework

The authors introduce I-SAFE (Interventional Secure, Accurate, Fair and Explainable), a post-hoc, model-agnostic auditing framework for black-box scientific predictors. The framework evaluates the coherence of a model's output distribution under interventions guided by an external structural prior.

Core Components:

Black-Box Predictor ( $f_{BB}$ ): A trained model accessed only via its input-output map.
Structural Prior ( $P$ ): An external map derived from domain knowledge (independent of the model) that identifies which input components are scientifically relevant for a given task. For example, in drug-target interaction (DTI), this prior identifies specific amino acid residues in a protein binding pocket.
Intervention Design: The framework constructs matched perturbation pairs:
- Mechanistic Perturbations ( $\phi_{mech}$ ): Modifications applied to components selected by the structural prior ( $P(x)$ ).
- Spurious Perturbations ( $\phi_{spur}$ ): Modifications applied to components outside the prior ( $S_{spur} \subset P(x)^c$ ), using the same perturbation operator and cardinality.
Interventional Response Profile: The framework compares the raw output distributions of the model before and after these perturbations on a fixed auditing set.

The Three Coherence Metrics:
I-SAFE quantifies the "ranking coherence" of the output distribution using three complementary metrics. Unlike scalar summaries, these metrics capture the geometry of the distributional shift.

Quantile-Based Metric (QBM): Measures location-level coherence. It compares the displacement of empirical quantiles (e.g., quartiles) between the original and perturbed outputs against the total mean squared error. Lower values indicate a larger shift in distributional location.
Wasserstein Coherence Metric (WCM): Measures ordinal coherence. Based on optimal transport theory, it evaluates how well the natural pairing of inputs and outputs is preserved after perturbation. It compares the displacement under the natural correspondence versus the optimal reordering (permutation) of outputs. It is scale-invariant and normalized.
Translation-Invariant WCM (TI-WCM): Measures shape coherence. This variant removes the effect of global mean shifts (translation), isolating changes in the distribution's shape (variance and higher-order moments).

Audit Logic:
The audit computes a prior-relative contrast ( $\Delta M = M_{spurious} - M_{mech}$ ) for each metric. A positive contrast indicates that perturbations of the prior-selected components induce a more coherent (i.e., more structured and predictable) reorganization of the output distribution than perturbations of non-prior components.

3. Key Contributions

Framework Introduction: The proposal of I-SAFE as a post-hoc auditing tool that shifts the evaluation object from scalar summaries to full output distributional changes under intervention.
Metric Definition: The formalization of three specific metrics (QBM, WCM, TI-WCM) that decompose distributional response into location, ordinal structure, and shape. These metrics are grounded in optimal transport theory and are designed to be model-agnostic.
Prior-Relative Contrast: The introduction of a contrastive evaluation strategy that isolates the effect of structural priors by comparing mechanistic perturbations against matched spurious controls, controlling for perturbation rules and cardinality.
Theoretical Properties: Proofs establishing that the metrics are bounded within $[0, 1]$ , scale-invariant (for WCM), and capable of distinguishing between different types of distributional reorganization.

4. Experimental Results

The framework was instantiated on the Davis kinase benchmark for Drug-Target Interaction (DTI) prediction.

Setup: Three sequence-based DTI models were audited: DeepDTA, DeepConvDTI, and TAPB. The structural prior was derived from KLIFS (Kinase–Ligand Interaction Fingerprints and Structures) binding-pocket annotations.
Predictive Regime: All three models demonstrated comparable predictive performance (AUROC) on the Davis test set, operating within a similar predictive regime.
Findings:
- TAPB: Showed significantly positive prior-relative contrasts for both QBM and WCM ( $\Delta QBM \approx +0.093$ , $\Delta WCM \approx +0.063$ ). This indicates that when KLIFS binding-pocket residues were perturbed, TAPB's output distribution reorganized in a highly coherent manner (preserving location and rank) compared to when non-pocket residues were perturbed.
- DeepDTA & DeepConvDTI: Showed negligible or negative contrasts. DeepDTA showed no prior-aligned advantage, and DeepConvDTI showed a negative ordinal contrast, suggesting its coherent responses were not specifically tied to the binding-pocket structure.
- Shape Analysis: The TI-WCM contrasts were negative or near-zero for all models, indicating that the coherent signals detected in TAPB were driven by location and ranking shifts rather than residual shape preservation.

Conclusion of Experiments: Despite similar accuracy, the models exhibited fundamentally different interventional behaviors. Only TAPB demonstrated that its predictions were organized around the scientifically relevant binding-pocket structure, a distinction invisible to standard accuracy-based evaluation.

5. Significance and Claims

The paper claims that I-SAFE provides a statistically grounded, prior-relative test for evaluating how trained scientific predictors operate under structurally meaningful perturbations.

Distinction from Causal Claims: The authors explicitly state that I-SAFE does not establish the causal validity of a model nor does it prove the model has recovered the true causal mechanism. Instead, it provides reusable metrics to evaluate whether a model's output distribution is coherently organized with respect to an external, domain-specific prior.
Beyond Scalar Auditing: The framework addresses the limitation of scalar auditing by capturing the geometry of distributional shifts (location, order, shape) that are invisible to aggregate performance metrics.
Model-Agnostic Utility: The method is applicable to any domain where inputs admit a structured decomposition and an external structural prior is available, making it a general tool for the structural auditing of black-box scientific AI.
Modest Scope: The authors emphasize that the audit is a diagnostic tool for existing models, not a training procedure. It helps identify whether a model's success is driven by relevant structural features or spurious correlations, supporting more precise evaluative claims in scientific AI.