🤖 AI

A Formal Methodological Framework for Auditing Robustness and Fidelity in Explainable AI: From Application to Trust Certification

This paper proposes a formal auditing framework that quantifies the robustness and fidelity of explainable AI methods to generate a Trust Score, demonstrating through a Madagascar malnutrition case study that high-performing models can yield uninformative explanations and that such auditing is essential for trustworthy decision-making in sensitive domains.

Original authors: Rosa Elysabeth Ralinirina, Jean Christian Ralaivao, Niaiko Michaël Ralaivao, Alain Josué Ratovondrahona, Thomas Mahatody

Published 2026-08-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rosa Elysabeth Ralinirina, Jean Christian Ralaivao, Niaiko Michaël Ralaivao, Alain Josué Ratovondrahona, Thomas Mahatody

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world, artificial intelligence has become a powerful tool for making sense of complex data, from predicting weather patterns to diagnosing diseases. However, many of the most powerful systems operate as "black boxes," meaning they can produce a result without revealing the steps they took to get there. To bridge this gap, scientists have developed tools that act like translators, pointing out which pieces of information a computer model relied on to make its decision. These tools are meant to build trust, allowing human experts to understand and verify the machine's logic. Yet, a critical question remains: if the computer's reasoning changes drastically when the input data is tweaked by just a tiny amount, can we really trust the explanation? This uncertainty is the central challenge addressed by a new study focusing on how to verify the reliability of these AI explanations.

Researchers from the University of Fianarantsoa in Madagascar set out to solve this problem by creating a formal method to audit these explanations. They focused on two specific qualities: stability and faithfulness. Stability asks whether the explanation remains consistent when the data is slightly disturbed, much like checking if a compass still points north when the ground beneath it shakes. Faithfulness asks whether the features the explanation highlights are actually the ones driving the model's decision, ensuring the tool is not just inventing reasons that sound plausible but are actually false. By combining measurements of these two qualities into a single score, the team created a way to rate how much trust a user should place in a specific AI explanation.

To test this new auditing system, the researchers applied it to a real-world dataset concerning food security in Madagascar. The data covered twenty-three regions over thirteen years, tracking eighty-three different factors such as rainfall, rice production, market prices, and child health rates. The goal was to predict the severity of malnutrition in these regions, which was categorized into four levels ranging from acceptable to critical. The team trained three different types of computer models on this data and then used two popular explanation tools to see how well they could explain the models' predictions. They ran their audit on a specific set of test cases to see if the explanations held up under scrutiny.

The results revealed a sobering reality: a model that is extremely accurate at making predictions is not necessarily trustworthy when it comes to explaining those predictions. Some of the models achieved near-perfect scores in predicting the malnutrition classes, yet the explanations they produced were either numerically broken or completely uninformative. In one striking case, a model that was highly confident in its answers caused one of the explanation tools to output a list of zero values for every single factor, effectively saying that nothing mattered because the answer was already obvious. When the researchers tried to measure how stable these explanations were, the numbers became impossible to calculate without a small mathematical fix, highlighting a fundamental flaw in how the explanation tool interacted with the overconfident model.

The study also found that the quality of an explanation depends heavily on how the model was built. When the researchers adjusted the models to be less prone to memorizing the specific data points—a process known as regularization—the explanations became much more reliable. For instance, one type of model that initially produced flat, uninformative explanations began to show a clear pattern where removing the most important factors actually changed the prediction significantly. This shift demonstrated that the explanation was finally reflecting the true inner workings of the model rather than just echoing its confidence. The researchers observed that the explanation tool based on tree-based models was generally more stable than the other, but both tools struggled when the underlying model was too confident in its own answers.

Ultimately, the research suggests that high accuracy in a machine learning model does not guarantee that its reasoning is sound or understandable. The team demonstrated that without a specific check for stability and faithfulness, users might be presented with explanations that look convincing but are actually fragile or misleading. By introducing a single score that combines these checks, they provided a practical way for decision-makers to know when an AI explanation is safe to use and when it should be ignored. This is particularly vital in sensitive fields like food security, where policies and aid are distributed based on these predictions. The study concludes that auditing these explanations is not an optional extra step but a necessary requirement to ensure that the artificial intelligence guiding critical human decisions is truly trustworthy.

Technical Summary: A Formal Methodological Framework for Auditing Robustness and Fidelity in Explainable AI

1. Problem Statement

The paper addresses a critical gap in the deployment of Explainable AI (XAI) within sensitive domains, specifically food security in Madagascar. While post-hoc explainers like SHAP and LIME are standard for interpreting black-box predictions, their outputs are often unreliable. The authors observed that small perturbations in input data can cause substantial variations in explanation outputs, and there is no inherent mechanism to verify if the highlighted features genuinely drive the model's predictions. This "illusion of transparency" poses a risk when policymakers rely on these explanations for decision-making. The core problem is the lack of a unified, quantitative protocol to audit both the stability (robustness) and faithfulness (fidelity) of XAI outputs simultaneously.

2. Methodological Framework

The authors propose a formal auditing pipeline that quantifies two distinct properties of any post-hoc explainer and synthesizes them into a single Trust Score.

2.1 Robustness (Local Stability)

Robustness measures the stability of an explanation under input perturbation.

Mechanism: For a given input $x$ , the auditor draws $N$ noise vectors $\delta$ from a Gaussian distribution $N(0, \sigma^2 I)$ .
Calculation: It computes the explanation vectors $E(x)$ and $E(x+\delta)$ , normalizes them to sum to 1, and measures their divergence using the Jensen-Shannon Divergence (JSD).
Metric: $R(x) = 1 - \frac{1}{N} \sum D_{JS}(\tilde{E}(x) \parallel \tilde{E}(x+\delta_i))$ .
Handling Edge Cases: A specific numerical issue is addressed where TreeSHAP on overfitted models may output zero importance for all features, rendering JSD undefined. The authors resolve this by adding a small epsilon ( $\epsilon = 10^{-10}$ ) to each component before normalization.

2.2 Fidelity (Faithfulness)

Fidelity assesses whether the features deemed important by the explainer actually influence the model's prediction.

Mechanism: The method employs feature ablation. The top- $k$ features identified by the explainer are masked (replaced with feature means) to create a perturbed input $x \setminus F_k$ .
Calculation: The change in the model's prediction $f(x)$ is measured relative to the original prediction.
Metric: $F_k(x) = 1 - \frac{|f(x) - f(x \setminus F_k)|}{|f(x)| + \epsilon}$ . The overall fidelity $F(x)$ is the mean across a set of ablation levels $K$ .
Interpretation: A high fidelity score indicates that removing the top features significantly alters the prediction, confirming the explainer's relevance.

2.3 Trust Score

The framework combines these metrics into a weighted sum:
$T(x) = \alpha R(x) + \beta F(x)$
where $\alpha + \beta = 1$ . By default, $\alpha = \beta = 0.5$ , giving equal weight to stability and faithfulness, though the authors note these weights can be adjusted based on domain requirements (e.g., prioritizing stability for non-technical stakeholders).

3. Experimental Setup and Case Study

The framework was validated on a multi-sectoral dataset regarding food security in Madagascar.

Data: 253 records covering 23 regions over 13 years (2010–2023), featuring 83 variables (climate, agriculture, socio-economics, nutrition, demographics). The target variable has 4 ordinal classes (Acceptable, Precarious, Alarming, Critical), with severe class imbalance (only 3 instances in the "Acceptable" class).
Models: Three classifiers were tested: Random Forest (RF), XGBoost, and a Neural Network (NN). Regularized variants of each were also trained to assess the impact of overfitting.
Explainability: SHAP (TreeSHAP for trees, KernelSHAP for NN) and LIME were used.
Protocol: The audit was run on 30 test instances with 10 perturbation trials ( $\sigma=0.1$ ) and ablation levels $K = \{3, 5, 10, 41, 82\}$ .

4. Key Results

The experiments yielded sobering findings regarding the relationship between predictive performance and explanation quality:

High AUC $\neq$ Trustworthy Explanations: Models achieved macro-average AUC scores above 0.99. However, this high predictive performance did not guarantee stable or informative explanations.
Numerical Degeneracy: In the XGBoost model, TreeSHAP produced zero-valued importance vectors for some instances due to near-certain predictions, causing the JSD calculation to fail without the proposed $\epsilon$ -smoothing fix.
Fidelity Collapse in Overfitted Models: The XGBoost model exhibited "flat" fidelity scores (approx. 0.620) across all ablation levels for both SHAP and LIME. Because the model was so confident (predicting near 0 or 1), removing features shifted the output by a constant amount regardless of which features were removed. This rendered the fidelity metric uninformative and unable to distinguish between good and bad explanations.
Impact of Regularization: Regularized models generally improved Trust Scores.
- RF and XGBoost: Regularization slightly improved robustness and modestly improved fidelity, though XGBoost's fidelity curves remained relatively flat.
- Neural Network: The regularized NN (NNreg) showed the most significant recovery in discriminative power. Its LIME fidelity dropped from 0.977 (top 3 features) to 0.802 (top 82 features), indicating that the explainer correctly identified the hierarchy of feature importance.
Robustness Comparison: SHAP consistently outperformed LIME in robustness on tree-based models (e.g., XGBoost: 0.978 vs. 0.658), attributed to TreeSHAP's exact computation path versus LIME's sampling variability.

5. Significance and Claims

The paper claims that auditing XAI outputs is not optional but necessary, particularly in sensitive domains like food security. The primary contributions are:

Unified Protocol: The framework provides a single pipeline to measure robustness and fidelity side-by-side, producing a comparable Trust Score across different model-explainer pairs.
Detection of "Uninformative" Explanations: The study demonstrates that the framework can detect when explanations become numerically degenerate or vacuous due to model overconfidence or overfitting. Specifically, it highlights that a model with near-perfect AUC can produce explanations that are either unstable or entirely uninformative.
The Necessity of Calibration: The results suggest that before trusting XAI outputs, practitioners must ensure the underlying model is well-calibrated and not severely overfitted. If a model is overconfident, fidelity metrics lose their discriminative power.
Practical Trade-offs: The Trust Score makes the trade-off between stability and faithfulness explicit, allowing domain experts to adjust weights ( $\alpha, \beta$ ) based on whether they prioritize consistent explanations for communication or strict adherence to model logic for regulatory compliance.

The authors conclude that while the framework is informative for well-calibrated models, its application to overfitted models requires caution, and future work should focus on domain-specific perturbation strategies and temporal data structures.