🤖 AI

ConceptSMILE: Auditing the Trustworthiness of Concept-Based Explainable AI

The paper introduces ConceptSMILE, a model-agnostic auditing framework that evaluates the trustworthiness of concept-based explainable AI by perturbing inputs to measure concept-response shifts and fit surrogate models, demonstrating its effectiveness in comparing the reliability of visual and semantic concept pathways on retinal fundus images.

Original authors: Mohadeseh Mollapour, Koorosh Aslansefat, Zeinab Dehghani, Bhupesh Kumar Mishra, Tejal Shah, Zhibao Mian

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohadeseh Mollapour, Koorosh Aslansefat, Zeinab Dehghani, Bhupesh Kumar Mishra, Tejal Shah, Zhibao Mian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a super-smart robot doctor that can look at pictures of eyes and tell you if something is wrong. You ask it, "Why do you think this eye is sick?" and it replies, "Because I see a lesion, a blood vessel, and the optic disc." Sounds great, right? It's using human words! But here's the twist: just because the robot says it's looking at a lesion doesn't mean it actually is looking at a lesion. It might be cheating, looking at a weird smudge on the camera lens or a date stamp on the photo instead.

This is exactly what the paper ConceptSMILE investigates. The authors are like "trust detectives" for AI. They introduce a new framework called ConceptSMILE (Concept-level Statistical Model-agnostic Interpretability with Local Explanations) to audit whether these "human-word" explanations are actually trustworthy or just fancy lies.

The Big Idea: Don't Trust the Words, Test the Reaction

The paper argues against a common assumption: that if an AI explanation uses words we understand (like "lesion" or "blood vessel"), it is automatically reliable. The authors say, "Nope!" A model can give a perfect-sounding answer while relying on hidden tricks, like noticing a specific logo on a photo or a weird artifact from how the picture was taken.

To catch these tricks, ConceptSMILE plays a game of "What If?"

The Setup: They take an eye photo and ask the AI, "What do you see?"
The Twist: They secretly mess with the photo. They cover up parts of the image (like putting a black patch over a blood vessel) or add fake stuff (like a date stamp or a logo).
The Test: They ask the AI again. "Okay, now that I covered up the blood vessel, do you still think it's there?"
- If the AI says, "Oh, you covered it up? Then I don't see it anymore," that's good. It means the AI was actually looking at the vessel.
- If the AI says, "I still see the vessel!" even though it's covered up, that's bad. It means the AI was guessing or looking at something else entirely.

The Two Contenders: The "Map Maker" vs. The "Storyteller"

To test their idea, the authors set up a showdown between two different ways of getting explanations from eye photos:

The Map Maker (MedSAM): This AI draws a map. It cuts out the blood vessels and the optic disc like a puzzle. It's very good at knowing where things are.
The Storyteller (VLM): This AI looks at the photo and writes a sentence describing what it sees. It's great at using language but might be a bit fuzzy on the exact location.

The authors put both through the ConceptSMILE "torture test" using 40 retinal images from four different public datasets (HRF, APTOS, ODIR, and IDRiD).

What They Found (The Scoreboard)

The results were a mix of "Wow, that's cool" and "Whoa, be careful."

The Map Maker (MedSAM) Wins on Precision: When it came to pinpointing exactly where things were, the Map Maker was the champion. It achieved a very high score for how well its internal logic matched the changes in the photo. Specifically, for the "blood vessel" and "optic disc" concepts, it had a surrogate fidelity (a measure of how well a simple model could predict its behavior) of R² = 0.8503 and R²w = 0.8465. This suggests that when the Map Maker says it sees a vessel, it's usually looking right at the vessel.
The Storyteller (VLM) Wins on "Faithfulness" for Vessels: Surprisingly, the Storyteller was actually better at being "faithful" when it came to blood vessels. This means when the researchers messed with the vessel part of the image, the Storyteller's answer changed in a very consistent, logical way across all four datasets. Its correlation scores (a measure of faithfulness) ranged from r = 0.5794 to r = 0.7133, which were statistically significant.
The "Stability" Surprise: The authors tested what happens if you add a fake date or a hospital logo to the photo.
- The Map Maker was super stable when looking at the optic disc (the center of the eye). Even with a fake logo, it still saw the disc correctly 98% of the time on some datasets.
- The Storyteller was surprisingly stable when talking about lesions (spots on the eye). It kept its cool even with fake dates added, scoring 1.00 (perfect stability) on some datasets.

The Catch: It's Not a Magic Bullet

The paper is very careful not to say this is a solved problem. They explicitly state that no single method was the winner in every category.

Sometimes the Map Maker was great at finding vessels but shaky on lesions.
Sometimes the Storyteller was great at being consistent but bad at pinpointing locations.
The "trustworthiness" of an explanation depends entirely on which concept you are talking about and which dataset you are using.

The authors also note that their test was a "proof-of-concept" using a limited set of images (10 from each of 4 datasets). They didn't test this on thousands of patients or in a real hospital yet. They also admit that their method of "covering up" parts of the image (masking) is a bit artificial and might not perfectly mimic real-world eye diseases.

The Bottom Line

ConceptSMILE doesn't tell us which AI is "the best." Instead, it gives us a report card that shows exactly where each AI is strong and where it might be bluffing.

The main takeaway is simple: Just because an AI speaks our language doesn't mean it understands us. You have to poke it, prod it, and see how it reacts before you trust its diagnosis. The authors suggest that in high-stakes fields like medicine, we need this kind of "audit layer" to make sure the AI isn't just making things up, even if the things it makes up sound very convincing.

Technical Summary: ConceptSMILE – Auditing the Trustworthiness of Concept-Based Explainable AI

Problem Statement
While Concept-Based Explainable AI (C-XAI) aims to make model reasoning more human-understandable by expressing predictions through high-level concepts (e.g., clinical biomarkers, objects, or semantic attributes), the presence of such concepts does not automatically guarantee trustworthy explanations. Models may produce concept-level outputs that appear plausible while relying on spurious correlations, unstable features, label leakage, or non-causal shortcuts. In high-stakes domains like medical imaging, a concept explanation can be human-readable yet unfaithful to the model's actual behavior. Existing evaluation practices often lack a unified framework to independently audit whether concept-based explanations remain accurate, faithful, stable, and consistent under controlled perturbations.

Methodology: The ConceptSMILE Framework
The paper introduces ConceptSMILE (Concept-level Statistical Model-agnostic Interpretability with Local Explanations), a model-agnostic, perturbation-based auditing framework designed to evaluate the reliability of concept-based explanations. Rather than replacing existing explanation methods, ConceptSMILE extends the perturbation-based logic of SMILE (Statistical Model-agnostic Interpretability with Local Explanations) from feature or region-level attribution to the auditing of human-understandable concept explanations.

The framework operates through the following pipeline:

Concept Extraction: It extracts concept-level outputs from a target model or explanation pathway. In the study, two distinct pathways were audited:
- MedSAM-based Visual Pathway: Uses the Medical Segment Anything Model to generate segmentation-guided visual concepts (lesions, blood vessels, optic disc).
- VLM-based Semantic Pathway: Uses a Vision-Language Model (Qwen2.5-VL) with structured prompts to generate semantic concept responses for the same retinal concepts.
Controlled Perturbation: The input image is divided into superpixels. Controlled perturbations are generated by masking specific superpixel regions (setting them to black), creating a local neighborhood of perturbed samples.
Response Shift Measurement: The framework measures the shift in concept responses ( $\Delta y$ ) between the original input and perturbed inputs.
Locality Weighting: Perturbed samples are assigned locality weights based on their distance (Cosine or Wasserstein) from the original input in an embedding space (using DINOv2). Closer perturbations receive higher weights.
Surrogate Modeling: A local XGBoost surrogate model is fitted to approximate the relationship between perturbation patterns and concept-response shifts. This surrogate captures non-linear relationships between input modifications and concept confidence.
Multi-Dimensional Reliability Evaluation: The framework assesses reliability across five dimensions:
- Attribution Accuracy: Whether identified important regions align with clinically meaningful evidence (measured via ACC, F1, AUROC).
- Fidelity: How well the surrogate model approximates the observed concept-response shifts (measured via WMSE, WMAE, $R^2$ , $R^2_w$ ).
- Faithfulness: Whether concept responses change significantly when concept-relevant regions are perturbed (measured via Pearson correlation).
- Stability: Robustness to non-clinical artefacts (e.g., added dates or logos) measured via Jaccard index overlap.
- Consistency: Reproducibility of explanations across repeated runs under identical settings (measured via variance and standard deviation).

Key Contributions

Concept-Level Auditing: Introduction of ConceptSMILE as a perturbation-based framework specifically designed to audit the trustworthiness of concept-based explanations, moving beyond visual plausibility to measurable reliability.
Extension of SMILE: Reformulation of the SMILE logic from superpixel-level attribution to concept-level reliability testing using controlled perturbations, locality weighting, and surrogate modeling.
Model-Agnostic Validation: Demonstration of the framework on retinal fundus images using two distinct concept-extraction pathways (MedSAM and VLM), providing a comparative analysis of visual versus semantic concept reliability.

Experimental Results
The framework was evaluated on retinal fundus images from four public datasets (HRF, APTOS, ODIR, IDRiD) focusing on three concepts: lesions, blood vessels, and the optic disc.

Surrogate Fidelity: The MedSAM pathway achieved significantly stronger surrogate fidelity ( $R^2 = 0.8503$ , $R^2_w = 0.8465$ with Wasserstein-weighted XGBoost) compared to the VLM pathway. This suggests that segmentation-guided visual concepts provide more locally predictable responses under superpixel perturbation.
Attribution Accuracy: MedSAM demonstrated stronger spatial attribution accuracy for well-defined structures like blood vessels and the optic disc. VLM-based semantic explanations were more variable and less spatially precise.
Faithfulness: The VLM pathway showed stronger and more statistically significant faithfulness for blood vessel concepts across all datasets ( $r \approx 0.58-0.71$ ), whereas MedSAM showed stronger faithfulness for lesion concepts in several datasets. This indicates that reliability is concept-dependent rather than pathway-dependent.
Stability: Stability varied by concept and artefact type. MedSAM optic-disc explanations were highly stable against date-added artefacts ( $J_D \approx 0.96-0.98$ ), while VLM lesion explanations showed high stability under both date and logo artefacts.
Consistency: Reproducibility was not uniform across pathways. MedSAM optic-disc explanations were highly consistent on HRF and ODIR, while VLM lesion explanations were most consistent on ODIR and IDRiD.
Robustness: Under acquisition artefacts (contrast variation and simulated blink occlusion), blood vessel and lesion explanations generally retained positive surrogate fidelity, whereas optic-disc explanations showed higher variability and occasional negative test $R^2$ values.

Significance and Claims
The paper claims that ConceptSMILE provides an independent audit layer for evaluating the trustworthiness of concept-based XAI. Its primary significance lies in demonstrating that:

Interpretability $\neq$ Trustworthiness: Human-understandable concepts are not inherently reliable; they must be audited for accuracy, faithfulness, stability, and consistency.
Reliability is Context-Dependent: No single pathway or concept is universally superior. MedSAM excels in spatial attribution and surrogate fidelity for structured anatomy, while VLMs can exhibit strong faithfulness for specific semantic concepts (like vessels) and stability against certain artefacts.
Audit Necessity: In high-stakes settings, concept-based explanations should be treated as testable claims. ConceptSMILE offers a protocol to identify fragile, poorly grounded, or inconsistent explanations before they are relied upon in critical decision-making.

The authors explicitly state that ConceptSMILE does not provide clinical certification or causal proof, but rather quantitative audit evidence to help researchers and clinicians distinguish between apparent interpretability and grounded reliability.