🤖 machine learning

When Confidence Lacks Concepts: Interpretable OOD Detection via Representation Perturbations

This paper proposes an interpretable Out-of-Distribution (OOD) detection framework for medical imaging that enhances safety by leveraging sparse autoencoders to extract class-specific concept vectors, using their perturbation-induced stability to distinguish in-distribution samples from OOD anomalies based on representational alignment.

Original authors: Anju Chhetri, Pratik Shrestha, Ramesh Rana, Prashnna Gyawali, Binod Bhattarai

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anju Chhetri, Pratik Shrestha, Ramesh Rana, Prashnna Gyawali, Binod Bhattarai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The Overconfident Robot Doctor

Imagine a highly trained robot doctor that is amazing at diagnosing diseases from medical images (like X-rays or endoscopy photos). It has studied thousands of pictures of healthy stomachs and specific types of cancer.

However, this robot has a dangerous flaw: it overgeneralizes. If you show it a picture of a completely different organ, or a weird artifact, or a disease it has never seen, it might still confidently say, "I know exactly what this is!" and give a diagnosis. In the real world, this is risky because the robot might be confidently wrong.

Current methods try to catch these mistakes by looking at the robot's "internal math" (like how sure it feels). But the problem is, these internal signals are like a black box. We know the robot is unsure, but we don't know why. It's like a car dashboard warning light that says "Engine Trouble" but gives no details. In healthcare, we need to know why the robot is unsure to trust it.

The Solution: The "Concept Perturbation" Test

The authors of this paper propose a new way to test the robot, which they call CAPS (Class-conditioned Activation Perturbations). Instead of just looking at the final answer, they poke the robot's brain to see how stable its thinking is.

Here is how it works, step-by-step:

1. Learning the "Vocabulary" of Disease (The SAE)

First, the researchers teach the robot a new way to think. They use a tool called a Sparse Autoencoder (SAE).

The Analogy: Imagine the robot usually thinks in a giant, messy soup of words. The SAE acts like a translator that breaks that soup down into distinct, single ingredients (concepts).
The Result: For a specific disease (e.g., "Pylorus"), the robot learns to identify specific "concept vectors." These are like mental blueprints for what a healthy Pylorus looks like (e.g., "round shape," "pink color," "specific opening").

2. The "What If?" Test (The Perturbation)

When the robot looks at a new patient image, it makes a guess. Let's say it guesses "Pylorus."

The Test: The researchers take the specific "Pylorus blueprint" (the concept vector) and gently nudge the robot's internal representation of the image in that direction. It's like asking the robot: "If I make this image look slightly more like a perfect Pylorus, does your confidence change?"

3. The Stability Check (ID vs. OOD)

This is where the magic happens. The researchers measure how much the robot's confidence changes after that nudge.

Scenario A: The Real Patient (In-Distribution)
- The Situation: The image is actually a real Pylorus.
- The Reaction: The robot's brain is already aligned with the "Pylorus blueprint." When you nudge it toward that blueprint, it barely flinches. Its confidence stays steady.
- The Verdict: "This is a safe, known sample."
Scenario B: The Fake/Unknown Patient (Out-of-Distribution)
- The Situation: The image is actually a weird artifact or a different organ, but the robot guessed "Pylorus" anyway.
- The Reaction: The robot's brain is not actually aligned with the "Pylorus blueprint." When you nudge it toward that blueprint, the robot gets confused. Its confidence swings wildly because the image doesn't actually fit the concept.
- The Verdict: "This is suspicious! The robot is guessing without a real basis. Flag this as Out-of-Distribution (OOD)."

Why This Matters: From Black Box to Glass Box

The paper claims this method is special because it is interpretable.

Old Way: "The robot is unsure." (We don't know why).
New Way: "The robot is unsure because the image lacks the specific visual features (concepts) that define a Pylorus, even though the robot guessed Pylorus."

The researchers tested this on three types of medical imaging:

Endoscopy (looking inside the stomach/intestines).
Histopathology (looking at tissue samples under a microscope).
Ophthalmology (looking at retinal scans of the eye).

What the Experts Said

To prove their "concepts" were real and not just math gibberish, they showed the results to a human doctor (a gastroenterologist).

The Findings: The doctor confirmed that when the robot was "stable," the images showed clear, real medical landmarks (like the opening of the appendix or the Z-line in the esophagus).
The Failure: When the robot was "unstable" (the OOD samples), the doctor confirmed those images were missing those landmarks or showed things that didn't belong (like a polyp in the wrong place or a software overlay).

Summary

The paper introduces a method to catch AI mistakes by checking if the AI's confidence is "grounded" in real, understandable features.

If the AI is confident and the image matches the "concept" of that confidence, it's likely safe.
If the AI is confident but the image doesn't match the "concept" (causing a big reaction when tested), it's likely a mistake.

This makes AI safer for high-stakes fields like medicine, not just by catching errors, but by explaining why an error might be happening.

Technical Summary: When Confidence Lacks Concepts: Interpretable OOD Detection via Representation Perturbations

Problem Statement
Deep neural networks, particularly in medical imaging, exhibit a tendency to overgeneralize when faced with distributional shifts (e.g., variations in imaging protocols, patient demographics, or novel diseases). While Out-of-Distribution (OOD) detection methods exist to mitigate the risks of unreliable predictions, most rely on opaque internal signals (such as softmax probabilities, energy scores, or feature distances) with poorly understood semantic meaning. This lack of interpretability limits trust in safety-critical clinical settings, where understanding why a sample is flagged as anomalous is as crucial as the detection itself. The paper addresses the need for an OOD detection framework that is both discriminative and interpretable, linking detection decisions to specific, clinically meaningful concepts.

Methodology: Class-conditioned Activation Perturbations (CAPS)
The proposed framework, CAPS, operates by probing the stability of model predictions under class-conditioned semantic perturbations within the representation space. The methodology proceeds in three stages:

Sparse Autoencoder (SAE) Training:
The authors employ Sparse Autoencoders to decompose dense intermediate activations (specifically the [CLS] token from a Vision Transformer) into sparse, semantically meaningful components. The SAE learns a sparse latent representation $s(h)$ and a decoder $D$ such that the original activation $h$ is approximated by a linear combination of decoder columns (features). The training objective balances reconstruction fidelity with sparsity ( $L = \|h - \bar{h}\|^2_2 + \gamma\|s(h)\|_1$ ).
Class-Specific Concept Extraction:
For each class $c$ , the authors identify a subset of SAE latents that are characteristically active by computing the mean latent activation over in-distribution (ID) training samples. The top- $k$ latent dimensions with the highest mean activation are selected to form a class-specific concept dictionary $K_c$ . These decoder columns represent the linear directions encoding class-defining features.
Perturbation and Stability Analysis:
At inference, for a test input $\bar{x}$ , the model first predicts a class $\hat{y}$ . The corresponding concept vector $v_{\hat{y}}$ is constructed by aggregating and normalizing the selected concept vectors for $\hat{y}$ . This vector is then injected into the model's intermediate activation at layer $l$ via a controlled perturbation: $\tilde{h}_l = h_l + \beta v_{\hat{y}}$ , where $\beta$ controls the perturbation strength.

The core hypothesis is that ID samples, which align with class-specific semantic directions, will exhibit low sensitivity to these perturbations (resulting in small changes to the class logits). Conversely, OOD samples, which lack alignment with these learned directions, will show amplified deviations in logits. The OOD detection score is defined as the magnitude of the logit shift $\Delta \ell_c$ . Samples exceeding a threshold $\tau$ (calibrated on a held-out ID set) are classified as OOD.

Key Contributions

Novel Framework: The proposal of CAPS, an OOD detection method that leverages concept-level perturbations in representation space rather than opaque output scores.
Cross-Domain Evaluation: Extensive evaluation across three distinct medical imaging modalities: endoscopy (Kvasir-v2), histopathology (NCT-CRC-HE-100K/CRC-VAL-HE-7K), and ophthalmic imaging (Retinal OCT).
Clinical Interpretability: Demonstration that the extracted SAE features correspond to clinically meaningful visual patterns (e.g., anatomical landmarks like the pyloric opening or physiological features like bile staining), validated through blinded clinical expert review.

Results
The method was evaluated against state-of-the-art baselines (including Energy, MSP, Mahalanobis, ViM, and NERO) using AUROC and FPR95 metrics:

Histopathology: CAPS achieved the best performance with 95.28% AUROC and 23.17% FPR95.
OCT: CAPS achieved 93.67% AUROC and 24.80% FPR95, outperforming all baselines.
Kvasir-v2 (Endoscopy): The method achieved 79.58% AUROC. The authors attribute the moderate performance here to the smaller dataset size (2,400 samples), which limits the SAE's ability to learn highly discriminative, class-specific directions compared to the larger histopathology and OCT datasets (>50k samples).
Robustness: Sensitivity analysis showed the method is stable across a wide range of hyperparameters ( $k$ and $\beta$ ) for datasets with sufficient data, though instability was observed in the low-data Kvasir-v2 regime.

Significance and Claims
The paper claims that CAPS provides a "discriminative OOD signal and an interpretable lens into the internal mechanisms driving model uncertainty." By framing OOD detection as a concept-conditioned stability analysis, the approach bridges the gap between high-performance detection and clinical trust. The authors demonstrate that OOD samples are detected not merely because they are "different," but because they lack the specific, expert-verified semantic features that drive confident predictions for in-distribution classes.

The authors maintain a modest tone regarding limitations, acknowledging that in low-data regimes, the difficulty of capturing semantically coherent directions may impact reliability. They posit that while modality-level pretraining helps, developing more robust SAE training strategies for data-constrained healthcare environments remains a necessary direction for future work. The work is presented as a step toward unifying representation-level interpretability with reliability assessment in deep neural networks.