🤖 machine learning

Subgroup Performance Analysis in Hidden Stratifications

This paper demonstrates that applying subgroup discovery to chest x-ray and skin lesion classification can effectively uncover hidden performance disparities and expose larger performance gaps than traditional metadata-based analysis, offering a novel evaluation framework for ensuring trustworthy AI in medicine.

Original authors: Alceu Bissoto, Trung-Dung Hoang, Tim Flühmann, Susu Sun, Christian F. Baumgartner, Lisa M. Koch

Published 2026-09-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alceu Bissoto, Trung-Dung Hoang, Tim Flühmann, Susu Sun, Christian F. Baumgartner, Lisa M. Koch

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving field of medical artificial intelligence, computers are increasingly tasked with reading X-rays and analyzing skin lesions to help doctors diagnose diseases. These systems learn by studying thousands of images, eventually becoming skilled at spotting patterns that indicate illness. However, a troubling reality has emerged: these models do not always perform equally well for every patient. While a computer might correctly identify a heart condition in one group of people, it could fail repeatedly for another. This inconsistency often stems from "hidden stratifications," a term describing subtle, unrecorded differences in the data that the computer learns to rely on instead of the actual disease. For instance, a model might learn to associate a specific type of hospital tag or a certain lighting condition with a positive diagnosis, rather than learning the medical signs themselves. When the model encounters a patient without that tag or with different lighting, it fails. Because these hidden factors are not recorded in patient records, traditional methods of checking for fairness often miss them entirely, leaving dangerous blind spots in the care these tools provide.

Researchers Alceu Bissoto and his colleagues at the University of Bern and the University of Tübingen set out to solve this problem by developing a new way to find these invisible groups. Instead of waiting for doctors to tell them which patients belong to which group based on known facts like age or sex, they taught a computer to find its own groups based on the visual patterns it sees in the images. The team tested this approach on two large medical datasets: one containing chest X-rays and another with images of skin lesions. They first created a controlled simulation where they knew exactly which hidden factors were causing the computer to fail, confirming that their new method could spot these issues when standard checks could not. Then, they applied the method to real-world data where the true causes of failure were unknown.

The results were striking. In their simulations, the new method successfully identified groups of images where the computer's accuracy dropped significantly, revealing performance gaps that traditional checks completely missed. When the researchers moved to real-world data, the difference was even more pronounced. On the chest X-ray dataset, the new approach uncovered subgroups of patients where the computer's accuracy fell below 60 percent, while the majority of other groups performed at around 90 percent. In contrast, the standard method, which relied only on recorded metadata like patient sex or age, failed to identify these struggling groups entirely, presenting a falsely optimistic view of the system's reliability. Similarly, in the skin lesion analysis, the method found a specific group of images where the computer was correct only 5 percent of the time, a critical failure that would have remained hidden under traditional scrutiny.

Crucially, the study showed that these newly discovered groups were not simply re-labeling patients by age or gender. In fact, the groups the computer found often had nothing to do with the demographic information available in the hospital records. Instead, they aligned with visual features, such as the specific color or size of a skin lesion, or subtle artifacts in an X-ray image that human observers might overlook. This suggests that the computer was indeed finding the real reasons for its confusion, which were buried deep within the visual details of the images rather than the patient's personal history. The researchers also found that they did not need specialized medical training for the computer to find these patterns; tools trained on general, everyday photographs were just as effective at exposing these hidden problems as those trained specifically on medical data.

The authors conclude that this technique offers a vital new layer of safety for medical AI. By allowing computers to discover their own subgroups based on what they actually see, hospitals and developers can identify and fix performance gaps before a system is widely deployed. This approach does not replace the need to check for known biases like age or sex, but it acts as a powerful additional safeguard, ensuring that the technology works reliably for every patient, regardless of the hidden characteristics of their medical images.

Technical Summary: Subgroup Performance Analysis in Hidden Stratifications

Problem Statement
Machine learning (ML) models in medicine often exhibit systematic performance disparities across patient subgroups, posing risks for safe deployment, particularly in underserved populations. Traditional subgroup analysis relies on available metadata (e.g., sex, age, ethnicity) to identify these disparities. However, such metadata is often limited and fails to capture the true factors of variation driving model performance, such as image artifacts, device manufacturers, or subtle visual characteristics. Consequently, "hidden stratifications" exist—cohesive subgroups defined by unobserved features that lead to significant, unnoticed performance gaps. While subgroup discovery techniques exist to algorithmically identify such groups based on learned feature representations, they remain underexplored for performance monitoring. A primary barrier is the lack of ground truth stratification labels in real-world data, making evaluation and validation of these methods challenging.

Methodology
The authors propose applying subgroup discovery to the downstream task of performance monitoring to expose hidden stratifications. They utilize DOMINO, a subgroup discovery algorithm that clusters samples based on feature representations and model predictions without requiring classification labels or metadata.

Algorithm: The method extracts feature representations $z(x)$ from images using an external pretrained model (e.g., CLIP or BiomedCLIP) and combines them with softmax predictions $\hat{y}(x)$ from the target classification model. Samples are clustered into subgroups $S$ using a Generalized Gaussian Mixture Model (GMM) by minimizing an objective function that balances the influence of embeddings and predicted labels via a hyperparameter $\gamma$ . Notably, the authors remove classification labels from the GMM input, enabling application in post-deployment scenarios with unlabeled test sets.
Evaluation Strategy: To address the lack of ground truth, the authors propose a two-tiered evaluation approach:
1. Synthetic Setting: Clinically inspired artifacts (e.g., hospital tags, hyperintense signals) are spurious correlated with disease labels to create known ground-truth subgroups. This allows for the measurement of "performance gaps" and "average purity" (alignment with known attributes).
2. Real-World Setting: The method is applied to chest x-ray (CheXpertPlus) and skin lesion (SLICE-3D) classification tasks. Here, performance is evaluated by comparing the performance gaps and subgroup cohesion against traditional metadata-based stratifications.
Metrics: Two novel metrics are introduced:
- Performance Gap ( $\Delta(S)$ ): The difference between the maximum and minimum performance across discovered subgroups.
- Average Purity (AP): A measure of subgroup cohesion, quantifying how well discovered subgroups align with known attributes (e.g., specific artifacts or metadata).

Key Results

Superiority over Metadata: In both synthetic and real-world experiments, subgroup discovery consistently exposed larger performance disparities than traditional metadata-based analysis. In the synthetic setting, the method successfully identified performance gaps caused by hidden artifacts that were missed by analysis based on known attributes.
Real-World Findings:
- CheXpertPlus: Subgroup discovery identified subgroups with accuracy below 60%, whereas metadata-based analysis failed to reveal such low-performing groups, showing a much narrower performance range. The discovered subgroups did not align well with patient demographics (sex, age), confirming that metadata often fails to reflect the primary drivers of performance variability.
- SLICE-3D: The method identified a subgroup with only 5% accuracy (721 negatives, 17 positives). Unlike the x-ray data, these subgroups showed high purity when aligned with visual lesion features (e.g., color, area), suggesting the algorithm captured clinically relevant visual stratifications that demographic metadata missed.
Feature Extractors: The study found that feature extractors trained on natural images (CLIP) were sufficient to expose meaningful performance gaps, performing similarly to those trained on biomedical data (BiomedCLIP). This suggests that subtle visual factors of variation in medical images can be captured by general-purpose representations.

Significance and Claims
The paper provides the first compelling evidence that subgroup discovery can serve as a vital tool for comprehensive performance validation and monitoring of trustworthy AI in medicine. The authors argue that:

Hidden Stratifications are Common: Significant performance disparities often exist in hidden subgroups that traditional metadata analysis cannot detect.
Feasibility without Labels: A simplified subgroup discovery method can expose these disparities without access to classification labels or metadata, making it suitable for post-deployment monitoring.
Complementary Role: Subgroup discovery should not replace but rather accompany traditional subgroup analysis as an additional safeguard during ML validation and deployment. It offers a mechanism to identify failure modes that are invisible to standard demographic reporting.

The authors conclude that while validation remains challenging due to the absence of ground truth labels, the robustness of their results across datasets and configurations supports the adoption of subgroup discovery as an effective, easily implemented tool for enhancing the safety of medical ML systems.

Technical Summary: Subgroup Performance Analysis in Hidden Stratifications

More like this