📄 health informatics

ShortKit-ML: an open toolkit for measuring shortcut potential in medical imaging embeddings

ShortKit-ML is an open toolkit that introduces a calibrated conditional probe to measure shortcut potential in frozen medical imaging embeddings, demonstrating that while its scores correlate with the reliance of confounded classifiers on sensitive attributes like sex, the tool alone cannot confirm or eliminate such reliance without also testing the classifier head itself.

Original authors: Cajas, S., Marzullo, A., Kapadia, S., Santos, F., Ocampo Osorio, F., Kong, Q., Quarta, A., Kuo, P.-C., Patel, M., Rojas Sillery, R. I., Celi, L. A.

Published 2026-11-07 (v2)

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Cajas, S., Marzullo, A., Kapadia, S., Santos, F., Ocampo Osorio, F., Kong, Q., Quarta, A., Kuo, P.-C., Patel, M., Rojas Sillery, R. I., Celi, L. A.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Medical imaging has reached a point where artificial intelligence can often match human experts in spotting diseases on X-rays. These systems work by first converting an image into a mathematical summary, a compact list of numbers that captures the essential features of the picture. This summary is called an embedding. Once created, this summary can be fed into a simpler computer program to make a diagnosis. However, a high score does not guarantee that the system is looking at the right thing. Sometimes, these systems learn to rely on "shortcuts." A shortcut is a signal that happens to predict the correct answer in the training data but is not part of the actual disease. For example, a model might learn to predict a broken hip not by looking at the bone, but by noticing the type of scanner used or the hospital where the photo was taken. If the data changes, these shortcuts fail, and the model makes dangerous mistakes.

The core problem is that these shortcuts often hide inside the mathematical summary itself. Even before a diagnosis is made, the summary might already contain hidden clues about a patient's age, sex, or race. If a model uses these clues to make a decision, it could produce unfair or inaccurate results for certain groups of people. The question researchers have long struggled with is whether they can look at just the mathematical summary and tell if it holds these hidden clues, without needing to see the original image or the final diagnosis. If they can measure this "shortcut potential" early, they might be able to stop bad models before they are ever used on patients.

A team of researchers has built a new open-source tool called ShortKit-ML to answer this question. Their work focuses on a specific type of test: can they look at a frozen mathematical summary and determine if it contains information about a sensitive trait, like a patient's sex or race, that the disease diagnosis itself does not explain? To do this, they developed a method that acts like a very careful filter. They take the summary and the known disease label, and they ask a simple computer program to guess the patient's sensitive trait. If the program can guess the trait better than chance, even after knowing the disease, it means the summary holds extra, unwanted information. To ensure this test is not fooled by random noise, the researchers shuffle the sensitive traits around within groups of patients who have the same disease status. This creates a baseline of what a random guess looks like, allowing them to measure exactly how much real information is hiding in the summary.

The researchers put this tool to the test on two large collections of chest X-rays, one from the United States and another from a different hospital system. They examined seven different types of pre-trained mathematical summaries, which are like different "eyes" that have been trained to see medical images. They checked these summaries for clues about sex, age, and race. The results were clear and consistent: every single summary they tested contained information about these traits. For sex, the hidden information was very strong; the tool could almost perfectly identify a patient's sex just by looking at the summary, even when the disease label was known. The results for age were also strong, though slightly less so. Race was harder to detect, and the strength of the signal varied depending on the specific collection of images and the type of summary used, but the tool still found traces of it in most cases.

To understand if finding these hidden clues actually mattered, the researchers ran a second experiment. They took these same summaries and trained a new, simple computer program to make a diagnosis, but they deliberately gave it training data where the disease and the sensitive trait were mixed up. In this messy data, a patient's sex or age was strongly linked to whether they had the disease, creating a perfect trap for a shortcut. They found that the summaries which showed the strongest hidden clues in the first test were the ones that led the new program to rely most heavily on those shortcuts. When the training data was messy, the programs built on the "leaky" summaries made large errors, predicting the disease based on the patient's sex rather than the X-ray itself. This confirmed that the tool's score is a useful warning sign: a high score on the summary suggests that a model built on it is likely to be unfair if the training data is not perfect.

The team also tried to fix the problem by using four different methods to clean the summaries before training the diagnosis program. They tried techniques that attempt to remove the hidden clues, such as adversarial training, which tries to trick the system into forgetting the sensitive trait. Surprisingly, these methods did not work well. Even after the summaries were "cleaned" by these tools, the diagnosis programs still found and used the shortcuts. The only method that successfully stopped the unfair behavior was to simply retrain the final diagnosis program on a perfectly balanced set of data, where every group of patients was represented equally. This worked, but it did not change the summary itself; it only changed how the final program was taught. This suggests that for many of these systems, the decision to use a shortcut is made by the final program, not the summary, and that cleaning the summary alone is often not enough.

The researchers also tested their tool on a fake shortcut they created. They took a set of X-rays and stamped a small white square onto the images of patients who had a specific disease. They then fed these images into the system and asked if the summary could detect the square. The tool found it immediately and correctly in every single test, proving that it can spot real, physical shortcuts when they exist. However, the study also revealed a limitation: the tool is very good at finding shortcuts that shift the average appearance of a group, but it missed more subtle patterns where the groups look the same on average but differ in other ways. This means a clean score does not guarantee a model is safe, but a dirty score is a definite warning.

Ultimately, this work provides a practical way to audit the hidden biases in medical AI before they cause harm. It shows that most standard medical image summaries do carry information about a patient's demographics, and that this information can easily lead to unfair predictions if the training data is flawed. The study concludes that while we can measure this potential for shortcuts, fixing it is not as simple as scrubbing the data. The final diagnosis program must also be tested and trained carefully. The tool is now available for others to use, offering a way to check the foundation of medical AI systems and ensure they are built on evidence rather than hidden biases.

Technical Summary: ShortKit-ML

Problem Statement

Deep learning models in medical imaging often achieve high accuracy by relying on "shortcuts"—signals that predict the task label in the training data but are not part of the underlying clinical mechanism. These shortcuts can lead to model failure under distribution shifts and exacerbate fairness issues, such as performance gaps between demographic groups.

A critical distinction exists between encoding (an embedding $E$ containing information about a sensitive attribute $A$ ), use (a classifier trained on $E$ relying on that information), and harm (the resulting performance gap). While general-purpose encoders are expected to encode demographic attributes due to correlations with visible features, the presence of such information does not guarantee it will be used by a downstream classifier.

The paper addresses two specific questions:

Can shortcut potential—defined as the conditional dependence $I(E; A | Y) > 0$ (information about $A$ in $E$ beyond what the task label $Y$ explains)—be measured on a frozen embedding alone with a known false positive rate?
Does this measured potential predict how strongly a classifier trained on confounded data relies on the attribute?

Methodology: ShortKit-ML

The authors introduce ShortKit-ML, an open Python toolkit designed to audit shortcut potential in frozen embeddings without requiring access to the encoder's training process or gradients.

Core Mechanism: Conditional Permutation Test

The primary innovation is a conditional probe calibrated via permutation.

Null Hypothesis ( $H_0$ ): $E \perp A \mid Y$ . The embedding holds no information about the attribute $A$ beyond what is already explained by the task label $Y$ .
Calibration: To establish a null distribution with a known false positive rate, the attribute $A$ is permuted within strata of the task label $Y$ . Crucially, when multiple images belong to a single patient, the permutation occurs between patients (keeping all images of a patient together) to prevent data leakage.
Primary Test: A logistic regression predicts $A$ from the concatenation of the standardized embedding $E$ and the task label $Y$ . The score is the out-of-fold AUROC from 5-fold cross-validation. The permutation $p$ -value is calculated by comparing the observed score against the distribution of scores from permuted datasets.

Secondary Detectors

Five secondary detectors are provided to offer descriptive insights, all calibrated by the same conditional permutation null:

Statistical: Fraction of embedding dimensions showing significant group differences (Mann-Whitney/Kruskal-Wallis) after Benjamini-Hochberg correction.
Geometric: Distance between group centroids relative to pooled within-group spread.
Bias Direction PCA: Standardized gap of group means projected onto the first principal component of group-mean differences.
HBAC: Normalized mutual information between $k$ -means clusters and the attribute.
Frequency: Conditional probe AUROC restricted to the top 5% of dimensions ranked by group difference.

Experimental Setup

The toolkit was evaluated on:

Synthetic Benchmarks: Data with known ground truth, including linear mean shifts, nonlinear XOR signals, and variance shifts, with controlled correlations between $A$ and $Y$ .
Clinical Cohorts:
- CheXpert: 2,000 images from 477 patients (7 frozen encoders).
- MIMIC-CXR: 1,491 subjects (4 frozen encoders).
- Attributes: Sex, Age (binary and multi-bin), and Race.
Classification-Head Experiment: Logistic and MLP heads were trained on confounded CheXpert splits (where $P(Y=1|A=1) = 0.9$ ) to measure actual reliance (True Positive Rate gaps, AUROC drops) and compare it against the audit scores.
Mitigation: Four methods (adversarial debiasing, contrastive debiasing, shortcut masking, and last-layer retraining/DFR) were tested to see if reducing the audit score reduced reliance.

Key Results

1. Validity and Power on Synthetic Data

False Positive Rate: The conditional probe maintained its nominal error rate (rejecting $H_0$ in 0.02 of null datasets), whereas an unconditional probe failed (0.95 rejection rate) when $A$ and $Y$ were strongly correlated.
Signal Detection: The detectors successfully identified linear mean-shift signals. However, no detector (including the primary probe) could detect nonlinear XOR signals or variance-shift signals that left group means unchanged. This limitation is inherent to the linear statistics used.

2. Clinical Audit Findings

Ubiquity of Encoding: The audit detected attribute information in 21 of 21 CheXpert cells and 12 of 12 MIMIC-CXR cells across all encoders and attributes (Sex, Age, Race).
Magnitude: Sex showed the highest shortcut potential (AUROC 0.84–1.00), followed by Age (0.67–0.89) and Race (0.57–0.76).
Cluster Analysis: Embedding clusters were most strongly associated with image acquisition factors (view, projection) rather than demographic attributes, suggesting demographics are encoded in directions that do not dominate the overall embedding structure.

3. Correlation with Model Reliance

Prediction of Use: On CheXpert, the conditional probe score correlated with the True Positive Rate (TPR) gap of heads trained on confounded data (Spearman $\rho = 0.60$ , 95% CI 0.31–0.80).
Attribute Specificity: The correlation was strong for Sex ( $\rho = 0.74$ ) but not statistically significant for Age ( $\rho = 0.19$ ), likely due to range restriction in the age gaps. For Race, the signal was too weak to establish a correlation.
Conclusion: A higher audit score predicts a higher likelihood of reliance for linear heads on confounded data, particularly for sex.

4. Mitigation Limitations

Embedding-Level Methods: Adversarial debiasing, contrastive debiasing, and shortcut masking reduced the audit score but failed to reduce the reliance (TPR gap) of the downstream head beyond their respective controls (changes $\le 0.01$ ).
Head-Level Method: Refitting the head on a group-balanced subset (DFR) successfully closed the TPR gap (from 0.18 to 0.01) but did not change the audit score (as the embedding remained unchanged) and reduced task AUROC due to the small size of the balanced subset.

Significance and Claims

The paper claims that ShortKit-ML provides a calibrated, linear test to measure shortcut potential on frozen embeddings with a known false positive rate, serving as a "first step" for cases where the attribute is known and the input is a frozen embedding.

Diagnostic Value: The audit serves as a "first check" to determine if an embedding could be exploited for shortcuts. It successfully identified a planted image marker in all seeds, proving its sensitivity to artificial shortcuts.
Predictive Value: The audit score correlates with the reliance of downstream classifiers trained on confounded data, offering a quantitative metric for risk assessment before training a full model.
Limitations and Modesty: The authors explicitly state that a positive audit does not prove a specific head will use the attribute, and a negative audit does not rule out shortcuts (specifically nonlinear ones). Furthermore, reducing the audit score via embedding-level mitigation does not necessarily reduce harm; the decision of whether a shortcut is used is often made in the classification head.
Practical Impact: The toolkit highlights that general-purpose encoders inevitably encode demographic attributes. The authors argue that mitigation strategies must target the head (e.g., retraining on balanced data) rather than just the embedding, and that the audit should be part of a broader evaluation pipeline (e.g., DOJO) rather than a standalone solution.

The paper concludes that while shortcut potential can be measured on embeddings, the "harm" is determined by the interaction between the embedding, the training data, and the classifier head. Therefore, auditing the embedding is a necessary but insufficient step; the trained head must also be tested.