Visual concept ranking uncovers medical shortcuts used by large multimodal models

1. Problem Statement

Large Multimodal Models (LMMs) have achieved state-of-the-art performance in medical tasks, such as classifying skin lesions or analyzing chest radiographs. However, these models often rely on spurious correlations (shortcuts) rather than genuine pathological features, leading to safety risks and demographic disparities.

Existing interpretability methods have limitations:

• Saliency Maps: Good for spatial localization but struggle with abstract concepts (e.g., "overall brightness") or diffuse features (e.g., "background body part").
• Correlational Auditing (e.g., MA-MONET): These methods identify features correlated with model outputs in a specific dataset. They fail to distinguish between causal dependencies (what the model actually uses) and spurious correlations present only in the evaluation data (e.g., a hospital-specific marker that correlates with disease in the test set but not in the training set).
• Counterfactual Generation: Difficult to maintain cycle-consistency when modifying large image regions.

The authors aim to develop a method that identifies causal visual concept dependencies in LMMs, specifically to uncover shortcuts that cause performance disparities across demographic subgroups (e.g., skin types) when using In-Context Learning (ICL).

2. Methodology: Visual Concept Ranking (VCR)

The authors propose Visual Concept Ranking (VCR), a method to identify the most statistically significant visual concepts influencing an LMM's output. VCR uses a Vision-Language Model (VLM) as an "explainer" to label a probe set of images, then analyzes the target LMM's internal activations.

The algorithm consists of four main steps:

1. Concept Labeling via VLM:

   • A pre-trained VLM (e.g., OpenCLIP) is used to compute the similarity between a probe set of images ($I$) and a large set of text concepts ($C$, e.g., 20,000 words).
   • This generates a concept label matrix $Y$, where each entry quantifies the visual presence of a concept in an image.

2. Learning Concept Activation Vectors (CAVs):

   • Activations are extracted from a specific layer ($\ell$) of the target LMM ($f$) for each image in the probe set.
   • A linear regression model is trained to predict the concept labels ($Y$) from the LMM's activation vectors ($A$).
   • The weights of this regression define a Concept Activation Vector (CAV) for each concept, representing the direction in the activation space corresponding to that concept.

3. Measuring Sensitivity (Directional Derivatives):

   • To determine importance, the method calculates the directional derivative of the task score (e.g., log-probability of "malignant") with respect to the CAV.
   • This measures how much the model's output changes if the activation is perturbed along the direction of the concept.
   • Unlike correlational methods, this uses gradients to measure causal sensitivity.

4. Significance Testing:

   • The process is repeated with bootstrap resampling of the probe set.
   • A two-sided t-test with Bonferroni correction is applied to determine if a concept's sensitivity score is statistically significantly different from zero.

3. Key Contributions

• Novel Method (VCR): A framework for auditing LMMs that combines VLM-based concept labeling with gradient-based sensitivity analysis to identify causal visual dependencies.
• Robustness to Distribution Shift: VCR is shown to correctly identify causal features even when the evaluation data has different feature-label correlations than the training data, a scenario where purely correlational methods fail.
• Discovery of Demographic Shortcuts: The method revealed that LMMs rely on blue/purple skin markings (used by dermatologists to mark lesions for biopsy) as a shortcut for malignancy. This dependency was found to be dynamic: it persisted for darker skin tones (FST V/VI) even with In-Context Learning (ICL) examples, but disappeared for lighter skin tones (FST I/II) when ICL was added.
• Discovery of Anatomical Biases: VCR identified that models systematically predict lower malignancy probabilities for lesions on faces/necks and scalps compared to generic skin backgrounds, likely due to the presence of "hair" or "collar" concepts.

4. Results

A. Synthetic Data Benchmarks

• Validation: On synthetic datasets with known ground-truth feature-outcome relationships, VCR sensitivity scores showed a strong correlation (Pearson's $r \approx 0.53$) with actual interventional effects.
• Distribution Shift: In adversarial settings where a "spurious" feature was negatively correlated with the label during training but positively correlated during testing:
  • VCR: Correctly identified the true causal effect 92% of the time.
  • Correlational Baseline (CLIP-only/MA-MONET): Correctly identified the effect only 18% of the time, falsely flagging the spurious feature as important.

B. Dermatology Experiments (DDI Dataset)

• Skin Markings: VCR identified "tattoo" and "Purpura/Petechiae" as top concepts. Manual intervention (adding blue/purple dots to synthetic skin) confirmed that these markings increased malignancy predictions for darker skin (FST V/VI) in both zero-shot and ICL settings, but only in zero-shot for lighter skin (FST I/II).
• Anatomical Context:
  • Concepts like "neck," "collar," and "teeth" were found to decrease malignancy predictions.
  • Intervention experiments showed that lesions on face/neck backgrounds were predicted as significantly less malignant than identical lesions on neutral backgrounds.
  • Similarly, "hair" concepts (specifically scalp images) reduced malignancy predictions.

C. Supplemental Experiments

• Chest Radiographs (CheXpert): VCR identified "wiring" (pacemakers, EKG leads) as a top predictor for "abnormal" scans, revealing a shortcut where the model detects medical devices rather than pathology.
• Natural Images (Imagenette): Demonstrated the method's versatility on non-medical tasks (e.g., distinguishing fish from golf balls), identifying semantically coherent concepts (e.g., "logos" for golf balls).

5. Significance and Implications

• Safety-Critical Auditing: VCR provides a rigorous tool to uncover hidden shortcuts in LMMs before clinical deployment, preventing models from making decisions based on irrelevant artifacts (like ink markings or hospital-specific markers).
• Demographic Equity: The study highlights that model behaviors are not static; they change based on the prompt (zero-shot vs. ICL) and vary across demographic subgroups. This suggests that careful curation of demonstration examples is crucial for equitable AI.
• Beyond Correlation: By leveraging model internals (gradients and activations), VCR overcomes the fundamental flaw of correlational auditing, ensuring that identified features are truly driving the model's decisions.
• Limitations: The method requires access to model gradients (precluding API-only models) and struggles with spatial location concepts (where saliency maps remain superior). It is best viewed as a hypothesis-generation tool requiring human validation.

In conclusion, Visual Concept Ranking offers a powerful, causally-grounded approach to auditing LMMs, revealing that these models often rely on subtle, context-dependent shortcuts that can lead to significant diagnostic errors and demographic disparities.