Cross-modal linkage risk in clinical vision-language models

Technical Summary: Cross-Modal Linkage Risk in Clinical Vision-Language Models

Problem Definition
Clinical Vision-Language Models (VLMs) trained on paired medical images and radiology reports learn a shared embedding space to facilitate tasks like retrieval, report generation, and zero-shot transfer. While this alignment is a functional strength, the authors identify a specific privacy vulnerability: cross-modal linkage risk. In deployment scenarios where radiographs and reports are deliberately kept separate after acquisition (e.g., image-only data sharing, access-controlled reports, or institutional data enclaves), the shared embedding space may inadvertently preserve instance-level correspondence. This allows a de-identified image to be re-linked to its original, information-rich narrative report solely through cosine similarity search, bypassing standard de-identification measures which typically remove explicit metadata but do not erase learned semantic bindings.

Methodology
The study formalizes this risk as an image-to-report retrieval problem. The authors evaluate whether a held-out chest radiograph can retrieve its true paired report from a candidate pool of distractor reports using only the shared embedding space.

• Datasets: The evaluation utilizes two large public cohorts as ground-truth benchmarks (where true pairings are known by design): MIMIC-CXR (43,793 held-out pairs) and CheXpert Plus (29,296 pairs). These serve to audit the model's internal structure rather than as the privacy attack scenario itself.
• Models: Four publicly available VLMs with increasing levels of domain specialization were tested:
  1. CLIP: General-domain.
  2. PubMedCLIP: Biomedical domain.
  3. BiomedCLIP: Biomedical domain.
  4. BioViL-T: Radiology-specific (chest radiography).
• Evaluation Protocols:
  • Random-Candidate (Setting A): Retrieval performance is measured across candidate pool sizes ($N = 100, 1,000, 10,000$, and full test set) to assess scalability.
  • Pathology-Matched Hard Negatives (Setting B): Distractor reports are selected to match the query's 14-label pathology vector (e.g., atelectasis, edema, pneumothorax) to determine if retrieval relies on broad disease similarity or finer-grained instance-level binding.
• Mitigation Strategy: The authors propose a targeted intervention where the image and text encoders are frozen, and only the shared alignment layer (projection heads) is fine-tuned.
  • Differentially Private (DP) Optimization: Applied to the projection heads with $\epsilon = 0.34$ and $\delta = 6 \times 10^{-6}$.
  • Comparator: An entropy-regularized head fine-tuning (non-private) was used to ensure that any reduction in linkage was due to the privacy mechanism and not merely architectural perturbation.
• Utility Assessment: To ensure the mitigation does not degrade clinical utility, image-side diagnostic performance was evaluated via linear probing on 14 chest radiograph abnormality tasks using the modified embeddings.

Key Results

1. Existence and Magnitude of Risk: Off-the-shelf VLMs successfully re-link de-identified images to their reports at rates significantly above chance. The risk scales with domain specialization:

   • On MIMIC-CXR with $N=100$, BioViL-T achieved a Recall@1 of 15.44% (vs. 1% chance), while general CLIP was near chance (1.20%).
   • At database scale ($N=10,000$), BioViL-T still retrieved the correct report 50 times above chance (0.496% vs. 0.010%).
   • The signal persists in external transfer (CheXpert Plus) and under hard-negative conditions, indicating the models learn instance-level bindings beyond coarse pathology labels.

2. Mitigation Efficacy: Freezing the backbones and applying DP optimization only to the projection heads significantly reduced linkage risk:

   • On MIMIC-CXR ($N=10,000$), DP head fine-tuning reduced Recall@1 by 61.8% (from 0.496% to 0.190%).
   • The mitigation transferred to the external CheXpert Plus dataset without retraining, reducing Recall@1 from 0.153% to 0.091%.
   • In contrast, the non-private entropy-regularized comparator increased linkage, confirming that the privacy benefit stems from the specific DP noise injection weakening the coupling, not just changing the projection space.

3. Privacy-Utility Tradeoff: The intervention demonstrated a highly asymmetric tradeoff favoring privacy preservation:

   • Privacy: Recall@1 was reduced by ~62%.
   • Utility: The macro AUROC for image-only classification across 14 labels shifted minimally from 79.63% to 79.43% (a decrease of 0.20 percentage points).
   • The per-label analysis showed small, mixed changes in AUROC, suggesting the underlying image representations remained clinically useful despite the weakening of cross-modal binding.

Significance and Claims
The paper argues that multimodal alignment is both a capability and a liability in clinical foundation models. The authors claim that:

• Current de-identification practices are insufficient for VLMs because the shared embedding space itself acts as a leakage channel, preserving recoverable instance-level structure even when explicit identifiers are removed.
• This risk is not an artifact of small candidate pools or broad disease similarity; it persists at database scale and under clinically constrained hard negatives.
• Targeted mitigation is feasible: One does not need to retrain large multimodal backbones (which is computationally expensive and unstable under DP) to mitigate this risk. Instead, applying differentially private optimization solely to the final alignment layer can substantially reduce cross-modal re-linkage without materially degrading the image representations required for clinical utility.
• The study calls for a shift in the evaluation of clinical multimodal models, urging that responsible deployment must account not only for what shared embeddings enable (e.g., retrieval, generation) but also for what they may unintentionally reconnect.

The authors maintain a modest scope, noting limitations such as the focus on chest radiography, the use of public datasets as proxies for private deployment scenarios, and the evaluation of utility via linear probing rather than a full suite of downstream clinical tasks. They conclude that while the proposed mitigation is a targeted reduction of a specific linkage mechanism rather than a complete solution to all multimodal privacy issues, it offers a practical pathway for safer deployment of existing clinical VLM checkpoints.