Technical Summary: Protecting Patient Privacy in Clinical Foundation Models

Problem Statement

Clinical foundation models (FMs), trained on large-scale patient data, are increasingly deployed for decision support, screening, and public health. While these models offer significant potential to improve patient outcomes, their expansion introduces novel privacy risks that existing regulatory frameworks (such as HIPAA and GDPR) struggle to address. Traditional frameworks focus on the direct handling of static data (collection, access, disclosure) and assume that de-identification effectively mitigates risk. However, FMs introduce model-mediated leakage, where sensitive information is disclosed not through direct data access, but through the model's learned parameters and generated outputs.

These leakage pathways are highly context-dependent, varying by deployment strategy (local vs. public) and data modality (text, imaging, biological measures). Current regulations lack guidance for "indirect threats" where models memorize training artifacts, reconstruct identifiable records, or reveal membership in specific cohorts, even when trained on de-identified data. The prevalence and severity of these risks remain poorly quantified, creating a gap between technical reality and legal compliance.

Methodology

The authors propose a two-dimensional, context-aware framework to assess privacy risk in clinical foundation models. This framework moves beyond binary classifications of "private" vs. "public" data to characterize leakage along two orthogonal axes:

1. Axis I: Prior Information Required: The amount and type of information an adversary must possess to trigger a leak.
   • No prior/public information: Leakage occurs via unconditional generation or prompts probing for memorized content without specific patient context.
   • Linkable personal information: Leakage requires partial inputs (e.g., a medication list, a specific diagnosis, or a demographic) that allow the model to infer additional sensitive details.
2. Axis II: Type and Extent of Leaked Information: The sensitivity of the information revealed by the model.
   • Linkable personal information: Reconstructed de-identified data that becomes identifying only when combined with external sources.
   • Identifiable personal information: Direct exposure of protected health information (PHI) or verbatim reproduction of private training data.

The methodology involves mapping realistic leakage scenarios across different deployment pathways (local deployment within healthcare systems vs. public release via APIs, web interfaces, or open weights) onto this risk space. These scenarios are then analyzed against current legal regimes in the US (HIPAA) and EU (GDPR, EU AI Act) to identify regulatory gaps and obligations.

Key Contributions

1. A Practical Risk Assessment Framework

The paper introduces a structured, two-axis framework that allows stakeholders to evaluate privacy risk based on the likelihood of obtaining prior information and the sensitivity of the potential leakage. This framework demonstrates that risk is continuous rather than categorical; a model behavior that is low-risk in a controlled local environment may become high-risk in a public release setting where auxiliary data is abundant.

2. Characterization of Leakage Scenarios

The authors detail six specific leakage scenarios (S1–S6) that illustrate how privacy breaches occur in practice:

• S1 (Reconstruction): Generative models producing near-replicas of training images (e.g., brain MRIs) that retain distinctive anatomical features.
• S2 (Memorization): Models generating clinical notes that closely match real patient histories when prompted with specific contextual clues (e.g., hospital location, date, symptoms).
• S3 (Leaked Autocompletion): Models inserting identifiers from one patient's record into another's during drafting (e.g., autocomplete suggesting a different patient's phone number).
• S4 (Contextual Autocompletion): Models generating highly specific narratives for rare clinical cases that mirror unique historical training examples.
• S5 (Membership Leakage): Inferring that a specific individual's data was part of the training set based on model embedding similarity or calibration behavior.
• S6 (Secondary Use): Reusing data consented for specific research to train general-purpose AI without renewed consent, creating "purpose drift."

3. Legal and Regulatory Analysis

The paper situates these technical risks within the US and EU legal landscapes:

• HIPAA Limitations: The authors note that HIPAA applies only to "covered entities" and "PHI." If data is stripped of 18 identifiers and the entity has no "actual knowledge" of re-identification risk, HIPAA may not be violated, even if the model later leaks information. The framework suggests that model-mediated leakage could constitute such "actual knowledge," but this remains legally uncertain.
• GDPR and EU AI Act: Under GDPR, "identifiability" is assessed relative to the means reasonably available to the controller. The paper argues that even de-identified data can be "personal data" if re-identification is reasonably possible via model outputs. The EU AI Act classifies many clinical FMs as "high-risk," requiring risk management, but currently lacks specific thresholds for memorization or reconstruction.

4. Mitigation Strategies

The paper outlines complementary technical and legal mitigations across the model lifecycle:

• Data Collection: Emphasizes the need for large, heterogeneous, multi-institutional cohorts to dilute rare samples and reduce memorization. It highlights the importance of consistent de-identification practices to prevent source inference.
• Model Development: Advocates for privacy-specific evaluations (membership inference, extraction tests, red-teaming) and technical controls like Differential Privacy (DP) to provide formal bounds on leakage.
• Deployment: Suggests graduated access controls, rate limiting, and architectural separation. It notes the limitations of current tort law in the US regarding unintentional data leakage and the lack of a private right of action under HIPAA.
• Post-Deployment: Proposes ongoing auditing, canary insertion, and "machine unlearning" to remove memorized content, though legal mandates for such remediation are currently under-specified.

Results and Findings

• Context is Critical: The same model behavior poses vastly different risks depending on the deployment setting. Public releases (open weights, APIs) significantly expand the threat surface compared to controlled local deployments.
• Regulatory Gaps: Existing frameworks are ill-equipped to handle dynamic, model-mediated leakage. They rely on static definitions of data handling rather than the probabilistic nature of inference and reconstruction in FMs.
• Risk Continuum: Privacy risk is not binary. It exists on a spectrum where the combination of low-barrier prior information and high-sensitivity leakage creates the most severe threats.
• Legal Uncertainty: There is significant ambiguity regarding whether model outputs constitute "disclosure" under current laws, particularly when the output is a probabilistic reconstruction rather than a direct data dump.

Significance

The paper claims that its primary significance lies in providing a practical, context-aware lens for assessing privacy risk that aligns technical realities with emerging regulatory expectations. By moving beyond abstract formal criteria to realistic usage scenarios, the framework helps:

1. Preserve Value: Enable the continued development and deployment of medical FMs for clinical benefit.
2. Safeguard Privacy: Rigorously identify and mitigate leakage pathways that current regulations overlook.
3. Guide Policy: Inform the evolution of legal standards to better reflect the nuances of inference, sensitivity, and probabilistic disclosure inherent in foundation models.

The authors conclude that while the legal landscape is in flux, a structured risk assessment is essential to balance innovation with patient safety, particularly as health AI systems expand beyond traditional clinical settings into broader public and commercial domains.