💻 computer science

A Calibrated Memorization Index (MI) for Detecting Training Data Leakage in Generative MRI Models

This paper proposes a calibrated Memorization Index (MI) that leverages MRI foundation model features and nearest-neighbor similarities to robustly detect and quantify training data duplication in generative MRI models, achieving near-perfect duplicate detection across diverse datasets.

Original authors: Yash Deo, Yan Jia, Toni Lassila, Victoria J Hodge, Alejandro F Frang, Chenghao Qian, Siyuan Kang, Ibrahim Habli

Published 2026-02-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yash Deo, Yan Jia, Toni Lassila, Victoria J Hodge, Alejandro F Frang, Chenghao Qian, Siyuan Kang, Ibrahim Habli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Photocopier" That Won't Stop Copying

Imagine you hire a very talented artist to learn how to paint landscapes. You show them 1,000 photos of mountains, rivers, and forests. Your goal is for them to learn the style and create new, original paintings.

However, this artist has a bad habit: instead of learning the style, they just memorize the photos. When you ask for a new painting, they hand you back one of the original photos you gave them, maybe with a tiny smudge of paint on it to make it look "new."

In the world of AI, this is called Data Leakage or Memorization.

Why is this bad? In medical imaging (like MRI scans), if an AI just copies a patient's real scan instead of creating a fake one, it violates patient privacy. It's like the AI accidentally handing you a stranger's medical record thinking it's a new drawing.
The Current Problem: We didn't have a good way to catch this. The tools we usually use to check if AI art looks "good" actually get better when the AI cheats! If the AI copies the training data, the "quality score" goes up, tricking us into thinking the AI is doing a great job when it's actually just cheating.

The Solution: A "Memory Detector" (The MI)

The authors of this paper built a new tool called the Memorization Index (MI). Think of this as a lie detector test for AI images.

Here is how it works, step-by-step:

1. The "X-Ray Vision" (Feature Extraction)

Instead of just looking at the picture with normal eyes, the tool uses a special pair of glasses called an MRI Foundation Model.

Analogy: Imagine looking at a fingerprint. A normal person sees a swirl. This special tool sees the ridges, the depth, the unique whorls, and the tiny scars. It looks at the image on many different levels of detail, from the big picture (the whole brain) down to the tiny textures.

2. The "Whitening" Process (Cleaning the Noise)

Medical images often have static or "noise" (like TV snow) that isn't important.

Analogy: Imagine you are trying to compare two voices in a noisy room. First, you turn off the background noise (the wind, the traffic) so you can hear the voices clearly. This tool does that mathematically, stripping away the "static" so it can focus only on the unique "voice" of the image.

3. The "Look-Alike" Search (Similarity Check)

The tool takes a new image generated by the AI and asks: "Does this look exactly like any of the photos I was trained on?"

Analogy: It's like a bouncer at a club checking a guest list. If the guest (the new image) looks 99% like someone on the VIP list (the training data), the bouncer flags them.
The Twist: Most tools get confused if the guest is wearing a hat or standing sideways. This tool is smart enough to realize, "Even though they turned their head, the bone structure is identical. This is a copy."

4. The Score: The "Novelty Meter" (ONI)

Finally, the tool gives a score between -1 and +1.

-1 (The Copy): "This is a direct copy of a training image. The AI is cheating."
0 (The Normal): "This looks like a normal image from the dataset, but not a specific copy."
+1 (The Original): "This is a brand new, unique creation that doesn't exist in the training data."

Why Is This Tool Special?

The paper tested this tool on brain, knee, and spine scans and found it beats all other methods for three main reasons:

It Doesn't Get Fooled by Tricks:
If you take a copied image and rotate it slightly or add a little digital noise, old tools get confused and say, "Oh, this is different!" This new tool says, "Nope, I still recognize you." It is 6 to 20 times more stable than previous methods.
It Works Everywhere (Universal):
Old tools gave different scores depending on whether you were looking at a brain or a knee. It was like a thermometer that said "Hot" for a fever in New York but "Cold" for the same fever in London. This new tool is calibrated, meaning a score of -0.8 means "Copy" whether you are looking at a brain or a spine.
It Catches the Culprit:
Most tools just give a score for the whole group of images (e.g., "This batch is 10% bad"). This tool can point to specific images and say, "Hey, Image #42 is a copy. Throw it out." This allows researchers to clean their data and fix the AI.

The Bottom Line

This paper introduces a smart, reliable lie detector for medical AI. It stops AI models from secretly copying patient data and passing it off as new work. By using this tool, hospitals and researchers can ensure that their AI is actually learning to create new, safe medical images rather than just photocopying private patient records.

In short: It turns the AI's "cheating" from a hidden secret into a flag that can be easily seen and fixed.

1. Problem Statement

Generative AI models used for medical imaging (specifically MRI) face a critical privacy risk: training data memorization (leakage). Unlike natural images, medical images often contain subtle, patient-specific anatomical details. If a model memorizes training data and reproduces it, it compromises patient privacy.

Current methods for detecting this leakage are inadequate for medical applications due to three main issues:

Misleading Fidelity Metrics: Standard metrics like FID and MMD measure distributional similarity. When a model memorizes data, the test distribution moves closer to the training distribution, causing these scores to improve (suggesting better quality) even though leakage has increased.
Feature Space Mismatch: Existing duplication detection metrics (e.g., CT-score, Vendi score) rely on features from models trained on natural images (e.g., InceptionV3). These fail to capture the high anatomical regularity and limited appearance diversity of medical images, often mistaking noise for diversity or failing to detect near-duplicates.
Lack of Calibration: Existing metrics suffer from arbitrary scaling effects across different datasets, making it impossible to set universal thresholds for what constitutes "memorization."

2. Methodology

The authors propose a Calibrated Memorization Index (MI) and an Overfit/Novelty Index (ONI). The pipeline consists of four key stages:

A. Domain-Specific Feature Extraction

Instead of generic vision models, the method uses MRI-CORE, a foundation model pretrained on diverse MRI datasets.

Multi-Scale Extraction: Features are extracted from specific transformer blocks ( $k \in \{3, 7, 11\}$ $k \in {3, 7, 11}$ ) of the ViT-B encoder.
- Early layers capture fine-grained textures.
- Deeper layers capture gross anatomical structures.
Pooling: Spatial tokens are reshaped and pooled via adaptive average pooling to create feature vectors $f^{(k)}$ .

B. Whitening and Similarity Computation

To ensure cross-layer comparability:

ZCA Whitening: Applied to training features per layer to decorrelate features while preserving the original feature space (unlike PCA).
Nearest-Neighbor Search: For each test sample, the maximum cosine similarity to any training sample is computed across all selected layers.

C. Multi-Scale Aggregation

The per-layer similarities are aggregated into a single sample-level score using a geometric mean (log-average).

Rationale: This acts as a "product-of-experts." A sample is only scored as highly similar if it is close to a training sample across all scales (early, mid, and late features). This suppresses outliers where only one layer finds a match and penalizes cases where any scale fails to find a neighbor.

D. Calibration and Indexing

To make the metric interpretable and dataset-independent:

Empirical Null Distribution: The authors bootstrap the training set (sampling independent subsets) to establish a baseline distribution of similarities between unrelated samples.
Memorization Index (MI): Test similarities are standardized using the null mean ( $\mu_{null}$ ) and standard deviation ( $\sigma_{null}$ ).
Overfit/Novelty Index (ONI): The MI is mapped to a bounded range $[-1, 1]$ $[- 1, 1]$ using a hyperbolic tangent function:
- $\approx -1$ : High similarity (Potential Memorization/Exact Copy).
- $\approx 0$ : Consistent with the null distribution (Typical similarity between unrelated samples).
- $\approx +1$ : High Novelty.

3. Key Contributions

Domain-Specific Metric: First metric specifically designed for medical imaging memorization, leveraging MRI foundation models rather than natural image features.
Calibration: Introduces a method to normalize scores across different datasets, enabling universal thresholds for leakage detection.
Robustness: Demonstrates superior stability against common data augmentations (noise, rotation, flips) compared to existing state-of-the-art metrics.
Sample-Level Detection: Unlike set-level metrics, this method provides per-sample scores, allowing for the identification and removal of specific leaked images.

4. Experimental Results

The method was evaluated on three public MRI datasets: BRATS (brain), Knee, and Spine. Controlled duplication levels (5% to 45%) were introduced with various augmentations.

Robustness to Augmentation:
- The proposed metric (MI) showed 6–20 $\times$ lower standard deviation across different augmentations compared to the CT-score.
- While CT-score values fluctuated wildly with noise and rotation, MI remained stable and monotonic.
Cross-Dataset Consistency:
- The Coefficient of Variation (CV) for MI across datasets was 5.5 $\times$ lower than CT-score.
- This allows for a universal threshold (e.g., ONI < 0.68 indicates potential leakage) regardless of the specific MRI dataset.
Detection Performance (Sample-Level):
- Clean/Noise/Intensity: Achieved near-perfect detection (AUC $\approx$ 1.0).
- Geometric Transforms: Maintained strong performance (AUC > 0.72) even with rotations ( $\pm 5^\circ$ ) and flips.
Comparison with Other Metrics:
- FID/MMD: Decreased as duplication increased (misleadingly suggesting better quality).
- Vendi Score: Showed negligible signal.
- AuthPct: Highly sensitive to augmentations.

5. Significance

This work transforms memorization assessment from a passive diagnostic tool into a practical curation tool for medical AI.

Privacy Assurance: It enables developers to detect and remove specific patient images that have been memorized, directly addressing privacy concerns in generative medical AI.
Reliability: By providing a calibrated, stable metric, it prevents the "false sense of security" provided by improving FID scores when leakage occurs.
Generalizability: While tested on MRI, the methodology is applicable to other medical imaging modalities provided a domain-specific foundation model is used.

Limitations: The method shows slightly reduced sensitivity under small geometric rotations/flips at very low leakage levels (5%), though the authors suggest lightweight remedies like rotation pooling or geometric jitter in the null calibration.