sFRC for assessing hallucinations in medical image restoration

This paper proposes sFRC, a novel method that performs Fourier Ring Correlation analysis over small patches to robustly detect and quantify hallucinations in deep learning-based medical image restoration across various undersampled imaging problems.

The Gist

Imagine you are looking at a medical scan, like an X-ray or an MRI. These images are crucial for doctors to diagnose diseases. Now, imagine a super-smart AI assistant is hired to clean up these images, making them look sharper and clearer, especially when the original scan was blurry or incomplete.

The problem? This AI is a bit of a creative writer. Sometimes, in its eagerness to make the picture look "perfect," it starts hallucinating. It might invent a blood vessel that doesn't exist, smooth over a tumor until it disappears, or add a fake plaque on an artery. To the naked eye, the image looks beautiful and smooth, but it's actually lying to the doctor. This is dangerous because a doctor might miss a real disease or treat a fake one.

This paper introduces a new tool called sFRC (scanning Fourier Ring Correlation) to catch these lies. Here is how it works, explained simply:

1. The Problem: The "Too Good to Be True" Image

Think of the AI like a photo editor who is given a low-quality, blurry photo of a forest and asked to make it high-definition.

• The Goal: The AI adds trees, leaves, and birds to fill in the missing details.
• The Hallucination: The AI gets carried away and adds a dragon sitting on a branch. The dragon looks realistic, but it's not real.
• The Danger: If a doctor looks at this "enhanced" image, they might think, "Wow, that's a clear image," and miss the fact that the dragon (or a fake tumor) isn't real.

2. The Solution: The "Microscope" Approach (sFRC)

Traditional ways of checking image quality are like looking at the whole forest from a helicopter. You might see that the forest looks green and healthy, but you can't see the dragon.

sFRC is different. Instead of looking at the whole image at once, it acts like a magnifying glass that scans the image in tiny, overlapping squares (patches).

Here is the step-by-step process:

• The Comparison: The tool takes the AI's "enhanced" image and compares it side-by-side with the "original" raw data (the blurry version) or a mathematically perfect version of what should be there.
• The Frequency Check: Imagine the image is a song. It has bass (low frequencies, like the shape of the body), mids (like organs), and treble (high frequencies, like tiny details and edges).
  • The AI is good at the bass and mids.
  • But when it tries to invent the treble (the tiny details), it often gets it wrong.
• The "Lie Detector": sFRC checks the "treble" part of the song in each tiny square. If the AI's version of the treble doesn't match the physics of how the original scan was taken, sFRC flags it.
• The Red Box: If a patch is found to be "lying" (hallucinating), sFRC draws a red box around it, telling the doctor: "Hey, look here. This part of the image was made up by the computer. Don't trust it."

3. Why This Matters: The "Smoothie" Analogy

Imagine you are making a smoothie.

• The Real Thing: You have real fruit (the patient's actual anatomy).
• The AI: It tries to blend the fruit into a perfect, smooth drink.
• The Hallucination: To make it taste "perfect," the AI secretly adds a spoonful of sugar that isn't in the recipe.
• Old Metrics: Old ways of checking quality just tasted the whole smoothie and said, "It tastes sweet and good!" (High score). They missed the extra sugar.
• sFRC: This tool tastes the smoothie in tiny sips. It says, "Wait, this sip tastes like sugar that wasn't in the fruit. That's a fake addition."

4. What the Authors Found

The researchers tested this tool on three different medical scenarios:

1. CT Scans (Super-Resolution): Making low-res CT scans look high-res.
2. MRI Scans: Filling in missing data to speed up scans.
3. Sparse Views: Reconstructing images from very few angles.

In all cases, they found that:

• The AI often created fake structures (like extra blood vessels or fake plaques) that looked real but were dangerous.
• Traditional "quality scores" (like PSNR or SSIM) were fooled. They gave the AI high grades even when it was lying.
• sFRC successfully caught these lies, highlighting exactly where the AI was making things up.

The Bottom Line

This paper gives us a lie detector for medical AI.

As we start using more AI to help doctors see inside our bodies, we need to make sure the AI isn't just "making things up" to look pretty. sFRC is a safety net. It doesn't just tell us if an image looks good; it tells us if the image is truthful. It ensures that when a doctor sees a tumor or a broken bone, it's actually there, and not just a hallucination created by a computer trying too hard to be helpful.

Technical Summary

Here is a detailed technical summary of the paper "sFRC for assessing hallucinations in medical image restoration."

1. Problem Statement

Deep learning (DL) methods are increasingly used to restore medical images from sparse-view, limited-data, or undersampled acquisitions (e.g., low-dose CT, accelerated MRI). While these methods often produce visually appealing results with high data-fidelity metrics (PSNR, SSIM), they suffer from "hallucinations."

• Definition: Hallucinations are false structures (additive) or missing true structures (subtractive) generated by the DL model that are not present in the patient but may be mistaken for real anatomy.
• The Gap: Existing evaluation metrics fail to detect these hallucinations:
  • Global Metrics (PSNR, SSIM, RMSE): Correlate poorly with the preservation of subtle features and often mask local errors.
  • Physical Metrics (MTF, NPS): Designed for linear systems and uniform phantoms; they cannot assess non-linear DL behavior on complex patient data.
  • Task-Based Observer Studies: Limited to specific signal-detection tasks and cannot comprehensively characterize unpredictable, variable hallucinations.
  • Global Distribution Metrics: Fail to capture locally confined hallucinations.

2. Methodology: Scanning Fourier Ring Correlation (sFRC)

The authors propose sFRC, a local, patch-based metric that adapts the Fourier Ring Correlation (FRC) technique (traditionally used for resolution estimation in microscopy) to detect hallucinations.

Core Rationale

1. Locality: Hallucinations are typically confined to small Regions of Interest (ROIs) rather than the entire image.
2. Frequency Isolation: Hallucinations often manifest as mismatches in specific spatial frequency bands (typically mid-frequencies) when comparing a DL output to a reference.

Algorithm Workflow

1. Patch Scanning: The DL output and a reference image (fully sampled, reconstructed via analytical methods like FBP or iFFT) are divided into small, overlapping patches (e.g., $64 \times 64$).
2. FRC Calculation: For each patch pair, the FRC is calculated. FRC measures the correlation between two images as a function of spatial frequency ($q$).
   $$FRC(q_i) = \frac{\sum_{q \in ring_i} \hat{f}_1(q) \cdot \hat{f}_2^*(q)}{\sqrt{\sum_{q \in ring_i} |\hat{f}_1(q)|^2 \sum_{q \in ring_i} |\hat{f}_2(q)|^2}}$$
3. Thresholding ($x_{ct}$): The FRC curve for a patch is analyzed to find the spatial frequency ($x_{ct}$) where the correlation drops below a predefined FRC Threshold ($Y$) (e.g., 0.5).
4. Hallucination Detection ($x_{ht}$): A Hallucination Threshold ($x_{ht}$) is established.
   • If a patch's $x_{ct} \le x_{ht}$, the patch is labeled as a hallucinated ROI.
   • $x_{ht}$ is tuned using a development set where experts or imaging theory have identified confirmed hallucinations. It acts as a cutoff: mismatches occurring at frequencies lower than $x_{ht}$ indicate the DL method failed to faithfully restore data up to that frequency.

Parameter Tuning

• Patch Size: Optimized to balance sensitivity (small patches catch local errors) and false positives (large patches dilute local errors with faithful regions). The paper finds $64 \times 64$ optimal for CT super-resolution.
• FRC Threshold ($Y$): Set based on the imaging system's resolving power.
• Hallucination Threshold ($x_{ht}$): Can be set via:
  • Expert Annotation: Using ROIs manually marked by radiologists/imaging scientists.
  • Imaging Theory: Using null-space hallucination maps derived from linear operator theory (e.g., Bhadra et al.'s approach).
  • Sampling Rate: Initialized based on the subsampling factor (e.g., $0.25 \times$ Nyquist for 4x downsampling).

3. Key Contributions

1. Novel Metric (sFRC): Introduces a local, frequency-based metric specifically designed to detect hallucinations in non-linear image restoration.
2. Objectivity & Automation: Unlike visual inspection, sFRC automatically flags candidate hallucinated ROIs using tunable thresholds derived from theory or expert input.
3. Generalizability: Demonstrated effectiveness across:
   • CT Super-Resolution: Using SRGAN and SR-WGAN.
   • MRI Subsampled Restoration: Using U-Net and PLS-TV (conventional regularization).
   • CT Sparse View: Using state-of-the-art Progressive Artifact Image Learning (PAIL).
4. Hallucination Operating Characteristic (HOC) Curve: Proposes a curve analogous to the ROC curve, plotting hallucination rate against the $x_{ht}$ threshold to summarize performance across different strictness levels.
5. Data Processing Inequality Validation: Shows that DL methods, despite high PSNR/SSIM, cannot fully compensate for information loss in subsampled data, as evidenced by persistent hallucination rates.

4. Results

The study evaluated sFRC on three distinct medical imaging problems:

• CT Super-Resolution (SRGAN/SR-WGAN):

  • sFRC successfully detected hallucinations such as "plaque-like" structures, indentations, and over-smoothing that were missed by PSNR/SSIM.
  • Out-of-Distribution (OOD) Test: When trained on smooth kernels and tested on sharp kernels, hallucination rates increased significantly (e.g., from 3.7% to 6.5%), which sFRC captured but global metrics did not.
  • Clinical Impact: Detected false "air" in fatty regions and added vessel structures, which could lead to misdiagnosis.

• MRI Subsampled Restoration (U-Net & PLS-TV):

  • sFRC results aligned well with theoretical hallucination maps (Bhadra et al.).
  • Detected subtle errors like sulci removal, grey matter thickening, and banding artifacts.
  • Comparison: U-Net (3x acceleration) had high PSNR/SSIM (comparable to iFFT 1x) but a hallucination rate of ~22%, whereas iFFT 1x had ~1%. This highlights the "illusion" of quality in DL outputs.

• CT Sparse View (PAIL):

  • Even with a state-of-the-art hybrid physics-AI model (PAIL), sFRC detected subtle hallucinations (e.g., smoothed bowel walls, lost muscle dividers) that were invisible to the naked eye without a reference.

• Comparison with Other Metrics:

  • Global metrics (PSNR, SSIM, Hellinger Distance) often ranked hallucinated DL outputs as superior or comparable to faithful reconstructions.
  • sFRC was the only metric that consistently correlated with the presence of local, clinically significant errors.

5. Significance and Implications

• Patient Safety: Provides a critical tool for regulatory bodies (like the FDA) and developers to ensure AI medical devices do not introduce dangerous false structures.
• Algorithm Optimization: Allows developers to visualize where and what type of hallucinations occur (via red bounding boxes), enabling targeted algorithmic improvements.
• Beyond DL: The method is applicable to any non-linear restoration technique, including conventional regularization methods, not just deep learning.
• Limitations & Future Work:
  • Requires a reference image (cannot be used for unconditional generation).
  • Requires careful tuning of $x_{ht}$ for specific modalities.
  • Future work involves validating the HOC curve against downstream clinical tasks (e.g., CAD performance) and creating large-scale annotated datasets for hallucination atlases.

In conclusion, sFRC bridges the gap between subjective visual assessment and quantitative global metrics, offering a robust, objective, and automated framework to quantify the reliability of AI-driven medical image restoration.