⚡ electrical engineering

Robustness Evaluation of a Foundation Segmentation Model Under Simulated Domain Shifts in Abdominal CT: Implications for Health Digital Twin Deployment

This study demonstrates that the Segment Anything Model (SAM) exhibits robust performance with minimal degradation in spleen segmentation accuracy across various simulated abdominal CT domain shifts, supporting its potential as a reliable foundation for health digital twin deployment.

Original authors: Sanghati Basu

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sanghati Basu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot assistant named SAM (Segment Anything Model). This robot was trained by looking at millions of photos of everyday things—cats, cars, trees, and people. It became incredibly good at drawing outlines around objects in those photos.

Now, doctors want to use this same robot to look at CT scans of human bodies to help build "Health Digital Twins." Think of a Digital Twin as a perfect, living digital copy of a patient that updates as they get new scans. For this digital copy to be accurate, the robot needs to be able to draw a perfect outline around organs, like the spleen, every single time, no matter what the CT machine looks like.

The problem is that CT machines aren't all the same. Some are older, some are newer, some take pictures that are a bit blurry, and some have different brightness settings. In the real world, this is called "domain shift." The big question this paper asks is: If we feed this robot a slightly "imperfect" or different-looking CT scan, will it get confused and draw the wrong outline?

Here is what the researchers did and found, explained simply:

The Experiment: The "Stress Test"

The researchers took 1,051 slices of CT scans showing the spleen from 41 different patients. They gave the robot a "perfect" hint (a box drawn exactly around the spleen) so the robot only had to focus on drawing the outline, not guessing where the spleen was.

Then, they played tricks on the images to simulate real-world problems, like:

Blur: Making the image look like it was taken with a shaky camera.
Noise: Adding static, like the "snow" on an old TV.
Brightness: Making the image too dark or too bright.
Zoom: Making the image look like it was taken from far away and then zoomed back in.

They ran the robot through 10 different versions of these "tricks" to see if its performance would crash.

The Results: The Robot is a Rock

The results were surprisingly good. Even when the images were messed with:

The robot didn't panic. Its ability to draw the correct outline (measured by a score called "Dice") stayed almost exactly the same. The score dropped by less than 1% in the worst cases, which is barely noticeable.
It didn't fail more often. Before the tricks, the robot failed to draw the outline correctly on less than 1% of the slices. After the tricks, it still failed on less than 1%.
Some tricks actually helped. Interestingly, adding a little bit of blur or noise actually made the robot slightly better. The researchers think this is because the blur smoothed out tiny, distracting "fuzz" in the medical images, making the edge of the spleen easier to see.

The "Digital Twin" Connection

The paper explains that for a Health Digital Twin to work, it needs a reliable "anatomical parser" (the part that identifies body parts). If the robot drawing the outlines gets confused because a scanner is slightly different, the whole digital twin becomes inaccurate.

The study concludes that SAM is stable enough to be used as the foundation for these digital twins in environments where scanners vary slightly (like different hospitals). It acts like a sturdy bridge that doesn't wobble when the wind (or scanner differences) blows.

Important Caveats (What the Paper Didn't Say)

While the news is good, the paper is careful to point out a few limits:

The "Perfect Hint": In this test, the researchers gave the robot a perfect box to start with. In the real world, the robot has to find the organ on its own. The paper says this test proves the drawing part is strong, but we still need to test if the finding part is just as strong.
One Organ Only: They only tested the spleen. The paper notes that other organs (like the liver or pancreas) might be harder to draw, and they haven't tested those yet in this specific study.
Not a "Cure-All": The robot is great at CT scans of solid organs, but the paper mentions it struggles with things like brain tumors or tiny cell structures (nuclei) because those look very different from the everyday photos the robot was originally trained on.

The Bottom Line

This paper is like a safety inspection for a new car engine. They put the engine (SAM) through a series of bumps, potholes, and rain (the image tricks) to see if it stalls. The engine didn't stall; it actually ran smoother on some bumps.

This gives researchers confidence that they can use this "foundation model" to build the next generation of Health Digital Twins, provided they keep testing it on other organs and with real-world "finding" tools. It's a strong step toward making these digital copies of patients reliable and trustworthy.

1. Problem Statement

Foundation models like the Segment Anything Model (SAM) have shown strong generalization in natural images but lack sufficient quantification of their robustness under clinically realistic medical imaging domain shifts. In clinical settings, Computed Tomography (CT) images vary significantly due to differences in scanner hardware, acquisition protocols, reconstruction kernels, and patient positioning.

For Health Digital Twins (HDTs)—dynamic computational models of patients that rely on continuous, accurate anatomical modeling—these domain shifts pose a critical risk. If the segmentation backbone (e.g., SAM) fails or degrades under inter-scanner variability, the digital twin's state updates, longitudinal monitoring, and downstream clinical decisions become unreliable. There is a need for a systematic audit to determine if foundation models can serve as stable baselines for HDT deployment.

2. Methodology

The study conducted a systematic, slice-level robustness audit of SAM (ViT-B) for spleen segmentation in abdominal CT scans.

Dataset: 41 abdominal CT volumes from the Medical Segmentation Decathlon (Task 09: Spleen), yielding 1,051 non-empty axial slices.
Preprocessing: Images were windowed (Level=50 HU, Width=400 HU), clipped to [0, 255], and converted to 3-channel RGB.
Prompting Protocol: A Ground Truth (GT)-derived bounding box was used as the prompt. This "oracle" approach isolates the encoder's robustness from prompt uncertainty, providing a clean upper-bound estimate of model stability.
Perturbation Conditions: Ten controlled perturbations were applied to simulate inter-scanner variability across five types:
1. Gaussian Blur: Kernel sizes $k=3, 7$ (simulating PSF variation).
2. Gaussian Noise: Standard deviations $\sigma=10, 25$ (simulating detector noise).
3. Resolution Mismatch: Downsample + Upsample factors of $0.5x$ and $0.25x$ .
4. Contrast Scaling: Multipliers $0.8$ and $1.2$.
5. Gamma Correction: Exponents $0.8$ and $1.2$.
Evaluation Metrics:
- Dice Similarity Coefficient (DSC) and Intersection over Union (IoU).
- $\Delta$ Dice: The difference between perturbed and clean baseline Dice scores.
- Failure Rate: Defined as slices with Dice < 0.5.
Statistical Analysis:
- Paired Wilcoxon signed-rank tests for $\Delta$ Dice significance.
- McNemar's test for changes in failure rates.
- Benjamini-Hochberg (FDR) correction applied to all $p$ -values to handle multiple comparisons.
- Bootstrap resampling (10,000 iterations) for 95% Confidence Intervals (CI).

3. Key Contributions

Standardized Evaluation Protocol: Introduced a GT-derived box-prompt protocol to isolate encoder robustness from prompt variability, establishing a reproducible framework for foundation model audits.
Large-Scale Robustness Audit: Evaluated 1,051 slices across 10 distinct domain shift conditions, quantifying performance degradation (or improvement) with statistical rigor.
Statistical Characterization: Provided quantified $\Delta$ Dice values, effect sizes (rank-biserial correlation), and failure rate analyses with FDR-corrected significance testing.
Digital Twin Integration Framework: Mapped SAM's role within a 5-stage Health Digital Twin pipeline, validating its suitability as the "Anatomical Parsing" layer.
Cross-Domain Benchmarking: Compared SAM's performance against specialist models (e.g., nnU-Net) across five medical domains (Spleen, Kidney, Liver, Brain MRI, Histopathology) to contextualize its zero-shot capabilities.

4. Key Results

Baseline Performance: Under clean conditions, SAM (ViT-B) achieved a mean Dice score of 0.9145 (95% CI: [0.909, 0.919]) with a failure rate of only 0.67%.
Robustness to Perturbations:
- Across all 10 conditions, the absolute mean $\Delta$ Dice remained below 0.01.
- Statistical Significance: While some conditions (e.g., mild blur, moderate downsampling) showed statistically significant improvements (likely due to noise suppression), and others (gamma reduction, contrast increase) showed slight degradation, the effect sizes were negligible ( $|r| < 0.08$ ).
- Failure Rates: No perturbation condition resulted in a statistically significant increase in the failure rate (Dice < 0.5) compared to the baseline (all FDR-adjusted $p > 0.05$ ).
Reliability Profile:
- 81.1% of slices achieved Dice $\ge$ 0.9.
- 99.3% of slices exceeded Dice $\ge$ 0.5.
Cross-Domain Comparison:
- High Performance: SAM performed best on CT parenchymal organs (Spleen, Kidney, Liver), with gaps to specialist models ranging from 0.049 to 0.090.
- Low Performance: Significant gaps were observed in Brain MRI (T1ce) and Histopathology (H&E), where natural image priors failed to align with medical visual features (e.g., edema vs. tumor, chromatin texture).

5. Significance and Implications

Validation for Health Digital Twins: The study confirms that SAM exhibits stable segmentation behavior under moderate inter-scanner variability. This makes it a viable, robust foundation baseline for the anatomical modeling layer of Health Digital Twins, ensuring that state updates and longitudinal monitoring are not corrupted by routine imaging variations.
Deployment Readiness: The findings suggest that for high-contrast CT structures (like the spleen), SAM can be deployed with confidence regarding encoder stability. However, the authors caution that:
- The use of GT-derived boxes is an "oracle" scenario; real-world automated prompting requires further validation.
- Performance varies by organ and modality; organs with complex shapes or low contrast (e.g., pancreas, brain tumors) may require specific validation.
Framework for Governance: The proposed audit protocol (controlled perturbation + paired statistical testing + reliability profiling) offers a reusable standard for validating any segmentation component before integration into clinical AI systems, a critical step toward trustworthy AI governance.

Conclusion: SAM (ViT-B) demonstrates sufficient robustness to serve as a reliable backbone for spleen segmentation in multi-site clinical environments, supporting its integration into Health Digital Twin architectures, provided that organ-specific and prompt-generation uncertainties are managed in future deployment phases.