Do Diabetic Foot Ulcer Segmentation Models Generalize? A Cross-Dataset Benchmark of CNN and Transformer Architectures

Technical Summary: Do Diabetic Foot Ulcer Segmentation Models Generalize?

Problem Statement
Deep learning models for Diabetic Foot Ulcer (DFU) segmentation routinely report high accuracy (Dice scores often >0.80) when trained and tested on the same dataset. However, these evaluations are almost exclusively "in-domain," failing to measure model behavior on data from different clinical sources, cameras, or patient populations. Prior cross-dataset studies have been limited to convolutional neural networks (CNNs) and prompt-based models (e.g., SAM), leaving the generalization capabilities of modern Transformer architectures in the DFU domain unmeasured. Furthermore, existing benchmarks often lack rigorous screening for data leakage (image overlap) between training and testing sets, potentially inflating generalization claims.

Methodology
The authors conducted a controlled, leakage-screened cross-dataset benchmark comparing three representative segmentation architectures:

• CNN Baselines: U-Net and DeepLabV3+ (both utilizing ResNet-34 encoders).
• Transformer Architecture: SegFormer-B2 (utilizing a MiT-B2 encoder).

Data Protocol:

• Training Source: A combined dataset of FUSeg and AZH wound images. The authors performed a rigorous audit using MD5 and perceptual hashing (dHash) to detect and remove image-level overlaps, treating FUSeg and AZH as a single source after confirming their significant overlap.
• External Test Sets: Two independent datasets with no image overlap with the training source:
  1. DFUC2022: 2,000 images from a different clinical site (External Test #1).
  2. Medetec: 160 images (External Test #2).
• Training Protocol: All models were trained for 30 epochs on 256×256 inputs using a combined Dice and binary cross-entropy loss. No fine-tuning was performed on the external test sets.
• Evaluation: Performance was measured using the Dice coefficient and Intersection-over-Union (IoU). Statistical significance was assessed via Wilcoxon signed-rank tests on per-image Dice scores. A per-image failure analysis defined "catastrophic failures" as images with Dice < 0.4.

Key Contributions

1. First Leakage-Screened Cross-Dataset Benchmark: This study is the first to apply a leakage-screening protocol to compare CNN and Transformer architectures for DFU segmentation across independent datasets.
2. Architecture Family vs. Complexity Analysis: The work isolates the impact of architecture family (Transformer vs. CNN) from model complexity, using DeepLabV3+ (complex CNN) and U-Net (simpler CNN) as baselines against SegFormer-B2.
3. Comprehensive Failure Analysis: Beyond aggregate metrics, the study provides a per-image failure analysis across 2,160 test images, categorizing failures and analyzing the severity of performance drops.

Results

• In-Domain Performance: All models achieved strong performance on the held-out FUSeg validation set (Dice 0.802–0.834), with SegFormer-B2 performing slightly better. The spread was narrow, suggesting architecture choice has minimal impact within a single data distribution.
• Cross-Dataset Generalization: All models experienced substantial performance degradation when tested on external datasets.
  • DFUC2022: SegFormer-B2 achieved the highest Dice (0.557), outperforming U-Net (0.501) and DeepLabV3+ (0.489). SegFormer-B2 also demonstrated superior stability, with a standard deviation (±0.003) an order of magnitude lower than U-Net (±0.029).
  • Medetec: SegFormer-B2 again led with a Dice of 0.786, compared to 0.737 (U-Net) and 0.730 (DeepLabV3+).
• Failure Rates: On DFUC2022, SegFormer-B2 produced the fewest catastrophic failures (31.1% of images with Dice < 0.4), compared to 38.5% for U-Net and 43.0% for DeepLabV3+.
• Statistical Significance: Wilcoxon signed-rank tests confirmed that SegFormer-B2 significantly outperformed both CNN baselines on both external datasets (p < 0.001). The difference between the two CNN models was not statistically significant on Medetec.
• Failure Severity: On Medetec, while SegFormer-B2 had a slightly higher count of failures than U-Net, its failures were less severe (clustering just below the 0.4 threshold), whereas CNN failures included a heavier "catastrophic tail" (Dice ≈ 0).

Significance and Claims
The paper concludes that architecture family is a more decisive factor for cross-hospital generalization than model complexity. Specifically, the Transformer-based SegFormer-B2 generalizes better than both the simpler U-Net and the more complex DeepLabV3+. The authors attribute this to the global self-attention mechanism in Transformers, which may capture broader contextual cues (e.g., skin texture, lighting gradients) that remain stable across hospitals, whereas CNNs with local receptive fields may overfit to dataset-specific spatial statistics.

A critical practical implication highlighted is that high in-domain Dice scores do not imply deployment readiness. Models achieving >0.80 Dice in-domain lost approximately 28–33 percentage points of performance when deployed across hospitals. While SegFormer-B2 offers measurably better generalization, the authors note that even the best model loses substantial accuracy across datasets, underscoring that cross-hospital robustness remains an open problem requiring further work in domain adaptation and test-time adaptation.