📄 health informatics

FootNet: A Multi-View Smartphone Dataset and Four-Model Benchmark for Clinical Foot Segmentation

This paper introduces FootNet, a multi-view smartphone dataset with expert-annotated masks for clinical foot segmentation, and establishes a benchmark demonstrating that a U-Net with a MobileNetV2 encoder significantly outperforms other models, including DeepLabV3, UNet++, and SAM ViT-B, in terms of segmentation accuracy.

Original authors: Vijay, A., Prabhune, A., Srihari, V. R., Rayampalli, A.

Published 2026-07-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vijay, A., Prabhune, A., Srihari, V. R., Rayampalli, A.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are trying to teach a robot to see the world, but instead of giving it a perfect, studio-lit photo of a shoe, you hand it a messy snapshot taken by a shaky hand in a busy clinic. This is the challenge of computer vision, a branch of science where we try to get machines to understand images just like humans do. Specifically, this paper dives into medical image segmentation, which is like playing a high-stakes game of "connect the dots" where the dots are pixels. The goal is to draw a perfect line around a specific object—in this case, a human foot—separating it from the background clutter of floors, clothes, and shadows. Why does this matter? Because if a computer can accurately measure a foot from a simple smartphone photo, doctors could easily track wounds, fit shoes for people with diabetes, or spot deformities without needing expensive, bulky 3D scanners. It's about bringing high-tech medical tools to the palm of your hand, even when the lighting is bad and the background is chaotic.

Now, meet FootNet, the new star player in this game. The researchers created a massive, multi-view photo album of 453 feet, taken with regular smartphones by clinic staff. These aren't perfect studio shots; they are real-world images with varied lighting and backgrounds, captured from six different angles: the top (dorsal), the side (medial), and the bottom (plantar) of both left and right feet. Every single foot in this album was carefully traced by human experts to create a "ground truth" map, showing exactly where the foot ends and the floor begins.

The team then set up a friendly (but fierce) competition between four different AI models to see which one could draw the best outline around these feet. Think of these models as four different artists with different styles:

U-Net (with a MobileNetV2 encoder): The classic, reliable artist known for preserving fine details.
UNet++: A more complex version of the first artist, trying to be extra thorough with nested pathways.
DeepLabV3 (with MobileNetV3-Large): A specialist in understanding context and scale.
SAM (Segment Anything Model): A famous, pre-trained "super-artist" that usually needs a hint (like a bounding box) to know what to draw.

The results were clear. U-Net took the crown, achieving the highest accuracy with an IoU (Intersection over Union, a score measuring how much the drawn line overlaps the real foot) of 0.9268 and a Dice score of 0.9608. In simple terms, its outline was almost perfectly aligned with the expert's drawing. The paper explicitly rules out the idea that the fancier, more complex models are always better; UNet++ did not significantly outperform the simpler DeepLabV3, suggesting that adding extra complexity didn't help in this specific task.

Even the "super-artist" SAM, when given a perfect hint (an "oracle" bounding box drawn around the foot), scored an IoU of 0.9219 on the subset of images it was tested on. While impressive, statistical tests showed that U-Net still significantly outperformed SAM (with a p-value of 0.005), proving that a model trained specifically on these foot photos is better than a general-purpose model even when the general model gets a perfect hint. The researchers also tested a "cleanup" trick called connected-component post-processing, hoping it would fix any messy bits in the drawings, but found it made almost no difference (improving the score by only 0.0003 on average), so they decided it wasn't worth using.

One interesting discovery was that the AI performed best on the plantar (bottom) views, with IoU scores around 0.95, likely because the sole of the foot creates a clear, high-contrast shape against the floor. The dorsal (top) views were trickier, especially for the right foot, where the background was more cluttered, causing more variation in the results.

In conclusion, this paper suggests that for the specific job of tracing feet from messy smartphone photos, the classic U-Net architecture is the most reliable tool, beating out both its complex cousin and the famous general-purpose AI. The dataset itself, FootNet, is now available for other researchers to use, with the original 191 images open to the public and the full set of 453 available upon request. The authors are careful to note that while the results are strong, they come from a single clinic, and the models haven't been tested on different devices or in different countries yet, so there is still work to be done before this becomes a universal medical standard.

Technical Summary: FootNet

Problem Statement
Automated whole-foot segmentation from consumer smartphone photographs is a critical preprocessing step for clinical workflows, including wound area measurement, deformity grading, and footwear fitting. However, existing public datasets primarily target wound-level segmentation (e.g., the Diabetic Foot Ulcer Challenge) rather than the entire foot. Furthermore, current whole-foot analysis often relies on controlled environments using 3D plantar scanners, thermal cameras, or depth sensors, which are unavailable in point-of-care settings. Real-world clinical photographs present significant challenges due to hand-held capture, ambient lighting, and varied, often skin-colored backgrounds, making segmentation substantially harder than in controlled environments. Prior to this work, no publicly available multi-view whole-foot dataset acquired under realistic point-of-care conditions existed.

Methodology
The authors introduce FootNet, a dataset and benchmark designed to address these gaps.

Dataset Construction: FootNet comprises 453 image-mask pairs collected from outpatient clinics using various consumer smartphones without tripods or controlled lighting. The dataset covers six anatomical views (dorsal, medial, and plantar for both left and right feet). Images were annotated by trained personnel using LabelMe to trace foot boundaries, converted to binary PNG masks, and de-identified. The dataset is split into an original public set (191 images) and an extended set (262 images) pending de-identification review.
Model Benchmarking: The study benchmarks four segmentation architectures under a controlled protocol:
1. U-Net with a MobileNetV2 encoder.
2. UNet++ with a MobileNetV2 encoder.
3. DeepLabV3 with a MobileNetV3-Large encoder.
4. Segment Anything Model (SAM) ViT-B, evaluated in a zero-shot setting using an "oracle" bounding box prompt (the tight ground-truth box padded by 20px) to establish an upper bound on prompt quality.
Training and Evaluation: Supervised models were trained on the full 453-image dataset using a combined Dice + Binary Cross-Entropy loss, AdamW optimizer, and specific augmentation strategies (flips, rotations, affine perturbations). Evaluation metrics included Intersection over Union (IoU) and Dice coefficient. Statistical significance was assessed using Bonferroni-corrected Wilcoxon signed-rank tests across six pairwise comparisons, with effect sizes reported as rank-biserial correlations.

Key Contributions

FootNet Dataset: A multi-view smartphone foot dataset with expert-annotated binary masks, available in two tiers (191 public images and 262 requestable images).
Controlled Benchmark: A comparative analysis of four distinct segmentation models (U-Net, UNet++, DeepLabV3, and SAM) on this specific domain.
Rigorous Statistical Analysis: Application of Bonferroni-corrected significance testing and bootstrap confidence intervals to validate performance differences.
Ablation Study: An evaluation of connected-component post-processing to determine its utility in refining segmentation outputs.

Results

Model Performance: U-Net with a MobileNetV2 encoder achieved the best overall performance with an IoU of 0.9268 and a Dice score of 0.9608.
- U-Net significantly outperformed DeepLabV3 (IoU 0.8984, $p < 0.001$ ) and UNet++ (IoU 0.8913, $p < 0.001$ ).
- UNet++ did not significantly differ from DeepLabV3 ( $p = 0.062$ ).
SAM Comparison: On the matched 191-image subset, U-Net (IoU 0.9274) significantly outperformed SAM ViT-B with oracle prompts (IoU 0.9219, $p = 0.005$ ). Conversely, SAM significantly outperformed DeepLabV3 on this subset ( $p < 0.001$ ), while the difference between UNet++ and SAM was not statistically significant after correction.
View-Specific Performance: Plantar views yielded the highest accuracy (e.g., Left Plantar IoU 0.9526), likely due to high-contrast silhouettes against floors. Right dorsal views showed the highest variance (std 0.102), attributed to complex backgrounds and varied patient postures.
Post-Processing: Connected-component analysis yielded negligible benefit, improving the mean IoU by only 0.0003 across the dataset.

Significance and Claims
The paper claims that U-Net with a MobileNetV2 encoder is the most robust model for this specific task, offering both higher mean accuracy and lower worst-case failure risk compared to DeepLabV3 and the prompt-based SAM. The authors attribute U-Net's success to its encoder-decoder skip connections, which preserve fine boundary details that atrous pooling (used in DeepLabV3) may discard at the 384×384 resolution used.

The study highlights that while SAM demonstrates strong zero-shot capabilities when provided with perfect prompts, it remains sensitive to within-box ambiguity and requires significantly more computational resources (375 MB checkpoint) compared to the prompt-free U-Net. The authors conclude that for downstream clinical pipelines, U-Net offers a superior balance of accuracy, efficiency, and reliability.

Limitations
The authors explicitly note several constraints: the data originates from a single clinical site with unknown inter-rater reliability; the train/validation split was performed at the image level rather than the patient level (potentially introducing data leakage); and the dataset size (453 images) is modest by deep-learning standards, limiting the generalizability of findings to transformer-based architectures or diverse acquisition conditions. No on-device inference benchmarks were conducted.

Technical Summary: FootNet

More like this