💻 computer science

Multimodal Deep Learning for Diabetic Foot Ulcer Staging Using Integrated RGB and Thermal Imaging

This study demonstrates that a multimodal deep learning approach combining RGB and thermal imaging, captured via a portable Raspberry Pi system, significantly outperforms single-modal methods in classifying diabetic foot ulcer stages, achieving a peak accuracy of 93.25% with the VGG16 model by leveraging complementary structural and thermal information.

Original authors: Gulengul Mermer, Mustafa Furkan Aksu, Gozde Ozsezer, Sevki Cetinkalp, Orhan Er, Mehmet Kemal Gullu

Published 2026-03-31

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Gulengul Mermer, Mustafa Furkan Aksu, Gozde Ozsezer, Sevki Cetinkalp, Orhan Er, Mehmet Kemal Gullu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🩺 The Problem: The Silent Threat on Your Feet

Imagine you have a chronic condition like diabetes. It's like having a "glitch" in your body's sugar management system. One of the scariest side effects of this glitch is Diabetic Foot Ulcers (DFUs). These are open sores on the feet that are hard to heal.

If these sores aren't caught early, they can get infected, lead to amputation, and ruin a person's quality of life. The current way to manage this is like a weekly check-up: you go to the doctor, they look at your foot, and you try to check your own feet at home. But this has problems:

The "Bad Angle" Problem: When you try to take a picture of your own foot with a phone, it's often blurry, too dark, or at the wrong angle.
The "Invisible Heat" Problem: Sometimes, a foot looks fine on the outside, but it's actually inflamed and hot underneath. A regular camera can't see this "heat."

🔍 The Solution: A "Smart Mirror" for Your Feet

The researchers in this paper wanted to build a better system. They created a portable device (built on a small computer called a Raspberry Pi) that acts like a high-tech smart mirror for your feet.

Think of this device as a super-spy camera that uses two different lenses at the exact same time:

The "Eagle Eye" (RGB Camera): This is a standard high-definition camera. It sees colors, textures, and open wounds just like your eyes do.
The "Heat Vision" (Thermal Camera): This is like the goggles a superhero wears to see heat. It detects temperature changes. If a part of your foot is infected, it will glow "hot" on this camera, even if the skin looks normal to the naked eye.

🧠 The Brain: Teaching the Computer to "See" Better

The researchers didn't just build the camera; they taught a computer (using Artificial Intelligence) how to interpret what it sees. They wanted to answer a simple question: "Is it better to look at the foot with just one eye, or with two?"

They trained the computer on three different ways of looking at the data:

Team Color: The computer only looked at the standard photos.
Team Heat: The computer only looked at the thermal (heat) images.
Team Super-Vision: The computer looked at both images combined, like a human having both normal vision and night-vision goggles active simultaneously.

🏆 The Results: Why Two Eyes Are Better Than One

The results were clear, like a race where the two-eyed team won by a landslide.

The "Heat Only" Team: This team was good at spotting big, angry infections (because those get very hot). But they struggled with early stages. Imagine trying to find a small spark in a dark room using only a heat sensor; if the spark is tiny, the sensor might miss it.
The "Color Only" Team: This team was great at seeing the shape of a wound. But sometimes, they got confused by shadows or skin color variations, missing the fact that the tissue underneath was actually inflamed.
The "Super-Vision" Team (RGB + Thermal): This was the winner. By combining the two, the computer could see the shape of the wound (from the color camera) and the inflammation (from the heat camera) at the same time.

The Analogy:
Imagine you are trying to identify a fruit in a basket.

If you only look at the color, you might mistake a red apple for a red ball.
If you only feel the temperature, you might not know if it's an apple or a rock.
But if you look and feel at the same time, you know instantly: "It's red, round, and cool to the touch. It's an apple!"

The study found that the "Super-Vision" AI was the most accurate at grading the severity of the ulcers (from Stage 0 to Stage 5). It was especially good at catching the tricky early stages where the other methods failed.

🔦 How the AI "Thinks" (The Secret Sauce)

The researchers used a special tool called Grad-CAM to see where the AI was looking.

When the AI only had heat, it got confused and looked at random spots on the foot.
When the AI only had color, it sometimes looked at the whole foot instead of just the sore.
When the AI had both, it zoomed in perfectly on the sore, ignoring the rest of the foot. It knew exactly where the danger was.

🚀 What This Means for the Future

This isn't just a lab experiment. The researchers built a working prototype that is small, cheap, and easy to use.

The Big Picture:
Imagine a future where you can stand in front of a "Smart Mirror" at home. You step on it, and it instantly takes a photo and a heat scan of your feet. It tells you, "Hey, your left foot looks a little hot and red in this spot. You should check it out before it becomes a big sore."

This technology could:

Save Limbs: Catch problems before they get bad enough to require amputation.
Save Money: Reduce the need for expensive hospital visits and long-term care.
Empower Patients: Let people monitor their own feet easily and accurately without needing a doctor to hold the camera.

⚠️ The Catch (Limitations)

The researchers are honest about what they still need to fix:

The Dataset: They only tested this on a specific group of people. They need to test it on more people from different places to make sure it works for everyone.
The Camera: The "Heat Vision" camera they used is a bit low-resolution (like an old TV) because high-res thermal cameras are very expensive. They hope to upgrade this in the future to make the "heat" picture even sharper.

💡 The Bottom Line

This paper proves that combining sight and heat is the secret weapon for fighting diabetic foot ulcers. By giving the computer "super-vision," we can detect foot problems earlier, treat them better, and keep people walking safely for longer.

1. Problem Statement

Diabetic Foot Ulcers (DFUs) are a severe complication of Diabetes Mellitus, leading to high rates of recurrence, hospitalization, and non-traumatic amputations. Current management relies on regular clinical examinations and patient self-monitoring, which often suffer from:

Logistical and economic burdens associated with frequent clinic visits.
Inconsistency in home monitoring: Smartphone-based "selfie" methods lack standardized capture geometry (distance, angle, lighting), leading to high variability and measurement noise.
Limitations of single-modality monitoring: While thermal monitoring can detect inflammation (temperature anomalies), it often lacks the textural and structural detail required for early-stage diagnosis. Conversely, RGB imaging captures morphology but may miss sub-surface inflammatory changes.
Lack of integrated home solutions: Existing smart mirror technologies focus on general health or depression, with no dedicated systems designed specifically for DFU staging using multimodal data.

The study aims to address these gaps by developing a multimodal deep learning framework that integrates RGB and thermal imaging to improve the accuracy and stability of DFU staging (Wagner classification).

2. Methodology

A. Data Acquisition System

The authors developed a custom, portable image acquisition system based on the Raspberry Pi 4B to ensure standardized data collection:

Hardware: Integrated a Raspberry Pi Camera Module V2 (8MP RGB) and a FLIR Lepton 3.5 Thermal Camera (160x120 pixels).
Design: A 3D-printed casing holds both sensors in a side-by-side configuration with overlapping Fields of View (FoV).
Interface: A 7-inch touchscreen GUI allows simultaneous triggering of both cameras.
Output: Captures RGB images (JPEG), color-mapped thermal images (JPEG), and raw thermal data matrices (TIFF).

B. Dataset Creation

Source: Clinical data collected from a hospital setting with ethical approval.
Size: 1,205 distinct samples.
Classes: Annotated by clinical experts into 6 Wagner stages (Grade 0 to Grade 5).
Modalities: Three distinct training sets were created:
1. RGB-only: Standard color images.
2. Thermal-only: Processed temperature maps.
3. RGB+Thermal (Multimodal): Thermal data integrated as a 4th channel alongside the standard RGB channels.

C. Pre-processing

Thermal Conversion: Raw 14-bit Kelvin data was converted to Celsius using the formula: $T(^\circ C) = \frac{\text{Pixel Value}}{100} - 273.15$ .
Adaptive Filtering: To handle ambient temperature fluctuations, a dynamic windowing approach was used. If the mean temperature was within 30°C–45°C, it was normalized; otherwise, thresholds were iteratively adjusted to preserve thermal contrast and segment the foot region.
Augmentation: Geometric transformations (cropping, flipping, rotation) and photometric adjustments (color jitter) were applied to mitigate overfitting.
Class Imbalance: A class-weighted loss function was implemented to penalize minority classes more heavily during backpropagation.

D. Model Architecture & Training

Backbones: Five pre-trained CNN architectures (ImageNet weights) were utilized: ResNet-50, VGG16, DenseNet-121, InceptionV3, and EfficientNetV2.
Custom Head: The original classification layers were replaced with a custom head consisting of two fully connected layers (1024 and 512 neurons), followed by ReLU, Batch Normalization, and 50% Dropout.
Training Strategy: 5-fold stratified cross-validation with a separate 15% test set.
Evaluation Metrics: Accuracy, Precision, Recall, F1-Score, Specificity, Sensitivity, and Matthews Correlation Coefficient (MCC).

3. Key Contributions

Hardware Prototype: Development of a low-cost, Raspberry Pi-based device capable of synchronized RGB and thermal capture, addressing the lack of standardized home monitoring tools.
Multimodal Integration Strategy: Successfully demonstrated a method to fuse thermal data as a 4th input channel, allowing CNNs to learn complementary features from both modalities simultaneously.
Comprehensive Comparative Analysis: Provided a rigorous comparison of five different CNN architectures across three data modalities (RGB, Thermal, and Multimodal), identifying the specific strengths and weaknesses of each.
Explainability (XAI): Utilized Grad-CAM to visualize model attention, proving that the multimodal approach focuses more accurately on clinically relevant regions (ulcers and inflammation) compared to single-modality models.

4. Results

Performance Comparison

The RGB+Thermal (Multimodal) approach consistently outperformed single-modality approaches across all architectures.

Best Model: VGG16 trained on the RGB+Thermal dataset achieved the highest performance:
- Overall Accuracy: 93.25%
- F1-Score: 92.53%
- MCC: 91.03%
Comparison:
- RGB-only VGG16: 90.22% Accuracy.
- Thermal-only VGG16: 90.96% Accuracy.
- Note: While Thermal-only performed well, the Multimodal approach provided the most robust and stable results, particularly improving the MCC metric.

Class-Wise Performance

Advanced Stages (Grade 2+): The model achieved high accuracy due to pronounced textural distortion and significant thermal anomalies.
Early Stages (Grade 0-1): These were the most challenging due to the lack of visible ulcers and minimal thermal contrast. However, the multimodal model reduced ambiguity by using RGB texture to compensate for weak thermal signals.
Confusion Analysis: Misclassifications were primarily between adjacent classes (e.g., Grade 0 $\to$ Grade 1), indicating the model follows clinical progression logic rather than random error.

Visual Analysis (Grad-CAM)

Thermal-only: Struggled to localize regions in early stages, producing diffuse heat maps.
RGB-only: Often focused on broad foot areas, sometimes missing specific lesion boundaries.
Multimodal: Produced compact, well-defined activation maps tightly aligned with the lesion, effectively combining the structural focus of RGB with the inflammatory detection of thermal imaging.

5. Significance and Future Work

Clinical Impact: The study validates that multimodal deep learning can significantly enhance DFU staging accuracy, potentially reducing amputation rates by enabling earlier, more accurate detection of recurrence and infection.
Telehealth Application: The proposed system serves as a foundation for home-based monitoring and remote clinical decision support, reducing the burden on healthcare systems.
Limitations:
- Dataset Size: Relatively small (1,205 samples) and single-center.
- Resolution: The thermal sensor (160x120) has lower resolution than the RGB sensor, limiting fine-grained thermal detail.
- Alignment: Slight misalignment between RGB and thermal optical axes prevents perfect pixel-level registration.
Future Directions:
- Collecting larger, multi-center datasets to improve generalizability.
- Integrating higher-resolution thermal sensors.
- Implementing automatic image registration techniques to perfectly align the two modalities.

In conclusion, this paper demonstrates that integrating thermal imaging with standard RGB photography creates a superior diagnostic tool for diabetic foot ulcer staging, offering a robust, explainable, and clinically viable solution for next-generation telehealth monitoring.