⚡ electrical engineering

Entropy-Guided Self-Supervised Learning for Medical Image Classification

This paper proposes a synergistic deep learning framework that combines ImageNet pre-trained and entropy-guided Masked Autoencoder pre-trained ConvNeXt-Tiny models into an ensemble, achieving state-of-the-art performance in medical image classification by effectively addressing challenges like limited annotated data and high intra-class variability across diverse datasets.

Original authors: Joao Florindo, Viviane Moura

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Joao Florindo, Viviane Moura

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach two different students how to identify specific diseases from medical images, like X-rays or ultrasound scans. The challenge is that medical images are tricky: there aren't many of them, they look very similar to each other, and the differences between a "sick" and "healthy" image can be incredibly subtle.

This paper proposes a clever way to train these students so they become experts, using a method called Entropy-Guided Self-Supervised Learning. Here is how it works, broken down into simple concepts:

The Two Students (The Two Models)

Instead of relying on just one student, the authors use two different "ConvNeXt-Tiny" models (think of these as two very smart, specialized students). They train them in two completely different ways to give them unique strengths:

The "World Traveler" (ImageNet Pre-training):
- How they learn: This student was first trained on a massive library of everyday photos (like cats, cars, and trees) from a dataset called ImageNet.
- Their strength: They are great at recognizing general shapes, edges, and basic patterns. They have a broad, general knowledge of how the world looks.
- The analogy: Think of this student as a general practitioner who has seen thousands of different types of photos and knows what a "blob" or a "line" usually looks like.
The "Medical Detective" (Entropy-Guided MAE Pre-training):
- How they learn: This student was trained only on medical images, but without any labels (they didn't know which images were sick or healthy). They used a technique called Masked Autoencoder (MAE).
- The Twist (Entropy-Guided): Usually, MAE works by hiding parts of an image and asking the computer to guess what's missing. This paper adds a special rule: it calculates an "entropy score" (a measure of confusion or information) for every tiny piece of the image.
  - If a part of the image is boring and uniform (low entropy), the computer adds a little bit of "noise" (static) to it.
  - If a part of the image is complex, detailed, or confusing (high entropy)—like a subtle tumor or a tricky lesion—the computer adds more noise to it.
- The Goal: By forcing the student to reconstruct the most confusing and detailed parts of the image, they are forced to pay extra attention to the most important, information-rich areas.
- The analogy: Imagine a detective looking at a blurry crime scene photo. Instead of just guessing the whole picture, the detective is forced to focus specifically on the blurry, chaotic parts where the clues are hidden. This "Entropy-Guided" method forces the model to become an expert at spotting the subtle, critical details that matter in medicine.

The Team-Up (Ensemble Learning)

Once both students are trained, they are fine-tuned on the specific medical task (like identifying breast cancer or skin lesions).

The Strategy: Instead of asking just one student for the answer, the system asks both of them.
The Decision: It takes the probability scores (how confident each student is) from both the "World Traveler" and the "Medical Detective" and simply averages them.
Why it works: The "World Traveler" brings general stability, while the "Medical Detective" brings deep, specific knowledge of medical quirks. Together, they cover each other's weaknesses. It's like having a generalist and a specialist vote on a diagnosis; the final decision is usually much more accurate than either one alone.

The Results

The authors tested this team-up on four different medical datasets (breast ultrasounds, skin images, endoscopy images, and chest X-rays for COVID).

The Outcome: The combined team consistently beat the best individual models and other top-tier methods currently available.
Key Finding: The "Medical Detective" (the entropy-guided model) was particularly good at learning from the specific medical data, while the "World Traveler" provided a solid foundation. When combined, they achieved record-breaking accuracy, especially on the breast ultrasound dataset.

In Summary

This paper doesn't just say "let's use AI." It says, "Let's train one AI to be a general expert and another AI to be a specialist who focuses specifically on the most confusing parts of medical images. Then, let's have them vote together." This combination creates a system that is more accurate and reliable than using just one type of training, helping doctors make better diagnoses.

Technical Summary: Entropy-Guided Self-Supervised Learning for Medical Image Classification

Problem Statement
Accurate medical image classification is critical for early disease diagnosis and treatment planning. However, deep learning models often struggle in this domain due to limited annotated data, high intra-class variability, subtle inter-class differences, and the presence of noise and artifacts. While Convolutional Neural Networks (CNNs) excel at feature extraction, models trained from scratch or fine-tuned solely on ImageNet often fail to capture domain-specific nuances. Conversely, Vision Transformers (ViTs) require vast datasets and high computational resources, which are often unavailable for medical imaging. The paper addresses the need for a robust framework that leverages both generalizable features and domain-specific representations to overcome data scarcity and improve classification performance.

Methodology
The authors propose a synergistic deep learning framework that integrates transfer learning, self-supervised learning (SSL), and ensemble learning. The architecture utilizes two parallel pathways, both employing a ConvNeXt-Tiny backbone:

ImageNet Pre-trained Pathway: A ConvNeXt-Tiny model initialized with weights pre-trained on the large-scale ImageNet dataset. This serves as a baseline to provide strong, generalizable feature extractors.
Entropy-Guided MAE Pre-trained Pathway: A second ConvNeXt-Tiny model undergoes self-supervised pre-training on the target medical dataset using a novel Entropy-Guided Masked Autoencoder (MAE).
- Mechanism: Inspired by "diffusion forcing," this approach calculates an entropy score for each image patch based on the frequency of color values. Gaussian noise is then applied to each patch independently, with the variance ( $\sigma^2$ ) controlled by the patch's Shannon entropy.
- Objective: This transforms the masking process into a continuous transform, focusing the reconstruction task on regions with higher information content or uncertainty. The model minimizes the Mean Squared Error (MSE) between the original and reconstructed masked patches.
Fine-Tuning: Both models are subsequently fine-tuned on labeled medical image datasets using Cross-Entropy loss.
Ensemble Strategy: The final classification decision is made by averaging the predicted probabilities from the two fine-tuned models. This ensemble approach aims to leverage the complementary strengths of general features (ImageNet) and domain-specific features (Entropy-Guided MAE).

Key Contributions

Entropy-Guided MAE: The introduction of a pre-training strategy that uses entropy scores to guide the masking and noise injection process, prioritizing informative regions during self-supervised learning on medical images.
Synergistic Framework: A novel architecture combining ImageNet transfer learning, entropy-guided SSL, and ensemble learning to enhance robustness and accuracy in medical image classification.
Extensive Validation: Comprehensive experiments conducted on four diverse datasets: Breast Ultrasound Images (BUSI), International Skin Imaging Collaboration (ISIC) 2018, Kvasir (endoscopy), and a COVID-19 chest X-ray dataset.
Ablation Analysis: A detailed study evaluating the impact of pre-training epochs and fine-tuning configurations on model performance.

Experimental Results
The framework was evaluated using Accuracy, Precision, Recall, F1-score, and Area Under the Curve (AUC). The optimal configuration identified was 1,000 epochs of MAE pre-training followed by 15 epochs of fine-tuning.

Performance: The ensemble model consistently outperformed individual models (ImageNet-only and MAE-only) and state-of-the-art (SOTA) methods, including SWIN, CMT, SMT, and the Central-Peripheral Vision Transformer (CPVT).
Specific Metrics:
- BUSI: Achieved 97.44% accuracy and 99.74% AUC, significantly surpassing CPVT (81.95% accuracy).
- ISIC 2018: Achieved 91.18% accuracy and 98.32% AUC, outperforming CPVT (87.98% accuracy).
- Kvasir: Achieved 94.30% accuracy and 99.66% AUC.
- COVID: Achieved 97.82% accuracy and 99.88% AUC.
Observations: The results suggest that extended self-supervised pre-training (1,000 epochs) effectively constructs domain-specific structural abstractions, helping the model handle high intra-class variance and complex tissue boundaries.

Significance and Claims
The paper claims that the proposed hybrid approach constitutes a highly reliable framework for generalized biomedical image classification. By combining generative mask reconstruction with broad natural image priors, the method prevents optimization collapses in data-constrained fields. The authors assert that the ensemble strategy successfully captures complementary information, leading to state-of-the-art results across diverse medical modalities. The work highlights the efficacy of entropy-guided masking in focusing on informative regions, thereby improving the discriminative power of features learned from unlabeled medical data. Future work is planned to explore more sophisticated fusion techniques (e.g., weighted averaging) and to investigate the interpretability of the learned features.

The Two Students (The Two Models)

The Team-Up (Ensemble Learning)

The Results

In Summary

More like this