💻 computer science

A multifractal-based masked auto-encoder: an application to medical images

This paper proposes Multifractal-Optimized Masked Autoencoder (MO-MAE), a novel framework that enhances medical image classification by using Renyi entropy to guide the masking strategy toward diagnostically critical, high-complexity regions, thereby achieving superior performance with minimal computational overhead compared to existing models.

Original authors: Joao Batista Florindo, Viviane de Moura

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Joao Batista Florindo, Viviane de Moura

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a student how to recognize different types of fruit by showing them pictures.

The Old Way (Traditional AI):
Usually, when we teach an AI to understand medical images (like X-rays or skin scans), we use a method called a "Masked Autoencoder." Think of this like a game where you cover up random parts of the fruit picture with a black square and ask the student to guess what's underneath.

The Problem: If you cover up a random spot on a banana, you might hide a tiny, crucial brown spot that tells you it's overripe. But if you cover up a random spot on a plain white background, it doesn't matter much. Traditional AI treats every part of the image the same, so it might waste time learning about empty space while missing the tiny, important details that actually diagnose a disease.

The New Way (The Paper's Solution):
The authors of this paper, Joao Batista Florindo and Viviane de Moura, came up with a smarter way to play this game. They call their new system MO-MAE.

Instead of covering up random spots, they use a special mathematical tool called Multifractal Analysis (specifically something called Renyi entropy) to act like a "complexity detector."

The Analogy: The "Busy" vs. "Quiet" Neighborhoods
Imagine the medical image is a city map.

Some areas are quiet suburbs with empty fields (these are the unimportant parts of an X-ray, like the empty space around the lungs).
Other areas are bustling city centers with crowded streets, tall buildings, and complex traffic patterns (these are the complex tissue structures where diseases hide).

The old AI would cover up random blocks of the map, sometimes covering the quiet suburbs and sometimes the city center.
The MO-MAE system looks at the map first and says, "Hey, this city center is very complex and full of information! Let's cover that up and force the student to figure it out."

How It Works (Step-by-Step):

Cutting the Pie: The AI cuts the medical image into many small square pieces (patches).
The Complexity Check: It uses the "complexity detector" to measure how much "information" or "texture" is in each piece. High complexity means the piece is likely important (like a tumor or a specific tissue pattern).
Smart Masking: It deliberately hides the most complex pieces.
The Reconstruction: The AI has to look at the remaining visible pieces and try to "paint back" the hidden, complex parts. Because it's forced to solve the hardest puzzles first, it learns to understand the most critical details of the image much better.
The Result: When the AI is finally tested on real patients, it is much better at spotting diseases because it spent its training time focusing on the "bustling city centers" of the image, not the empty fields.

What They Found:
The researchers tested this new method on two main things:

MedMNIST: A huge collection of 708,000 different medical images (like skin scans, eye scans, and blood cells).
COVID-CT: A set of CT scans to detect COVID-19.

The Verdict:
The new "Smart Masking" system (MO-MAE) did a better job than many other top-tier AI models. It was especially good at handling difficult images where the details are subtle.

It didn't need to be a giant, slow computer program to work; it was efficient.
It beat several other famous AI models (like ResNet and MedVIT) in accuracy.
It worked well even when there wasn't a massive amount of labeled data to train on, which is a common problem in medicine.

In Short:
This paper introduces a way to make AI "pay attention" to the most complicated and important parts of a medical image by hiding those specific parts during training. By forcing the AI to reconstruct the most difficult sections, it learns to become a sharper, more accurate diagnostician without needing extra complex hardware or massive amounts of data.

Technical Summary: A Multifactor-Based Masked Auto-Encoder for Medical Images

Problem Statement
Traditional Masked Autoencoders (MAEs) utilize a random masking strategy during self-supervised pre-training. While effective in general computer vision, this approach is suboptimal for medical imaging. In medical contexts, subtle changes in specific regions often hold critical diagnostic value, whereas random masking may overlook these high-information areas or focus on less relevant background noise. Existing improvements to MAE masking often rely on learnable auxiliary networks or supervised attention maps, which can introduce computational overhead or require external training procedures.

Methodology: MO-MAE
The authors propose the Multifractal-Optimized Masked Autoencoder (MO-MAE), a framework that replaces random masking with a physics-based complexity measure to guide patch selection. The methodology proceeds as follows:

Patching: Input images are partitioned into non-overlapping rectangular patches (e.g., $16 \times 16$ ).
Multifractal Analysis (The Selector): Instead of random selection, the model computes the Rényi entropy (a multifractal measure) for each patch. This measure quantifies the complexity and information content of the patch at different scales.
- The partition function method is used to estimate the probability distribution of pixel intensities within a patch.
- The $q$ -th moment partition function is calculated, leading to the estimation of the multifractal spectrum and the associated Rényi entropy ( $R_q$ ).
- Patches are sorted in descending order based on their entropy values.
Masking Strategy: The model selects the top $n_S$ patches (where $n_S$ is determined by the mask ratio $r$ ) with the highest entropy. These patches represent regions of high complexity and information richness.
Pre-training: The selected high-entropy patches are fed into a Vision Transformer (ViT) encoder. The decoder attempts to reconstruct the original image from this partial input. The loss function measures the discrepancy between the original and reconstructed images.
Fine-tuning: The pre-trained encoder is then fine-tuned on labeled medical data for specific downstream tasks (e.g., disease classification).

Key Contributions
The paper highlights four primary innovations:

Novel Masking Strategy: The development of a multifractal-based masking strategy that improves medical image classification results compared to standard random masking.
First Integration: To the authors' knowledge, this is the first work to associate multifractal analysis (specifically Rényi entropy) with masked auto-encoders.
Physics-Based Complexity: This is the first exploration of a physics-based complexity measure to guide MAE masking, distinguishing it from strategies that rely on learnable parameters or external training algorithms.
Interpretability and Efficiency: The approach offers an interpretable masking criterion and adds minimal computational overhead, as the entropy calculation is straightforward and does not require learning extra parameters during pre-training.

Experimental Results
The MO-MAE was evaluated on two primary benchmarks:

MedMNIST-V2: A collection of 2D biomedical images covering various modalities (pathology, X-ray, dermatoscopy, etc.).
- Performance: MO-MAE demonstrated a general increase in both Accuracy (ACC) and Area Under the Curve (AUC) compared to the original MAE with random masking.
- Comparison: The model achieved the highest AUC/ACC in most datasets, outperforming state-of-the-art models like MedVIT-S and MedVIT-L in several scenarios, despite MedVIT being a computationally intensive architecture combining CNNs and Transformers.
- Hyperparameter Sensitivity: Experiments indicated that $q=2$ provided the best compromise between short and long-range fractality for patch selection.
COVID-CT: A dataset for predicting COVID-19 cases from CT scans.
- Performance: MO-MAE achieved an Accuracy of 0.85 and an F1 Score of 0.85.
- Comparison: The model significantly outperformed other literature methods, including DenseNet-169 and ResGANet-101, which possess a much larger number of learnable parameters.

Significance and Claims
The authors claim that the MO-MAE framework offers a promising direction for improving the accuracy and efficiency of deep learning models in medical image analysis. By forcing the model to focus on reconstructing the most complex and information-rich regions of an image, the approach enhances the model's ability to capture diagnostically relevant features.

The paper emphasizes that this self-supervised paradigm is particularly valuable in medical scenarios where labeled data is limited. Furthermore, the authors suggest that the method is generalizable, potentially applicable to other domains beyond medicine and to related tasks such as image segmentation, though these applications are noted as areas for future investigation rather than current results. The study concludes that using multifractal analysis to guide masking is a viable and effective strategy that avoids the need for complex learnable masking mechanisms.

Technical Summary: A Multifactor-Based Masked Auto-Encoder for Medical Images

More like this