💻 computer science

An Attention-Enhanced MobileViT Framework for Multi-Task Skin Lesion Classification and Segmentation

This paper proposes an attention-enhanced, lightweight MobileViT-XS framework that effectively balances computational efficiency and diagnostic reliability for joint skin lesion classification and segmentation, achieving competitive performance on the ISIC 2019 dataset with significantly fewer parameters than heavyweight models.

Original authors: Yanqiang Ge, Bingqian Lu, Haoyu Wang, Zixu Yang, Zhan Zhang

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yanqiang Ge, Bingqian Lu, Haoyu Wang, Zixu Yang, Zhan Zhang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every year, millions of people worldwide develop skin lesions, ranging from harmless moles to dangerous cancers like melanoma. Catching these conditions early is a matter of life and death; when melanoma is found before it spreads, the five-year survival rate is nearly perfect, but once it reaches distant organs, that chance drops drastically. Currently, the first line of defense is often a general doctor in a local clinic, not a specialist dermatologist. These primary-care providers face a difficult reality: they must spot subtle signs of cancer in skin images without the years of specialized training a dermatologist possesses, and they often lack the expensive, heavy computer systems that power the most advanced diagnostic tools. The challenge, then, is to build a digital assistant that is smart enough to see what a specialist sees, yet light enough to run on the modest computers found in a standard clinic or even a mobile device.

A team of researchers has addressed this challenge by creating a new type of artificial intelligence system designed specifically for this environment. They built a framework that can look at a picture of a skin lesion and do two things at once: decide what kind of lesion it is and draw a precise outline around it. The core of their system is a lightweight neural network, a type of computer program inspired by the human brain, which they chose for its ability to balance high performance with low energy use. To make this system even sharper, they added a mechanism that helps the computer focus on the most important parts of the image, much like a human eye naturally ignores a busy background to focus on a specific object. They also designed the system to be cautious; if the computer is unsure about a diagnosis, it is programmed to flag the case for a human doctor rather than guessing confidently, a safety feature intended to prevent dangerous errors.

The researchers tested their creation using a large collection of skin images from a public medical database. The results showed that their lightweight system could correctly identify the type of skin lesion in more than 84 percent of cases, a level of accuracy that rivals much larger and more complex models that require powerful, expensive hardware. In addition to classifying the lesion, the system successfully outlined the boundaries of the growth in the image with a high degree of precision. Perhaps most importantly, the study demonstrated that teaching the computer to perform both tasks simultaneously—identifying the lesion and drawing its shape—actually made it better at identifying the lesion than if it had been trained to do only one of those jobs. This suggests that the two tasks help each other, with the act of outlining the lesion helping the computer understand its features more deeply.

Despite these successes, the researchers are careful to note that their system is not yet ready to replace a doctor. The model was trained on images taken under controlled, ideal conditions, and it may struggle with the messy, varied lighting and angles found in real-world clinics or home self-checks. Furthermore, the system is designed to be a screening tool, not a final diagnostic authority. When the computer encounters a difficult or ambiguous image, its built-in safety protocol directs it to send the case to a human expert for review. The team envisions this technology as a way to empower general practitioners, giving them a reliable second opinion that can help catch dangerous skin cancers earlier, even in settings where specialist resources are scarce. Before this tool can be used in hospitals, however, it must undergo further testing in real clinical environments to ensure it works safely across different skin types and medical situations.

Technical Summary: An Attention-Enhanced MobileViT Framework for Multi-Task Skin Lesion Classification and Segmentation

1. Problem Statement

The paper addresses the challenge of deploying automated skin lesion diagnosis in resource-constrained primary-care settings. While deep learning has advanced skin lesion analysis, a significant trade-off exists between diagnostic accuracy and computational efficiency.

Heavyweight models (e.g., standard Transformers, large U-Nets) offer strong performance but possess prohibitive parameter overheads, making them unsuitable for mobile or edge devices.
Existing lightweight networks often lack robust global feature modeling and spatial feature enhancement, limiting their reliability for fine-grained tasks like distinguishing ambiguous lesion boundaries.
Safety concerns: Most current models provide deterministic predictions without explicitly quantifying uncertainty, potentially leading to overconfident outputs on ambiguous or atypical samples, which increases clinical misdiagnosis risks.

2. Methodology

The authors propose a lightweight, attention-augmented multi-task framework based on MobileViT-XS for joint skin lesion classification and semantic segmentation.

Backbone Architecture: The framework utilizes MobileViT-XS as a shared backbone. This architecture combines Convolutional Neural Networks (CNNs) for local texture extraction with Vision Transformers (ViTs) for global dependency modeling, effectively capturing the multi-scale and boundary-ambiguous characteristics of dermoscopic lesions.
Attention Mechanism (CBAM): A Convolutional Block Attention Module (CBAM) is integrated at the end of the backbone, prior to the task-specific heads. CBAM sequentially applies channel attention (identifying informative channels via global pooling and MLP) and spatial attention (aggregating channel information to generate a spatial mask) to refine feature representations.
Multi-Task Learning (MTL) Design: The architecture features a dual-branch structure:
- Classification Branch: Outputs probabilities for eight lesion categories (AK, BCC, BKL, DF, MEL, NV, SCC, VASC) using global average pooling, dropout, and a fully connected layer.
- Segmentation Branch: Outputs a pixel-level lesion mask using a lightweight decoder that upsamples features and applies convolutions.
- Asymmetric Promotion: The design leverages pixel-level segmentation supervision to assist image-level classification optimization, creating an "asymmetric task promotion paradigm" where segmentation aids classification more significantly than vice versa.
Loss Function: The total loss is a weighted sum of classification and segmentation losses ( $L_{total} = \lambda_{cls}L_{cls} + \lambda_{seg}L_{seg}$ $L_{t o t a l} = λ_{c l s} L_{c l s} + λ_{se g} L_{se g}$ ).
- Classification Loss: A weighted cross-entropy loss is used to address class imbalance, with weights inversely proportional to class frequency.
- Segmentation Loss: A combination of Dice loss and Binary Cross-Entropy (BCE) loss is used to optimize region overlap and pixel-wise accuracy.
Conservative Inference & Risk Stratification: To mitigate overconfidence, a safety-first inference strategy is employed.
- Review Thresholds: Samples are flagged for expert review if the maximum class probability ( $p_{max}$ ) is below 0.50 or the margin between the top two classes ( $\delta$ ) is below 0.15.
- Risk Stratification: For non-reviewed samples, the cumulative probability of malignant classes (MEL, BCC, SCC, AK) is calculated. Risks are stratified as High (predicted malignant), Medium (benign prediction but $p_{malignant} \ge 0.25$ ), or Low.

3. Key Contributions

Validation of MobileViT-XS: The study systematically validates MobileViT-XS as a superior lightweight backbone for skin lesion analysis, demonstrating it balances local texture and global context better than pure CNNs (e.g., EfficientNet-B0) or heavy Transformers.
Attention-Enhanced Multi-Task Paradigm: The work establishes a collaborative learning framework where segmentation supervision significantly refines spatial features for classification. It demonstrates that the CBAM module yields substantial performance gains specifically when coupled with multi-task learning, overcoming the limited attention gains observed in single-task frameworks.
Safety-Oriented Prediction: The introduction of a conservative prediction constraint and risk stratification mechanism addresses the issue of model overconfidence, flagging uncertain samples for human review to enhance diagnostic safety.

4. Experimental Results

Experiments were conducted on the ISIC 2019 dataset (25,331 images, 8 classes) with segmentation masks matched from ISIC 2018.

Performance Metrics:
- Classification: The proposed model achieved 84.37% Accuracy, 79.46% Macro-F1, 79.68% Sensitivity, and 97.22% Specificity.
- Segmentation: The model achieved a Dice coefficient of 86.37% and mIoU of 78.09%.
- Efficiency: The model operates with only 2.62 Million parameters.
Comparative Analysis:
- The proposed method outperformed lightweight baselines like EfficientNet-B0 (81.63% Accuracy) and significantly surpassed ResNet-50 and GS-TransUNet.
- While single-task segmentation models (U-Net, DeepLabV3+) achieved slightly higher Dice/mIoU scores, the proposed multi-task model offers a superior accuracy-efficiency trade-off by performing both tasks simultaneously with minimal parameters.
Ablation Studies:
- Adding CBAM to a single-task MobileViT-XS baseline yielded marginal gains.
- Introducing Multi-Task Learning (MTL) significantly boosted classification accuracy (from ~72% to ~83.6%).
- Combining MTL with CBAM achieved the optimal results, confirming that dense segmentation supervision activates the full potential of the attention mechanism.

5. Significance and Claims

The paper claims to provide a well-balanced trade-off among accuracy, computational efficiency, visual interpretability, and clinical prediction safety.

Primary Care Applicability: The framework is positioned as a deployable, lightweight solution for primary-care skin lesion screening and portable edge devices, where computational resources are limited.
Interpretability: The integration of Grad-CAM heatmaps and overlay visualizations enhances model interpretability, allowing clinicians to understand the reasoning behind predictions.
Safety: By explicitly handling uncertainty through conservative thresholds, the system aims to reduce the risk of hazardous overconfident predictions in ambiguous cases.

Limitations and Future Outlook:
The authors acknowledge that the model was trained exclusively on public ISIC datasets under controlled imaging conditions, which may lead to domain shifts in real-world primary-care or self-screening scenarios. The current loss weighting is fixed, and the prediction thresholds are empirically defined. The paper concludes that prospective multi-center clinical validation is required before real-world clinical application can be realized. Future work will focus on collecting diverse clinical datasets, exploring dynamic task-balancing mechanisms, and implementing mobile/edge deployment via ONNX or TFLite.