💻 computer science

PolypVision: A Three-Stage Hierarchical Deep Learning Framework for Classification and Segmentation of Colorectal Polyps

This paper introduces PolypVision, a device-independent, three-stage hierarchical deep learning framework that integrates EfficientNetV2 and UNet++ architectures to achieve state-of-the-art performance in the classification, segmentation, and subtype analysis of colorectal polyps across multiple public datasets.

Original authors: Hamidreza Bolhasani, Hamidreza Rastad, Amir Mohammad Akbari, Mohammad Tashakoripour, Parnian Asadollahi, Ata Khodami, Mojgan Forootan

Published 2026-08-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hamidreza Bolhasani, Hamidreza Rastad, Amir Mohammad Akbari, Mohammad Tashakoripour, Parnian Asadollahi, Ata Khodami, Mojgan Forootan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Detective's Toolkit: A Journey into the Gut

Imagine your body as a vast, winding city, and inside that city, the colon is a long, dark tunnel. Sometimes, tiny bumps called polyps grow on the tunnel walls. Most of the time, these bumps are harmless, like harmless weeds in a garden. But some are sneaky troublemakers that can turn into something much more dangerous, like a weed that turns into a thorny vine. The big challenge for doctors is spotting these bumps early and figuring out which ones are the "weeds" and which are the "vines" before they cause trouble.

To do this, doctors use a special camera on a long, flexible tube (an endoscope) to take pictures inside the tunnel. But looking at thousands of these pictures is hard work, and even expert doctors can sometimes disagree on what they are seeing. This is where computer science steps in with a tool called "deep learning." Think of deep learning as a super-smart student that has studied millions of pictures. It learns to recognize patterns—like the shape of a bump or the texture of the skin—much faster than a human can. The goal isn't to replace the doctor, but to give them a high-tech assistant that never gets tired and can spot the tiny details that might be missed.

Meet PolypVision: The Three-Step Detective

In this paper, a team of researchers introduces a new computer program called PolypVision. You can think of PolypVision as a three-step detective squad designed to solve the mystery of colorectal polyps. Instead of just looking at a picture and guessing, this squad works in a specific order, passing clues from one detective to the next to get the whole story.

Step 1: The Gatekeeper (Is it a problem?)
The first detective, built on a smart brain called EfficientNetV2-M, looks at an image and asks a simple question: "Is this a harmless bump (hyperplastic) or a risky one (adenomatous)?" But this detective is extra sharp. While deciding, it also describes the bump's shape and its tiny surface patterns (known as Paris and JNet classifications). To make sure it doesn't get confused by rare cases, it uses a special trick called Focal Loss, which is like telling the computer, "Pay extra attention to the tricky, rare bumps!"

Step 2: The Outliner (Where does it end?)
Once the first detective says, "Yes, this is a risky bump," the second detective takes over. This one uses a tool called UNet++ to draw a perfect outline around the bump, pixel by pixel. Imagine tracing a shape on a map with a laser pointer; that's what this step does. It figures out exactly where the bump starts and stops. But it doesn't stop there! Based on the size and shape it just traced, this detective also suggests the best way to remove it—whether to use a simple cold snare, a standard removal, or a more advanced technique. It learns from the first detective's notes, so it already knows what kind of bump it is looking at.

Step 3: The Sub-Classifier (What kind of troublemaker is it?)
If the bump is confirmed to be risky, the third detective steps in. This one zooms in to sort the risky bumps into three specific families: tubular, tubulovillous, or villous. Think of these like different breeds of dogs; they all look like dogs, but some are more likely to cause trouble than others. This detective uses the knowledge gained from the previous steps to make a very precise guess. To help it learn better, the team used a technique called MixUp, which is like blending two different pictures together to create a new training example, helping the computer understand that bumps can look slightly different but still be the same type.

What Did They Find?

The researchers tested PolypVision on three different sets of pictures from the public world (datasets named PolypGen, Kvasir-SEG, and CVC-ClinicDB). The results were impressive.

Super Accuracy: When it came to classifying the frames, PolypVision achieved an AUC of approximately 0.99. In the world of computer testing, this is like getting a 99% score on a very hard exam. It performed just as well as, or better than, the best methods currently available.
Great at Finding Bumps: On the Kvasir-SEG dataset, it found and correctly identified polyps with a score of 94.4% mAP@50. This means it was very good at spotting the bumps and drawing the right boxes around them.
It Knows What It's Looking At: The team used a tool called Grad-CAM to see where the computer was "looking" when it made its decisions. The heatmaps showed that the computer was focusing on the actual bumps and their textures, not just random parts of the image. However, they also noted that when the computer made a mistake, it sometimes looked at the wrong area, which gives them a clue on how to improve it later.

Why This Matters

The most exciting part of PolypVision is that it doesn't need a special camera to work. It is device-independent, meaning it can look at pictures from almost any standard endoscope, whether it's an old model or a new one, without needing to be re-tuned for specific hardware.

The researchers have even made this tool available to everyone as a free web application at https://polypvision.com. They suggest that this three-step, "hierarchical" approach—where each step builds on the last—is a strong way to handle the complex job of analyzing polyps. While the system is very promising, the authors are careful to note that it still needs to be tested in real-world hospitals with new patients to prove it works perfectly in every situation. For now, it stands as a powerful, free, and highly accurate assistant that could help doctors catch trouble early, right from the comfort of their screens.

Technical Summary: PolypVision

Problem Statement
Colorectal cancer (CRC) remains a leading cause of cancer-related mortality, typically originating from precancerous polyps. While early detection and removal of these polyps can prevent malignant progression, accurate endoscopic characterization is challenging due to significant inter-observer variability among gastroenterologists. Existing deep learning approaches often treat polyp detection, segmentation, and classification as independent tasks, failing to leverage the complementary information inherent in these related clinical processes. Furthermore, many models require hardware-specific adaptation or struggle with limited labeled data in gastrointestinal pathology.

Methodology
The authors propose PolypVision, a three-stage hierarchical deep learning framework designed to sequentially perform classification, segmentation, and subtype analysis using a unified, transfer-learning-driven pipeline. The system operates on standard endoscopic images without hardware-specific adaptation.

Backbone Architecture: The framework utilizes EfficientNetV2-M as a shared backbone across all stages. Pre-trained on ImageNet-21k, it provides rich hierarchical representations.
Stage 1: Binary and Morphological Classification:
- Task: Classifies polyps as either adenomatous or hyperplastic. Simultaneously, it predicts Paris classification (lesion morphology) and JNet classification (pit and vascular patterns).
- Architecture: EfficientNetV2-M backbone with Global Average Pooling, Dropout, and task-specific linear heads.
- Loss Function: Focal Loss ( $\gamma=2, \alpha=0.25$ ) is employed to address class imbalance.
- Output: The trained backbone serves as the encoder initialization for Stage 2.
Stage 2: Segmentation and Resection Planning:
- Task: Performs pixel-level polyp segmentation and predicts the recommended resection method (cold snare polypectomy, EMR, or ESD).
- Architecture: A UNet++ decoder with nested skip connections, utilizing the Stage 1 EfficientNetV2-M backbone as the encoder.
- Loss Function: A combination of Dice Loss and Binary Cross-Entropy (BCE) for segmentation, plus cross-entropy for resection prediction.
- Output: The encoder is transferred to Stage 3.
Stage 3: Adenoma Subtype Classification:
- Task: Classifies adenomatous polyps (filtered from Stage 1) into three histological subtypes: tubular, tubulovillous, and villous.
- Architecture: EfficientNetV2-M initialized from the Stage 2 encoder.
- Loss Function: Focal Loss for class imbalance, combined with MixUp data augmentation applied specifically at this stage to improve generalization between visually similar subtypes.

Key Contributions

Hierarchical Pipeline: A novel three-stage framework that couples binary classification, pixel-level segmentation, and adenoma subtype classification through shared representations and progressive transfer learning.
Progressive Transfer Learning: The architecture reuses learned representations from Stage 1 to initialize Stage 2, and from Stage 2 to Stage 3, enabling efficient learning from limited labeled data.
Task-Specific Optimization: Implementation of Focal Loss for classification stages and Dice+BCE for segmentation, alongside MixUp augmentation for fine-grained subtype differentiation.
Device Independence: The system is designed to operate across diverse endoscopic imaging systems without requiring calibration to specific hardware or proprietary imaging modes.
Clinical Utility Extension: Beyond binary typing, the model simultaneously predicts Paris/JNet classifications and recommends specific resection techniques based on lesion boundaries and morphology.
Open Access: The framework is deployed as a freely accessible web application (https://polypvision.com) with a free usage tier.

Experimental Results
PolypVision was evaluated on three public datasets: PolypGen, Kvasir-SEG, and CVC-ClinicDB.

Classification Performance: The model achieved an AUC of approximately 0.99 for frame classification on both Kvasir-SEG and CVC-ClinicDB, matching or exceeding state-of-the-art methods (e.g., ResNet-50, VGG16, InceptionV3).
Detection/Segmentation Performance: On the Kvasir-SEG dataset, PolypVision achieved a detection mAP@50 of 94.4%, outperforming or matching other methods including YOLOv8-s (91.16%) and CRH-YOLO (90.7%), and approaching YOLO-LAN (96.19%).
Interpretability: Grad-CAM analysis confirmed that the model focuses on clinically relevant features such as surface texture, mucosal pit patterns, and lesion margins. Misclassified cases showed dispersed attention, providing feedback for refinement.

Significance and Claims
The paper claims that PolypVision demonstrates that a hierarchical, transfer-learning-driven pipeline with task-specific loss functions offers a robust, device-independent, and clinically meaningful approach to automated colorectal polyp analysis. By mirroring the clinical workflow—from identifying neoplastic lesions and characterizing morphology to assessing boundaries, determining resection strategy, and subclassifying histology—the framework provides a comprehensive assessment within a single unified system. The authors emphasize that while the system shows state-of-the-art performance, prospective validation on external datasets from diverse clinical centers is required before full clinical deployment. The work is presented as a step toward reducing reliance on subjective judgment in endoscopy through accessible, automated tools.

The Detective's Toolkit: A Journey into the Gut

Meet PolypVision: The Three-Step Detective

What Did They Find?

Why This Matters

Technical Summary: PolypVision

More like this