🤖 AI

Differential Attention-Augmented BiomedCLIP with Asymmetric Focal Optimization for Imbalanced Multi-Label Video Capsule Endoscopy Classification

This paper proposes a Differential Attention-Augmented BiomedCLIP framework enhanced with asymmetric focal optimization and temporal smoothing strategies to address extreme class imbalance in multi-label video capsule endoscopy classification, achieving competitive performance on the RARE-VISION test set with efficient inference.

Original authors: Podakanti Satyajith Chary, Nagarajan Ganapathy

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Podakanti Satyajith Chary, Nagarajan Ganapathy

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to find a single, tiny, red thread hidden inside a massive, tangled ball of white yarn. That is essentially what doctors face when they review Video Capsule Endoscopy (VCE) footage.

A patient swallows a pill-sized camera that takes photos of their entire digestive tract. In just one exam, the camera snaps 50,000 to 130,000 pictures. A human doctor has to look at every single one to find rare problems like bleeding or ulcers, which might only appear in a handful of frames. It's like searching for a needle in a haystack, but the haystack is moving, and the needle is invisible to the naked eye half the time.

This paper describes how a team called MINDH Lab built a super-smart AI assistant to help doctors do this job faster and better. Here is how they did it, explained simply:

1. The Brain: A "Medical Expert" with a New Way of Looking

The team started with a pre-trained AI called BiomedCLIP. Think of this AI as a medical student who has already read millions of medical textbooks and looked at millions of medical images. It already knows what a healthy stomach looks like.

However, the standard way this AI "looks" at an image is a bit noisy. It gets distracted by the background, like a student trying to read a book in a noisy cafeteria.

The Fix: They replaced the AI's standard "attention" mechanism with something called Differential Attention.
The Analogy: Imagine you are trying to hear a whisper in a loud room. Standard attention is like trying to listen to everything at once. Differential Attention is like wearing noise-canceling headphones that subtract the background noise, leaving only the whisper. By comparing two slightly different "views" of the image and subtracting the common parts, the AI cancels out the "visual noise" and focuses intensely on the tiny, subtle changes that indicate a disease.

2. The Problem: The "Needle in the Haystack" Issue

The biggest challenge is that the data is extremely unbalanced.

The Situation: Out of 500,000 frames, maybe 99.9% show normal, healthy tissue. Only 0.1% show a problem (like bleeding).
The Trap: If you train a standard AI on this, it becomes lazy. It learns that if it just guesses "Healthy" for every single picture, it will be right 99.9% of the time. It becomes a "Yes-Man" that never finds the problems.
The Solution: The team used a multi-pronged strategy to force the AI to pay attention to the rare problems:
- The "Rare Candy" Sampler: Instead of feeding the AI random pictures, they made sure it saw the rare, sick pictures much more often during training (like giving the student extra practice problems on the hard topics).
- The "Strict Teacher" Loss Function: They used a special math formula (Asymmetric Focal Loss) that punishes the AI heavily if it misses a rare disease, but doesn't care as much if it misses a common "healthy" frame. It forces the AI to stop being lazy.
- Mixing the Data: They used a technique called "Mixup," which blends two images together (like mixing red and blue paint to make purple) to teach the AI to be more flexible and less likely to memorize specific pictures.

3. The "Time Travel" Post-Processing

The AI looks at one frame at a time, but diseases don't just appear and disappear instantly; they persist.

The Problem: The AI might get confused and say "Bleeding!" for one frame, then "No bleeding" for the next, then "Bleeding!" again. This looks like a glitchy video.
The Fix: After the AI makes its guesses, they run a "smoothing" filter.
The Analogy: Imagine the AI is a shaky hand drawing a line. The post-processing is like a steady hand going over the drawing, filling in the gaps, and connecting the dots. If the AI sees a "bleeding" sign for a few frames, the system merges them into one solid event. If it sees a single "bleeding" frame surrounded by "healthy" frames, it assumes it was a mistake and erases it.

4. The Results: Fast and Accurate

The team tested their system on a massive set of real-world video data (over 160,000 frames).

Speed: The whole process took about 8.6 minutes on a single powerful computer chip. That's incredibly fast compared to a human doctor who might spend hours reviewing the same footage.
Accuracy: While the AI still missed some very rare, tiny problems (because there simply weren't enough examples to learn from), it successfully identified the major events and localized them in time very precisely. The difference between finding an event "roughly" and "exactly" was very small, meaning the AI knows exactly when the problem starts and stops.

Summary

In short, MINDH Lab took a smart medical AI, gave it noise-canceling headphones to see subtle details, forced it to study the rare cases harder than the common ones, and then smoothed out its answers so they make sense as a continuous video.

It's not a perfect cure-all yet (some tiny problems are still too hard to find), but it turns a 10-hour manual search into an 8-minute automated scan, acting as a tireless, hyper-focused second pair of eyes for doctors.

1. Problem Statement

The paper addresses the critical challenge of multi-label classification in Video Capsule Endoscopy (VCE) using the Galar dataset. The core difficulties are:

Extreme Class Imbalance: Pathological findings (e.g., active bleeding, angiectasia, ulcers) constitute less than 0.1% of annotated frames, while anatomical regions (stomach, small intestine, colon) dominate. Some classes have positive-to-negative ratios exceeding 1:3,000.
Data Volume: A single VCE exam generates 50,000–130,000 frames, making manual review by gastroenterologists time-consuming and prone to fatigue.
Subtle Visual Features: Pathologies often manifest as subtle mucosal changes that are visually indistinguishable from healthy tissue at low magnification, leading to high false-negative rates in standard models.
Standard Model Failure: Conventional cross-entropy optimization leads models to trivially predict the majority class, achieving high accuracy but failing to detect rare, clinically significant events.

2. Methodology

The authors propose a four-stage pipeline centered around a modified BiomedCLIP foundation model.

A. Backbone: Differential BiomedCLIP

Foundation: Uses BiomedCLIP (ViT-B/16 image encoder + PubMedBERT text encoder), pre-trained on 15 million biomedical image-text pairs.
Differential Attention Mechanism: Replaces standard multi-head self-attention in all 12 transformer blocks.
- The $h$ attention heads are split into two groups ( $h/2$ ).
- Two softmax attention maps are computed ( $A_1$ and $A_2$ ).
- The output is the difference: $DiffAttn = (A_1 - \lambda \cdot A_2)V$ .
- Goal: Subtracting the second map cancels "common-mode" noise (uniform or low-rank patterns), amplifying signals from rare, diagnostically meaningful spatial regions.
Weight Transfer: Pretrained weights are fully transferred, with QKV projections decomposed to accommodate the differential structure while preserving biomedical knowledge.

B. Classification Architecture

The model utilizes a dual-path approach:

Classification Path: Image features pass through an Excitation Block (Squeeze-and-Excitation inspired) with channel-wise gating, followed by a linear layer projecting to 17 class logits.
Contrastive Path: Normalized image features are matched against frozen text embeddings (generated from clinical prompts like "a video capsule endoscopy image showing [label]") to enforce semantic alignment.

C. Handling Class Imbalance (Multi-Level Strategy)

To combat the skewed distribution, the authors employ a coordinated strategy:

Sampling: A sqrt-frequency weighted sampler assigns weights proportional to $1/\sqrt{f_c}$ , providing moderate oversampling of rare classes without the instability of linear inverse weighting.
Loss Function: Asymmetric Focal Loss ( $\gamma_+ = 1, \gamma_- = 4$ ) aggressively down-weights easy negative predictions while preserving gradients for positive examples.
Thresholding: Per-class threshold optimization on the validation set (ranging from 0.05 for rare classes to higher values for common ones) to maximize class-specific F1 scores.
Regularization: Mixup ( $\alpha=0.3$ ), label smoothing ( $\epsilon=0.05$ ), and strong geometric/photometric augmentations. An Exponential Moving Average (EMA) of weights ( $\beta=0.999$ ) is used for inference.

D. Temporal Post-Processing

To convert frame-level predictions into clinically relevant event-level JSON:

Binarization: Apply per-class optimized thresholds.
Smoothing: Apply a median filter (window size $w=7$ ) to suppress isolated false positives.
Gap Merging: Fill gaps of $\le 5$ frames between same-class events to reflect the continuous nature of capsule transit.
Filtering: Remove events shorter than 3 frames.

3. Key Contributions

Differential Attention for Biomedical Vision: Successfully adapted differential attention to suppress noise and enhance the detection of subtle mucosal pathologies in VCE data.
Holistic Imbalance Strategy: Demonstrated that combining sampling, asymmetric loss, augmentation, and threshold calibration is more effective than single-level fixes for extreme imbalance.
Temporal Coherence: Implemented a robust post-processing pipeline that aligns predictions with the physical continuity of VCE data, improving event localization.
Efficiency: Achieved high-performance inference (8.6 minutes for ~161k frames) on a single GPU using a foundation model.

4. Results

Evaluated on the ICPR RARE-VISION Challenge 2026 held-out test set (3 NaviCam examinations, 161,025 frames):

Overall Performance:
- mAP@0.5: 0.2456
- mAP@0.95: 0.2353
- The narrow gap (0.0103) between mAP@0.5 and mAP@0.95 indicates highly precise temporal boundary localization.
Per-Class Breakdown:
- Anatomical Regions: High performance (AP > 0.84) for stomach, small intestine, and colon due to sufficient data.
- Rare Pathologies: Extremely low performance (AP < 0.03) for classes like active bleeding, erythema, and hematin, highlighting the fundamental difficulty of learning from <0.05% of the data.
Training Dynamics: The model showed controlled divergence between training and validation loss (factor of ~2.3x), unlike preliminary experiments without regularization which diverged by >15x.
Inference Speed: Completed inference on all test videos in ~8.6 minutes on a single NVIDIA RTX PRO 6000 GPU.

5. Significance and Limitations

Significance: The work provides a state-of-the-art framework for automated VCE analysis, demonstrating that foundation models can be effectively adapted for extreme long-tail medical problems through architectural and optimization innovations. The temporal post-processing ensures the output is clinically actionable.
Limitations:
- Rare Class Failure: Despite advanced techniques, classes with fewer than 200 training frames (e.g., active bleeding) remain undetectable (AP < 0.01).
- Visual Ambiguity: Pathologies like angiectasia and erythema are visually indistinguishable from normal tissue variations, limiting model performance regardless of algorithmic improvements.
- Domain Shift: Performance variance across test videos suggests issues with domain shift between training data (Olympus/PillCam) and test data (NaviCam).
Future Work: The authors suggest incorporating temporal modeling (RNNs or sequence attention) to leverage sequential context, as pathological events often persist and evolve over time, which frame-level classifiers miss.