CapCLIP: A Vision-Language Representation Alignment Approach for Wireless Capsule Endoscopy Analysis

Technical Summary: CapCLIP – A Vision-Language Representation Alignment Approach for Wireless Capsule Endoscopy Analysis

Problem Statement
Wireless Capsule Endoscopy (WCE) enables non-invasive visualization of the small bowel but faces significant clinical constraints due to the massive volume of frames generated per examination and the difficulty of identifying subtle abnormalities under highly variable imaging conditions. Existing machine learning approaches for WCE are predominantly vision-only, often restricted to narrow pathology sets, and demonstrate limited generalizability across different datasets and clinical centers. These models typically rely on fixed class labels, resulting in outputs that lack semantic interpretability and struggle with distribution shifts (e.g., varying pathology prevalence, acquisition devices, or image appearance). Furthermore, there is a lack of domain-specific multimodal frameworks for WCE, as naturally paired image-text data (clinical reports linked to specific frames) are unavailable at scale, and existing medical vision-language models are often trained on radiology or other modalities with different characteristics.

Methodology
The paper proposes CapCLIP, a domain-specific vision-language representation learning framework designed to align WCE frames with clinically grounded textual descriptions. The methodology addresses the absence of natural image-text pairs through a structured caption generation pipeline:

1. Caption Generation: Frame-level labels are transformed into rich textual descriptions using predefined caption templates, domain-specific nomenclature (derived from Delphi-consensus studies), and Large Language Models (LLMs) guided by a contextual knowledge base of medical literature. These captions encode visual characteristics, pathology semantics, and clinical relevance (e.g., linking "Erythema" to "vascular lesion" or "Abnormal" to "Crohn's disease").
2. Architecture: The framework adopts a CLIP-style architecture with two encoders:
   • Image Encoder: A Vision Transformer (ViT), evaluated in two variants (ViT-B/16 and ViT-L/14), initialized with OpenAI CLIP weights.
   • Text Encoder: A 12-layer Transformer (similar to the original CLIP setup) using Byte-Pair Encoding (BPE).
3. Training Objective: The model is trained using a symmetric cross-entropy loss to align image and text embeddings within a shared hyperspherical space. The training utilizes mini-batches where matched image-text pairs are encouraged to have high cosine similarity, while unmatched pairs are pushed apart.
4. Variants: Three training modes are explored:
   • CapCLIP-B: Binary classification (Normal vs. Abnormal).
   • CapCLIP-M: Multiclass classification (specific pathology labels).
   • CapCLIP-MIX: A hybrid approach combining both binary and multiclass pairings.

Experimental Setup and Evaluation
The framework is evaluated under strict zero-shot conditions using unseen, out-of-distribution WCE datasets (CrohnIPI, KID2, Galar) that were not used during training. The training data consists of the SEE-AI and Kvasir-Capsule datasets. The evaluation compares CapCLIP against several baselines:

• Vision-only Foundation Models: EndoSSL, EndoFM, EndoFM-LV.
• General/Medical Vision-Language Models: CLIP (OpenAI), MedCLIP, BioMedCLIP, MedImageInsight.

Performance is assessed across three downstream tasks:

1. Zero-shot K-Nearest Neighbour (KNN) Classification: Probing the semantic structure of visual embeddings.
2. Zero-shot CLIP-style Image-Text Classification: Evaluating cross-modal discriminative capability using text prompts.
3. Zero-shot Text-to-Image Retrieval: Assessing the ability to retrieve relevant frames based on textual queries (both coarse "abnormality" level and fine-grained "pathology" level).

Key Results

• KNN Classification: CapCLIP consistently outperformed baselines on the KID2 and CrohnIPI datasets, with gains of up to +11.2% in weighted F1-score for multiclass settings. On the Galar dataset, performance was comparable to baselines, attributed to the dataset's high redundancy which makes KNN probing less discriminative.
• Image-Text Classification: CapCLIP demonstrated substantial improvements over all baselines. In binary classification, it achieved AUROC/AUPRC improvements of up to +54.5% over the best baseline. In multiclass classification, gains were even more pronounced, with improvements of up to +206.7% in weighted F1-score on the CrohnIPI dataset.
• Text-to-Image Retrieval: CapCLIP significantly improved retrieval performance. For coarse abnormality retrieval, it improved mAP by up to +94.2% on CrohnIPI. For fine-grained pathology retrieval, it achieved gains of up to +260.8% in weighted Precision@200 on the Galar dataset.
• Embedding Visualization: t-SNE visualizations confirmed that CapCLIP learns embeddings with clearer cluster structures and better inter-class separability compared to both vision-only and existing medical vision-language baselines.

Significance and Claims
The paper claims that CapCLIP represents a significant step toward foundation models tailored specifically for capsule endoscopy. The primary significance lies in demonstrating that language-guided representation learning can yield embeddings that are more transferable and semantically informative than those produced by existing general-domain or medical-domain baselines.

Key contributions include:

1. The development of a systematic zero-shot benchmarking framework for evaluating multimodal models on WCE data.
2. The introduction of CapCLIP, which enables language-grounded representation learning for a modality lacking natural image-text pairs.
3. Evidence that aligning images with structured, pathology-aware captions improves generalization to unseen datasets and enhances the semantic interpretability of WCE analysis.

The authors conclude that while current vision-only approaches are limited by fixed taxonomies, the proposed vision-language approach offers a more flexible path for clinical decision support, allowing physicians to query WCE data using natural language concepts rather than being restricted to predefined labels. The paper acknowledges limitations, including the reliance on generated rather than naturally paired captions and the relatively small scale of WCE training data compared to general foundation models, suggesting future work should focus on larger datasets and unified training strategies.