Technical Summary: MEDICALNARRATIVES

Problem

Multi-modal models require vast amounts of data for effective representation learning, yet medical image datasets lack the scale and diversity found in natural image repositories. Existing medical vision-language (VL) models often struggle with spatial localization and semantic understanding simultaneously. While some recent datasets attempt to provide spatial annotations, they frequently rely on sub-optimally pre-trained models to generate text descriptions and Regions of Interest (ROIs) for images lacking ground truth, potentially propagating model biases and errors. Furthermore, standard medical datasets often lack the "localized narratives" where text is explicitly grounded to specific image regions through human interaction, a protocol crucial for training models to connect spatial image regions with clinical concepts.

Methodology

The authors propose MEDICALNARRATIVES, a large-scale dataset curated from unstructured pedagogical medical videos and scientific articles, designed to bridge the gap between medical vision and language with localized, spatially grounded narratives.

Data Curation Pipeline:

1. Sourcing: The dataset aggregates data from two primary sources:
   • YouTube: Over 738,000 videos were searched using domain-specific keywords and a "video-first" strategy to identify educational content.
   • PubMed: 5.4 million articles were processed to extract figures and captions.
2. Filtering and Extraction:
   • Video Filtering: Videos were filtered based on duration, presence of speech, and the percentage of keyframes identified as medical images using domain-specific classifiers. Narrative-style videos were identified by detecting "narrative streaks" (periods where the narrator pauses to analyze static frames) or persistent narration over dynamic content.
   • Text Processing: An open-source ASR model (Whisper) transcribed speech, followed by LLM-based correction of medical terminology and extraction of medically relevant text and Regions of Interest (ROI) descriptions.
   • Image Processing: Keyframes were extracted, denoised, and for compound figures in articles, sub-figures were separated using object detection (YOLO) and paired with sub-captions using OCR and LLMs.
3. Localization (Traces): A core innovation is the extraction of mouse cursor traces from videos. The authors modified the Quilt-LLaVA approach to identify "stable chunks" (frames with minimal background movement) and isolated cursor movements by subtracting a median background frame. This creates a temporal association between the spoken text and specific pixels, mimicking the "think-aloud" protocol used by medical experts.
4. Interleaving: The dataset includes interleaved samples where a single narrative connects multiple modalities (e.g., X-ray and CT for the same patient) or multiple sub-images within a single figure, enhancing cross-modal learning.

Dataset Statistics:

• Total Size: 4.7 million image-text pairs across 12 medical domains (11 medical + 1 pseudo-medical).
• Localized Narratives: 1 million samples contain dense spatial annotations in the form of cursor traces (101.8 million unique trace points), which can be reformatted into bounding boxes or masks.
• Video Component: 118,000 videos centered on trace events, preserving temporal alignment between narration and pointing.

Key Contributions

1. MEDICALNARRATIVES Dataset: A novel dataset of 4.7M image-text pairs, with 1M samples featuring expert-validated spatial traces, enabling spatiotemporal grounding beyond single frames.
2. Scalable Curation Protocol: A robust pipeline for extracting grounded vision-language data from noisy, unstructured sources (YouTube videos) and structured sources (scientific articles), balancing collection ease with training utility.
3. GENMEDCLIP: A vision-language model trained on MEDICALNARRATIVES using a CLIP-like objective. The model was evaluated on a new benchmark spanning 11 medical modalities.
4. Trace-to-Segmentation Utility: Demonstration that the extracted traces can serve as sparse supervision (points/scribbles) for interactive medical image segmentation (IMIS) models like ScribblePrompt.

Results

The authors evaluated GENMEDCLIP on a constructed benchmark covering 29 downstream datasets across 11 medical domains.

• Zero-Shot Classification: GENMEDCLIP outperformed state-of-the-art models (including BIOMEDCLIP, PUBMEDCLIP, and PMC-CLIP) across the benchmark. Specifically, it achieved an average improvement of 4.75% over BIOMEDCLIP.
• Linear Probing: In supervised settings, GENMEDCLIP outperformed baselines by an average of 2.8% over BIOMEDCLIP and 1.9% over PUBMEDCLIP.
• Cross-Modal Retrieval: The model demonstrated superior performance in zero-shot image-to-text and text-to-image retrieval, outperforming BIOMEDCLIP by 26.07% on average.
• Ablation Studies: Experiments comparing models trained on articles only versus articles plus videos showed that adding video-derived data increased classification performance by 11.65% and retrieval performance by 53.6%, highlighting the value of dynamic, trace-aligned data.

Significance

The paper claims that MEDICALNARRATIVES addresses the critical scarcity of large-scale, grounded multimodal medical datasets. By leveraging the "think-aloud" protocol inherent in medical pedagogy videos, the dataset provides a unique form of weak supervision where text is temporally and spatially aligned with image regions via cursor traces.

The authors argue that this protocol can serve as a de facto standard for annotating future multimodal medical datasets due to its flexibility and cost-effectiveness compared to manual dense annotation. The results suggest that training on this diverse, trace-grounded data enables models to achieve state-of-the-art performance in both classification and retrieval tasks, even with fewer training samples (4.7M) compared to prior works (e.g., BIOMEDCLIP trained on 15M+ samples). The dataset is positioned to catalyze progress in spatially-controllable report generation, interactive segmentation, and generative medical imaging models.

Limitations Acknowledged:

• The dataset lacks human-annotated bounding boxes, limiting direct overlap assessment for dense tasks like semantic segmentation.
• There is an overrepresentation of abnormal cases, reflecting clinical suspicion biases, which may impact model generalizability.
• While traces are shown to be useful for IMIS, the paper does not train models specifically on traces for downstream spatial tasks, leaving this for future work.