Technical Summary: MedPMC

Problem Statement

The development of medical foundation models is currently constrained by a scarcity of large-scale, high-quality, publicly accessible multimodal data. While clinical data is essential, it is often fragmented, restricted by privacy and licensing, and costly to annotate at scale. Consequently, many medical AI systems are developed within single-modality, single-task settings, leading to a "benchmark-to-bedside" gap where models perform well on public benchmarks but fail to generalize to real-world clinical heterogeneity.

Although PubMed Central (PMC) offers a vast repository of expert-authored image-text data, existing resources derived from it suffer from three critical limitations:

1. Low Fidelity: A significant portion of visual content in biomedical literature consists of non-medical elements (e.g., graphs, charts, molecular schematics) that are not useful for learning clinically relevant representations.
2. Structural Misalignment: Biomedical figures are frequently compound, multi-panel images. Existing datasets often treat these as single units or fail to accurately decompose them into subfigures and align them with corresponding subcaptions, degrading the image-text correspondence required for effective contrastive learning.
3. Static Nature: Most existing datasets are released as static snapshots, failing to continuously incorporate the rapidly expanding body of biomedical literature, leading to dataset staleness.

Methodology

The authors introduce MedPMC, an automated, systematic, and modular framework designed to transform permissively licensed biomedical literature into high-fidelity medical multimodal data. The framework operates through five sequential stages, each handled by a purpose-built model:

1. Initial Screening: A text classifier (PubMedBERT) uses figure captions and associated inline reference text to distinguish candidate medical figures from non-medical content (e.g., graphs, diagrams) before downloading images. This stage filters 6.1 million PMC articles down to 3.7 million candidate pairs.
2. Multi-panel Figure Detection: An image classifier (Vision Transformer) identifies whether a retained image is a single-panel or a multi-panel figure.
3. Multi-panel Figure Separation: A detection model (YOLOv10) decomposes multi-panel figures into individual subfigures, generating bounding boxes for panel-level crops.
4. Caption Separation and Alignment: A supervised Multimodal Large Language Model (InternVL-2.5-4B) jointly performs caption decomposition and subfigure-subcaption alignment. It takes the original multi-panel figure, extracted subfigures, and the full caption as input to generate a list of subcaptions corresponding to the input subfigures. This joint approach avoids the error propagation of sequential task execution.
5. Medical Figure Classification: A final image classifier filters the decomposed subfigures to remove any remaining non-medical visual elements (e.g., charts within a multi-panel figure).

Training Data Augmentation: To address the lack of public benchmarks for specific pipeline stages (particularly initial screening and caption alignment), the authors generated synthetic annotations from sampled PMC figures, supplementing existing labeled datasets (ImageCLEF, MedICaT, DocFigure) to train the component models.

Model Training:

• MedPMC-CLIP: A CLIP-style vision-language model trained via contrastive learning on the curated 11 million image-text pairs. The architecture and training protocol were kept identical to the BMC-CLIP baseline to isolate the effect of data quality.
• Multimodal LLM Integration: The MedPMC-CLIP vision encoder was used to initialize the LLaVA-Med framework (replacing the original vision encoder) to test transferability to generative multimodal reasoning tasks.

Key Contributions

1. MedPMC Framework: A reusable, continuously updatable infrastructure for curating medical multimodal data, moving beyond static dataset releases to a modular pipeline that can be systematically evaluated and refined.
2. Curated Corpus: A dataset of approximately 11 million medical image-text pairs derived from 6.1 million PMC articles (as of June 2024). This includes 3.1 million multi-panel figure pairs, 0.6 million single-panel pairs, and 7.3 million subfigure-subcaption pairs.
3. Component Benchmarks: A comprehensive benchmark suite constructed from repurposed existing data, synthetic annotations, and newly annotated human samples to evaluate each stage of the curation pipeline independently.
4. Pretrained Models: Publicly released checkpoints for MedPMC-CLIP and the integrated LLaVA-Med variant.

Results

Pipeline Performance

Component-level evaluations demonstrated strong performance across the curation stages:

• Initial Screening: F1 = 93.2 (substantially outperforming keyword-matching baselines).
• Multi-panel Detection: F1 = 96.5.
• Figure Separation: mAP = 89.8.
• Caption Separation & Alignment: F1 = 81.4; ROUGE-L = 85.3 (outperforming rule-based and GPT-4 zero-shot approaches).
• Medical Figure Classification: F1 = 96.5.

Data Quality

Manual review by five annotators (three with medical training) found that 95.3% of curated MedPMC images were medically relevant. In contrast, a prior large-scale PMC-derived dataset (BIOMEDICA) contained only 19.7% medically relevant images (80.3% non-medical). The MedPMC corpus covers 11 specialties, including radiology, pathology, dermatology, and endoscopy, with diverse modalities such as slit-lamp photography and fMRI.

Foundation Model Evaluation

• Zero-Shot Image Classification: MedPMC-CLIP was evaluated on 26 benchmarks across 11 specialties. It improved the average zero-shot AUC by 7.1 percentage points over the strong BMC-CLIP baseline, despite being trained on fewer than half the number of image-text pairs. It outperformed the baseline in 10 of 11 specialties.
• Multimodal Question Answering (QA): When used as the vision encoder in LLaVA-Med, MedPMC-CLIP improved performance by 1.9 percentage points on MMMU (Medical) and 16.9 percentage points on OmniMedVQA. Gains were particularly pronounced in tasks requiring visual recognition (modality recognition, anatomy identification).
• Clinical Transfer (Dermatology): In a morphology-to-image retrieval task using 10,524 internal patient images from Yale New Haven Health System, MedPMC-CLIP improved Recall@5 by 11.7 percentage points over BMC-CLIP.
• Embedding Space Analysis: Analysis in the DINOv2 embedding space showed that MedPMC dermatology images had the lowest nearest-neighbor cosine distance to real-world clinical images compared to other public dermatology datasets (Fitzpatrick17k, SCIN, DermNet), indicating better alignment with the visual distribution of clinical practice.

Significance and Claims

The paper claims that high-fidelity literature curation is a scalable and effective strategy for strengthening medical multimodal foundation models. The key significance lies in demonstrating that:

1. Data Quality over Quantity: Transforming raw literature into panel-level aligned, medically relevant data yields superior visual representations compared to larger, noisier datasets, even when training on fewer samples.
2. Infrastructure as a Solution: Treating data curation as a modular, reproducible engineering problem allows for continuous updates and systematic improvement, addressing the issue of dataset staleness.
3. Complementary Role: MedPMC is positioned not as a replacement for clinical data, but as a complementary resource. It provides scale, broad coverage across specialties and modalities, and rich descriptive context, serving as an effective pretraining backbone that can be subsequently adapted to specific clinical settings where local data is available.

The authors emphasize that these findings bridge the gap between literature-derived data and clinical utility, offering a path toward more generalizable and reproducible medical AI systems.