Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography

Technical Summary: Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography

Problem Statement
Vision-language pre-training (VLP) has shown promise in medical AI by leveraging radiology reports as rich textual supervision. However, existing methods face two critical limitations when applied to 3D Computed Tomography (CT) imaging:

1. Inefficient Visual Backbones: Most models repurpose backbones designed for 2D natural images. Extending Vision Transformers (ViT) to 3D CT via volumetric patchification incurs excessive computational costs and sacrifices spatial resolution, leading to the loss of critical details in small lesions. Conversely, 3D Convolutional Neural Networks (CNNs) capture local textures well but lack compatibility with mainstream VLM architectures, hindering the transfer of cross-modal priors learned during large-scale pre-training.
2. Coarse Semantic Alignment: Current approaches rely on global image-text contrastive learning, ignoring fine-grained correspondences between specific pathological findings and their textual descriptions. Even organ-level alignment methods often conflate distinct diseases (e.g., "atelectasis," "emphysema," and "lung nodules") within the same organ under a generic "abnormal" label, obscuring disease-specific associations and impairing diagnostic discrimination.

Methodology
The authors propose CT-DiagVLM, a tailored VLP framework centered on the co-design of efficient representation, fine-grained alignment, and robust inference. The framework consists of three core components:

1. CNN–ViT Hybrid Visual Encoder:

   • To balance local detail capture with global attention, the model replaces the standard ViT patch embedding module with a 3D CNN backbone (specifically a 3D ResNet-18).
   • It employs a Multi-Scale Fusion Patch Embedding (MSF-PE) mechanism. Features from four CNN stages are projected, aligned via trilinear interpolation, and fused to create rich token initializations.
   • These embeddings are fed into a ViT encoder initialized with pre-trained SigLIP-2 weights. This design preserves global attention mechanisms and cross-modal priors while efficiently capturing multi-scale local anatomical details without the computational burden of direct self-attention on high-resolution 3D volumes.
   • A U-Net-style decoder predicts organ masks, which are used to filter ViT tokens and generate organ-specific visual embeddings via cross-attention with learnable organ queries.

2. Disease-Level Contrastive Learning:

   • Moving beyond organ-level alignment, the framework introduces a mechanism to disentangle co-occurring pathologies.
   • Learnable Query Tokens: For each organ, a set of learnable query vectors represents all possible conditions (specific diseases + healthy state).
   • Dynamic Extraction: These queries perform cross-attention over the full radiology report to dynamically extract condition-specific textual semantics without requiring explicit report decomposition.
   • Contrastive Objective: The model aligns organ-level visual features with these disease-specific textual embeddings. It pulls visual features toward textual embeddings of the same disease present in the batch while pushing them away from other conditions. This allows training on both ground-truth and LLM-generated pseudo-labels.

3. Diagnosis-Aware Prompt Strategy:

   • To bridge the gap between pre-training and zero-shot inference, the authors replace hand-crafted binary prompts (e.g., "CT shows disease X") with real clinical phrases.
   • Prototype Construction: For each condition, the system retrieves diverse reports from the training set (guided by labels or pseudo-labels) and aggregates their condition-specific embeddings into a global disease prototype.
   • Inference: Diagnostic scores are computed via cosine similarity between organ-level visual embeddings and these aggregated prototypes, ensuring semantic fidelity and robustness against phrasing variations.

Key Contributions

• Hybrid Architecture: A CNN-ViT hybrid encoder that integrates multi-scale 3D CNN features into a pre-trained ViT, achieving efficiency on 3D volumes while retaining VLM compatibility.
• Fine-Grained Alignment: A disease-level contrastive learning mechanism using learnable query tokens to extract and align specific disease semantics from unstructured reports, enabling the disentanglement of multiple pathologies within a single organ.
• Robust Inference Strategy: A diagnosis-aware prompt strategy that leverages real clinical language and aggregated prototypes to enhance zero-shot diagnostic reliability and reduce the pre-training–inference gap.

Experimental Results
The model was evaluated on two public chest CT benchmarks: CT-RATE (internal test) and Rad-ChestCT (external test).

• Standard Benchmarks: CT-DiagVLM achieved a 84.4% AUC on CT-RATE, outperforming the previous state-of-the-art (fVLM) by 5.1% and ViSD-Boost by 5.4%. On the external Rad-ChestCT dataset, it achieved 75.4% AUC, a 5.4% improvement over fVLM.
• Scalability (60 Diseases): Using Qwen3-Max to generate pseudo-labels for 60 distinct conditions, the model demonstrated significant scalability, achieving an 85.6% AUC compared to a base model's 75.8% (a +9.8% gain). While sensitivity decreased slightly due to the model's conservative, noise-robust decision boundaries, specificity and AUC improved substantially.
• Downstream Tasks: The pre-trained model showed strong transferability to radiology report generation, achieving an F1-score of 45.5% on CT-RATE, surpassing existing methods.
• Ablation Studies: Experiments confirmed that each component (hybrid encoder, disease-level loss, and prompt strategy) contributes uniquely and synergistically. The hybrid encoder alone provided a +1.0% AUC gain, while the full framework delivered the total +5.1% improvement.

Significance
The paper argues that the proposed co-design strategy is essential for translating the potential of foundation models into effective medical VLP. By addressing the inefficiency of visual backbones in 3D and the semantic conflation of distinct diseases, CT-DiagVLM demonstrates that disease-centric pre-training can achieve state-of-the-art performance and strong generalization. The authors claim this work provides a promising direction toward clinically viable, general-purpose medical AI systems capable of accurate diagnosis and reasoning with the nuance of real clinical language. The approach highlights the importance of fine-grained semantic alignment and the utility of leveraging LLMs for scalable, disease-specific learning in the absence of dense manual annotations.