Revisiting 2D Foundation Models for Scalable 3D Medical Image Classification

1. Problem Statement

The paper addresses the challenge of scalability in 3D medical image classification. Traditional approaches require training separate, task-specific 3D models from scratch for every new clinical application. This paradigm suffers from:

• Data Scarcity: 3D models often overfit on small, per-task datasets.
• Inefficiency: Training a new model for every task is computationally expensive and slow to deploy.
• Three Critical Pitfalls in Current Research:
  1. Data-Regime Bias: Prior studies evaluate Foundation Models (FMs) primarily in low-data (few-shot) regimes, failing to assess their performance with realistic, sufficient training data where clinical relevance is determined.
  2. Suboptimal Adaptation: Existing methods for adapting 2D FMs to 3D tasks (e.g., freezing 2D backbones and using simple average/median pooling) are inefficient, leaving significant performance gains untapped.
  3. Insufficient Task Coverage: Current benchmarks are narrow, often limited to single modalities or anatomical regions, preventing a comprehensive assessment of generalization.

2. Methodology: AnyMC3D

The authors propose AnyMC3D, a scalable 3D classifier adapted from 2D Foundation Models. The core philosophy is to leverage powerful 2D pre-trained weights while adding minimal, task-specific learnable parameters.

Core Architecture

• Frozen 2D Backbone: A pre-trained 2D Vision Transformer (ViT) (e.g., DINOv3, MedImageInsight) is kept frozen.
• In-Plane Adaptation (LoRA): To adapt the 2D backbone to a new task without full fine-tuning, Low-Rank Adaptation (LoRA) adapters are applied to the patch embedding and self-attention projection layers. This introduces only ~1M trainable parameters per task.
• Slice-wise Encoding: The 3D volume is decomposed into 2D slices along a chosen axis. Each slice is processed by the adapted 2D FM to extract a class-token embedding.
• Permutation-Invariant Slice Aggregation: Instead of using sequential models (RNNs/Transformers) which assume strict slice ordering (problematic due to anisotropic spacing), the authors propose a Query-Based Attention Pooling mechanism. A learnable task-specific query attends to the slice embeddings, weighting them based on relevance to the specific task.

Versatile Extensions

The framework supports three advanced capabilities:

1. Multi-View/Modality Learning: Supports inputs with multiple views (e.g., sagittal, coronal) or sequences (e.g., T1, T2 MRI) by using view-specific LoRA adapters and queries, followed by a fusion step.
2. Auxiliary Pixel-Level Supervision: For tasks with segmentation masks, the model can be trained with an auxiliary segmentation loss using a lightweight 3D decoder, improving classification accuracy for subtle findings without increasing inference cost.
3. Interpretable Heatmaps: Generates 3D heatmaps by combining per-slice 2D attention maps with the learned slice importance scores, allowing clinicians to visualize which anatomical regions drove the decision.

3. Key Contributions

1. Identification of Pitfalls: The paper systematically identifies and analyzes the three pitfalls (Data-Regime Bias, Suboptimal Adaptation, Insufficient Tasks) that have obscured the true potential of medical FMs.
2. AnyMC3D Framework: A novel, scalable architecture that achieves state-of-the-art (SOTA) performance across diverse 3D tasks using a single frozen 2D backbone and lightweight plugins.
3. Comprehensive Benchmark: Establishment of a new benchmark comprising 12 diverse tasks covering multiple modalities (CT, MRI), anatomical regions (Head, Chest, Abdomen, Shoulder), and pathologies, with realistic dataset sizes.
4. Critical Insights:
   • Adaptation is Key: Proper adaptation (LoRA + Attention Pooling) is essential; naive freezing yields poor results.
   • General vs. Medical FMs: General-purpose 2D FMs (e.g., DINOv3), when properly adapted, can match or exceed medical-specific FMs.
   • 2D > 3D for Classification: 2D-based methods with slice fusion consistently outperform native 3D architectures for 3D classification tasks, challenging the assumption that 3D inputs require 3D models.

4. Experimental Results

The authors evaluated AnyMC3D against SOTA baselines, including 3D backbones (ResNet, DenseNet), 3D medical FMs (MedicalNet, VoCo), and task-optimized solutions.

• Performance: AnyMC3D achieved the highest average AUC (0.894) across 10 diverse tasks, outperforming all 3D baselines.
• Parameter Efficiency: It uses only ~1.3M trainable parameters per task, which is 10–50x fewer than competitive methods (e.g., MST uses 23M, VoCo uses 50M).
• Challenge Wins:
  • VLM3D Challenge: Won 1st place among 118 participants on the CT-RATE dataset (18 chest abnormalities) using only ~0.5M parameters, beating models with expensive large-scale pretraining.
  • PDAC Detection: Outperformed the challenge-winning PanDx solution (0.962 AUC vs. 0.949) using only classification labels, and further improved to 0.973 AUC with auxiliary supervision.
  • Head CT Triage: Surpassed the specialized DeepCNTD model in classifying 75 head CT findings (0.820 vs. 0.768 average AUC).
• Data Efficiency: In low-data regimes (20% of training data), AnyMC3D significantly outperformed 3D DenseNet, demonstrating a 3x data efficiency advantage.

5. Significance and Impact

• Paradigm Shift: The paper challenges the prevailing belief that 3D medical tasks require 3D foundation models. It demonstrates that 2D FMs are superior for 3D classification when combined with effective fusion strategies, likely because 2D models learn sharper boundaries and richer textures from natural images that transfer well to medical slices.
• Scalability: By eliminating the need for task-specific model training and heavy fine-tuning, AnyMC3D enables rapid deployment of AI solutions for new clinical applications with minimal computational resources.
• Clinical Relevance: The framework provides interpretable heatmaps and handles multi-view inputs, addressing critical needs for clinical trust and workflow integration.
• Future Direction: The findings suggest that the community should focus less on building massive, modality-specific 3D pretraining datasets and more on developing robust adaptation strategies for existing general-purpose 2D foundation models.

In conclusion, AnyMC3D proves that a single, scalable framework adapted from 2D foundation models can outperform specialized 3D models across a wide spectrum of medical imaging tasks, offering a highly efficient and effective path forward for clinical AI deployment.