⚡ electrical engineering

Halfway to 3D: Ensembling 2.5D and 3D Models for Robust COVID-19 CT Diagnosis

This paper proposes a robust deep learning framework for COVID-19 diagnosis from chest CT scans that ensembles a 2.5D DINOv3-based branch for slice-level features and a 3D ResNet-18 branch with advanced pretraining for volumetric context, achieving state-of-the-art performance on the PHAROS-AIF-MIH benchmark.

Original authors: Tuan-Anh Yang, Bao V. Q. Bui, Chanh-Quang Vo-Van, Truong-Son Hy

Published 2026-03-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tuan-Anh Yang, Bao V. Q. Bui, Chanh-Quang Vo-Van, Truong-Son Hy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery: Is a patient's chest infected with COVID-19, or is it something else?

Doctors usually look at CT scans, which are like 3D movies of the inside of a lung. But looking at these scans is hard because every hospital uses different machines, different settings, and sees different types of patients. It's like trying to recognize a friend's face when they are wearing sunglasses, a hat, and standing in a foggy room. Sometimes, the "fog" (different data sources) tricks the computer into making mistakes.

This paper describes a clever team of "detectives" (AI models) that solved this problem by using two different ways of looking at the same picture and then having them vote on the answer together.

Here is how they did it, broken down into simple concepts:

1. The Two Detectives

Instead of relying on just one AI, the researchers built a team of two specialists who look at the lung scans differently.

Detective A: The "3D Architect" (The 3D Model)
- How they work: This detective looks at the entire 3D block of the lung at once. Imagine holding a loaf of bread and looking at the whole loaf to understand its shape and texture.
- Their superpower: They are great at seeing the "big picture" and how different parts of the lung connect in 3D space.
- Their training: To make sure this detective doesn't get confused by different hospitals, they were trained using a special technique called "VREx." Think of this as training the detective to ignore the color of the walls in the room and focus only on the furniture, so they can recognize the room no matter where it is.
Detective B: The "2.5D Slice Master" (The 2.5D Model)
- How they work: This detective doesn't look at the whole loaf of bread. Instead, they take the bread and slice it into thin pieces (like a sandwich). They look at the Axial (top-down), Coronal (front-facing), and Sagittal (side-facing) slices.
- Their superpower: They are incredibly sharp at spotting tiny details on a single slice, like a small speck of dust or a tiny crack that the 3D detective might miss because they are looking at the whole loaf.
- Their training: They were trained using a massive library of general images (like cats, cars, and trees) before being shown medical scans. This is like hiring a detective who has seen millions of photos of everything in the world, so they instantly recognize patterns in the lung slices.

2. The Strategy: "The Committee Vote"

Individually, both detectives are good, but they have blind spots.

The 3D Architect might miss a tiny detail.
The Slice Master might miss how a disease spreads through the whole volume.

So, the researchers created a Committee. They let both detectives make their own guesses, and then they combined the results.

If Detective A says "90% sure it's COVID" and Detective B says "85% sure," the committee takes the average and makes a final, very confident decision.
This is called Ensembling. It's like asking two experts for advice before making a big decision; you are much less likely to be wrong than if you asked just one person.

3. The Results: Winning the Challenge

The team tested their method on a huge challenge called PHAROS-AIF-MIH, which is like the "Olympics" for medical AI. The data came from many different hospitals with different equipment (the "foggy rooms").

The Outcome: Their "Committee" approach finished 2nd place in the main challenge and 3rd place in the fairness challenge.
Why it matters:
- Robustness: Their system didn't break when the data changed. It worked well whether the scan came from Hospital A or Hospital B.
- Fairness: They made sure the AI worked equally well for men and women. Often, AI gets better at diagnosing one gender over the other, but this team adjusted their "voting weights" to balance the scores, ensuring no one was left behind.

The Big Takeaway

The paper teaches us a valuable lesson: Don't put all your eggs in one basket.

By combining a model that sees the whole 3D world with a model that sees detailed 2D slices, and by training them to be fair and adaptable, the researchers created a medical AI that is tougher, smarter, and more reliable than previous attempts. It's a step toward a future where AI can help doctors diagnose diseases accurately, no matter where the patient is or what machine was used to scan them.

1. Problem Statement

The paper addresses the challenges of developing robust deep learning models for COVID-19 detection and disease classification using chest CT scans. Key issues include:

Domain Shift: Clinical datasets are collected from multiple institutions with varying scanners, acquisition protocols, and patient demographics, leading to performance degradation when models are applied to unseen data sources.
Fairness: Medical AI models often exhibit performance disparities across demographic groups (e.g., gender), raising ethical concerns.
Representation Limitations: Pure 3D volumetric models can be computationally expensive and may miss fine-grained slice-level patterns, while 2D models ignore crucial volumetric context.
Benchmark Requirements: The PHAROS-AIF-MIH benchmark specifically demands methods that are robust to heterogeneous data sources and maintain fairness across demographics.

2. Methodology

The authors propose a hybrid deep learning framework that integrates two complementary branches: a 3D volumetric branch and a 2.5D multi-view branch, combined via logit-level ensembling.

A. Data Preprocessing

Raw axial CT slices are reconstructed into full 3D volumes (128×128×128).
Duplicate slices are removed, followed by 3D Gaussian denoising and mask sharpening.
Voxel intensities are normalized to [0, 255].

B. The 3D Branch (Volumetric Context)

Architecture: A ResNet-18 architecture adapted for single-channel 3D input (originally designed for video processing).
Training Strategy (Two-Stage):
1. Domain Generalization Pretraining: Uses Variance Risk Extrapolation (VREx) to enforce domain-invariant representations across different data sources. The loss function combines cross-entropy with a variance regularization term.
2. Task-Specific Fine-tuning: The model is fine-tuned using Supervised Contrastive Learning to improve feature separability, combined with MixUp augmentation to smooth decision boundaries.

C. The 2.5D Branch (Slice-Level Features)

Approach: Extracts orthogonal slices (Axial, Coronal, Sagittal) from the reconstructed volumes.
Architecture: Utilizes a DINOv3 Vision Transformer backbone, pretrained via self-supervised learning on large-scale natural image datasets.
Processing:
- Slices are resized to 224×224.
- 8–12 representative slices are sampled per view.
- Features are extracted independently for each view and fused at the feature level.
- Training involves a progressive unfreezing strategy (frozen backbone $\to$ partial unfreezing $\to$ full end-to-end fine-tuning).

D. Ensemble Integration

Predictions from both branches are combined via logit-level averaging.
This strategy leverages the global context of the 3D model and the high-resolution, fine-grained details of the 2.5D model to stabilize predictions across heterogeneous sources.

3. Key Contributions

Hybrid Framework: Proposes a novel 2.5D–3D framework that effectively combines multi-view slice representations with volumetric modeling for CT-based diagnosis.
Advanced Training Strategies: Incorporates VREx for domain generalization and Supervised Contrastive Learning in the 3D branch to enhance cross-source robustness.
Pretrained Transformer Backbone: Designs a multi-view 2.5D pipeline using DINOv3, leveraging powerful pretrained visual representations for medical imaging.
Empirical Validation: Demonstrates that ensembling 2.5D and 3D models significantly improves robustness and performance on the challenging PHAROS-AIF-MIH benchmark, particularly regarding fairness and multi-source generalization.

4. Experimental Results

The method was evaluated on the PHAROS-AIF-MIH benchmark across two tasks:

Task 1: Binary COVID-19 Detection
- Performance: The ensemble achieved the runner-up position in the challenge.
- Metrics: Best ensemble Macro F1-score of 0.751 on the test set.
- Observation: The ensemble outperformed individual models, showing improved generalization across hospitals (especially H2), though H3 remained challenging for all methods.
Task 2: Fair Multi-class Disease Diagnosis
- Performance: Ranked third place.
- Metrics: Best Macro F1-score of 0.633 (achieved with a 0.7-weighted ensemble).
- Fairness: The ensemble significantly improved performance for male patients (who had lower scores in standalone models), balancing the gender gap. The 2.5D model generally outperformed the 3D model in multi-class classification, suggesting slice-level features are sufficient for distinguishing complex pathologies.
Ablation Insights:
- The 2.5D DINOv3 model consistently outperformed the 3D ResNet-18 in standalone performance.
- Ensembling provided the most significant gains for the binary detection task (combining global and local cues) but offered diminishing returns for the multi-class task where the 2.5D model was already optimal.
- Both models struggled with Squamous Cell Carcinoma (Class G), indicating a specific difficulty in distinguishing this pathology.

5. Significance

Robustness to Domain Shift: The study validates that combining domain generalization techniques (VREx) with hybrid architectures creates models resilient to the variations found in real-world multi-institutional data.
Complementary Representations: It highlights a functional dichotomy: 3D volumetric models excel at capturing global context for binary tasks, while 2.5D slice-based models (powered by large-scale pretrained transformers) excel at fine-grained, multi-class pathology grading.
Fairness in AI: The work demonstrates that ensemble weighting can be tuned to balance accuracy and fairness, reducing performance disparities between demographic groups (e.g., gender).
Clinical Viability: The proposed framework offers a practical pathway toward clinically viable diagnostic tools that can withstand the domain shifts inherent in real-world hospital environments.