💻 computer science

Beyond Accuracy: A Comprehensive Evaluation of ResNet50 and Vision Transformer for Trustworthy Pneumonia Detection from Chest X-Ray Images

This study presents a comprehensive evaluation of ResNet50 and Vision Transformer (ViT-B16) for pneumonia detection from chest X-rays, revealing that while ResNet50 offers superior accuracy, calibration, and efficiency, ViT-B16 demonstrates greater robustness against image degradations, thereby highlighting the necessity of multi-dimensional assessment for trustworthy diagnostic systems.

Original authors: VISHAL MANTA

Published 2026-07-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: VISHAL MANTA

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of looking for fingerprints, you are looking for invisible clues inside a person's chest. This is the world of medical imaging, where doctors use special cameras to take pictures of lungs to see if they are sick. For a long time, the best detectives were human doctors, but they get tired, and sometimes the clues are so tiny or blurry that even the sharpest eyes miss them. To help out, scientists have built "digital detectives" using a branch of computer science called Deep Learning. Think of these digital detectives as super-smart robots that learn by looking at thousands of pictures until they can spot a problem on their own.

There are two main types of these robot detectives. The first type is like a careful artist who looks at a picture one tiny brushstroke at a time, noticing how edges and textures connect. This is called a Convolutional Neural Network (CNN). The second type is like a bird flying high above a forest; it doesn't look at every single leaf but instead sees how the whole forest connects and interacts from a distance. This is called a Vision Transformer (ViT). Both are incredibly powerful, but doctors need to know which one is actually better at saving lives before they trust them in a hospital. It's not enough to just be fast or to get the right answer most of the time; the robot also needs to be honest about how sure it is, tough enough to handle bad photos, and easy to explain to a human doctor.

In this study, a researcher named Vishal Manta from the Indian Institute of Technology Guwahati decided to put two of these digital detectives head-to-head in a fair fight to see which one is the most trustworthy for spotting pneumonia, a serious lung infection. The two contenders were ResNet50, the careful artist (CNN), and ViT-B16, the high-flying bird (Vision Transformer). They didn't just ask, "Who got the most answers right?" Instead, they set up a massive obstacle course to test everything: how accurate they were, how honest they were about their confidence, how fast they were, and how well they could handle a blurry or noisy picture.

The results of this race were quite clear. When it came to the main job of spotting pneumonia, ResNet50 was the champion. It correctly identified pneumonia cases 84.56% of the time, while the Vision Transformer managed 81.14%. The researcher used a special statistical test called McNemar's test to make sure this wasn't just a lucky fluke, and the math confirmed that ResNet50 was indeed significantly better. But the story didn't end there. The study found that ResNet50 was also the more honest detective. When it said it was "90% sure," it was usually right. The Vision Transformer, however, tended to be overconfident, often saying it was sure when it actually wasn't. In the world of medicine, being overconfident is dangerous because a doctor might trust a wrong answer too much.

However, the Vision Transformer wasn't a total loser; it had a superpower. When the researchers deliberately messed up the test photos by adding static (noise), making them blurry, or changing the brightness, the Vision Transformer held its ground better than ResNet50. While both models got worse with bad photos, the Vision Transformer's accuracy dropped less. It seems that because the "bird" looks at the whole picture at once, it can still figure things out even if parts of the image are fuzzy. But this superpower came with a heavy price tag. The Vision Transformer was a giant, requiring 85.80 million adjustable settings (parameters) to work, compared to ResNet50's 23.51 million. It also took much longer to think, needing 55.60 seconds to process a batch of images versus ResNet50's 38.57 seconds, and it demanded much more computer power to run.

To make sure the robots were actually looking at the lungs and not just guessing, the researcher used a tool called Grad-CAM, which acts like a heat map to show where the robot is looking. The careful artist (ResNet50) focused its attention tightly on the sick parts of the lung, which is exactly what a doctor wants to see. The high-flying bird (ViT) was a bit more scattered in its attention. When the robots made mistakes, they mostly missed the pneumonia cases, confidently saying "Normal" when the lungs were actually sick. This suggests that while these robots are getting better, they still struggle with the sneakiest, most subtle cases of the disease.

Ultimately, this study suggests that if you need a reliable, fast, and honest assistant for a hospital that might not have a super-computer, the careful artist ResNet50 is currently the better choice. It is more accurate, more honest about its confidence, and cheaper to run. The Vision Transformer is a tough competitor that handles bad photos well, but it is too heavy and too expensive for many real-world clinics right now. The main takeaway is that building a trustworthy medical AI isn't just about who gets the highest score on a test; it's about finding the right balance between being smart, being honest, being fast, and being tough enough for the real world.

Technical Summary: Beyond Accuracy: A Comprehensive Evaluation of ResNet50 and Vision Transformer for Trustworthy Pneumonia Detection from Chest X-Ray Images

Problem Statement
Pneumonia remains a leading cause of global respiratory morbidity and mortality, necessitating accurate and timely computer-aided diagnostic (CAD) systems. While deep learning has advanced automated chest X-ray (CXR) analysis, existing literature predominantly focuses on classification accuracy (e.g., precision, recall, F1-score). There is a critical gap in evaluating other dimensions essential for trustworthy clinical deployment, including prediction reliability (calibration), robustness to image degradation, computational efficiency, statistical significance, and interpretability. Furthermore, direct comparisons between Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) are often hindered by inconsistent experimental settings, such as varying datasets and preprocessing pipelines.

Methodology
This study presents a systematic, fair comparative evaluation of two representative architectures: ResNet50 (a CNN) and Vision Transformer (ViT-B16) (a transformer-based model). The evaluation was conducted using the RSNA Pneumonia Detection Challenge dataset, comprising 26,684 frontal chest radiographs.

Experimental Setup: Both models were trained and evaluated under identical conditions, including the same data partitions (Training: 18,678; Validation: 4,003; Testing: 4,003), preprocessing (resizing to 224×224, normalization), optimization strategies (10 epochs, batch size 32), and initialization (ImageNet pretraining).
Evaluation Framework: The study employed a multidimensional assessment strategy:
1. Classification Performance: Accuracy, precision, recall, specificity, F1-score, balanced accuracy, and ROC–AUC.
2. Calibration: Expected Calibration Error (ECE), Brier Score, and reliability diagrams to assess confidence reliability.
3. Robustness: Evaluation under Gaussian noise, Gaussian blur, and brightness perturbations without retraining.
4. Computational Efficiency: Analysis of trainable parameters, Floating-Point Operations (FLOPs), and inference time.
5. Statistical Significance: McNemar's test with 95% confidence intervals to validate performance differences.
6. Explainability: Gradient-weighted Class Activation Mapping (Grad-CAM) and qualitative error analysis of misclassified cases.

Key Contributions

Fair Comparative Evaluation: A systematic comparison of ResNet50 and ViT-B16 under strictly identical preprocessing, training, and testing protocols.
Holistic Evaluation Framework: Moving beyond accuracy to include calibration, robustness, efficiency, statistical significance, and explainability.
Reliability and Robustness Assessment: Quantitative evaluation of model confidence (ECE, Brier Score) and resilience to image corruptions (noise, blur, brightness).
Clinical Deployment Analysis: A comprehensive trade-off analysis involving computational complexity, inference time, and visual interpretability via Grad-CAM.

Experimental Results

Classification Performance: ResNet50 outperformed ViT-B16 across all metrics. ResNet50 achieved 84.56% accuracy compared to 81.14% for ViT-B16. ResNet50 also demonstrated superior precision (72.85% vs. 63.97%), recall (52.44% vs. 41.26%), and F1-score (60.98% vs. 50.17%).
Statistical Significance: McNemar's test confirmed the performance difference was statistically significant ( $\chi^2 = 45.00$ , $p = 1.97 \times 10^{-11}$ ).
Calibration: ResNet50 exhibited better calibration with a lower Expected Calibration Error (0.033 vs. 0.082) and Brier Score (0.112 vs. 0.136), indicating more reliable confidence estimates. ViT-B16 showed signs of overconfidence.
Robustness: Contrary to classification performance, ViT-B16 demonstrated greater robustness against image degradations. It retained higher accuracy and showed smaller performance drops under Gaussian noise, blur, and brightness shifts compared to ResNet50.
Computational Efficiency: ResNet50 was significantly more efficient, requiring fewer trainable parameters (23.51M vs. 85.80M), lower FLOPs (4.13 vs. 16.87 GFLOPs), and faster inference times (38.57s vs. 55.60s).
Explainability: Grad-CAM visualizations revealed that ResNet50 produced more localized activation maps focused on diseased lung regions, whereas ViT-B16 generated more spatially distributed attention. Both models struggled with subtle or diffuse opacities, often resulting in high-confidence false negatives.

Significance and Claims
The paper claims that while ResNet50 offers superior diagnostic accuracy, calibration, and computational efficiency—making it more suitable for resource-constrained clinical settings—ViT-B16 offers distinct advantages in robustness against image quality degradation due to its global self-attention mechanisms.

The authors conclude that model selection for clinical deployment should not rely solely on predictive accuracy. Instead, a multidimensional evaluation strategy is required to balance predictive performance with reliability, robustness, efficiency, and interpretability. The study highlights the complementary strengths of CNNs and Transformers, suggesting that a "trustworthy" AI system requires a holistic assessment of these diverse performance dimensions. The authors acknowledge limitations, including the use of a single dataset and synthetic perturbations, and suggest future work should validate these findings across multi-institutional datasets and diverse imaging devices.

More like this