Evaluating image upsampling strategies for downstream microscopy image classification

This study demonstrates that while bicubic interpolation significantly degrades downstream microscopy image classification performance, deep learning-based super-resolution methods can effectively recover class-relevant information, sometimes even outperforming original ground-truth data, highlighting the critical need for confidence-aware evaluation metrics in reconstruction pipelines.

The Gist

Imagine you are a doctor trying to identify different types of blood cells under a microscope. To do this, you need a clear, high-resolution picture. But sometimes, due to storage limits or slow internet, those pictures get shrunk down to a tiny, blurry thumbnail (64x64 pixels). Before your AI assistant can help you diagnose, you have to blow that tiny thumbnail back up to a full-screen size (224x224 pixels).

The big question this paper asks is: Does it matter how you blow the picture up?

The researchers tested three different ways to "zoom in" on these blurry blood cell photos and saw how well a computer (an AI) could still identify them.

The Three "Zoom" Methods

Think of the three methods like different ways to restore an old, faded photograph:

1. The "Blender" (Bicubic Interpolation): This is the standard, automatic zoom on your phone. It takes the blurry pixels and tries to guess what should be in between by smearing colors together. It's fast, but it often makes the image look soft and mushy, like a watercolor painting left in the rain.
2. The "Pixel-Perfect Architect" (SwinIR Classical): This is a smart AI trained to be a perfectionist. Its goal is to make the new pixels match the original high-quality photo as mathematically as possible. It's like a forensic artist trying to recreate a crime scene photo exactly, pixel-for-pixel.
3. The "Creative Artist" (SwinIR RealGAN): This is a different kind of AI. It doesn't care about matching every single pixel perfectly. Instead, it cares about making the image look real and sharp to the human eye. It's like an artist who sees a blurry face and paints in the missing details (like the texture of skin or the sharpness of an eye) based on what a face should look like, even if they are inventing some details.

The Experiment

The researchers took a dataset of blood cells, shrank them down, and then used these three methods to blow them back up. They then fed these four versions of the photos (the original high-res, the "Blender," the "Architect," and the "Artist") into two different AI classifiers (ResNet-50 and Vision Transformer) to see which one could identify the blood cells best.

The Surprising Results

Here is where the story gets interesting. You might think the "Pixel-Perfect Architect" would win because its photos looked the most mathematically accurate. Or you might think the "Blender" would be okay because it's simple.

The results were the opposite:

• The "Blender" (Bicubic) was the worst. It made the AI confused. The soft, mushy images hid the tiny details the AI needed to tell the cells apart.
• The "Pixel-Perfect Architect" was good, but not the best. It was very accurate to the original, but it didn't help the AI perform any better than the original high-res photos.
• The "Creative Artist" (RealGAN) was the champion. Even though its photos were less mathematically similar to the original (it invented some textures), the AI understood them better. The AI was more confident and more accurate when looking at the "Artist's" version.

The "Why" (The Analogy)

Why did the "Creative Artist" win?

Imagine you are trying to recognize a friend in a crowd.

• If you look at a blurry, smudged photo (Bicubic), you can't see their features. You might guess wrong.
• If you look at a perfectly accurate but flat drawing (Classical), you see them exactly as they are, but maybe the lighting is a bit dull.
• If you look at a vivid, high-contrast painting (RealGAN) that exaggerates their sharp jawline and bright eyes, your brain (or the AI) might actually recognize them faster because the important features are highlighted, even if the painting isn't 100% realistic.

The "Creative Artist" AI added sharp, clear textures that helped the classification AI "see" the differences between cell types more clearly, even if it technically added some "fake" details.

The Big Takeaway

This paper teaches us a valuable lesson for the future of medical AI:

Don't just trust the "fidelity" scores.
Usually, scientists measure image quality by how close a restored image is to the original (using metrics like PSIM and SSIM). This paper shows that being mathematically perfect isn't the same as being useful.

Sometimes, an image that looks "better" to a human or an AI (because it has sharp, clear textures) is actually better for making decisions, even if it's not a perfect copy of the original.

In short: When preparing images for AI to analyze, don't just use the automatic "zoom" button. Using advanced AI to "re-imagine" the missing details can actually make the AI smarter and more confident in its diagnoses.

Technical Summary

1. Problem Statement

Microscopy images are frequently downsampled due to storage and computational constraints, necessitating reconstruction (upsampling) before downstream analysis. While Super-Resolution (SR) techniques are commonly evaluated using pixel-level fidelity metrics (e.g., PSNR, SSIM), their actual impact on Deep Learning (DL) model behavior and classification performance remains poorly understood.

The core research question is: How do common upsampling methods (interpolation vs. DL-based SR) affect pixel-level fidelity, downstream classification accuracy, and prediction confidence in microscopy image analysis? The authors argue that conclusions about model performance are often confounded by the specific data preprocessing pipeline used to scale input resolution.

2. Methodology

A. Dataset and Experimental Design

• Dataset: The study utilizes BloodMNIST, a multi-class benchmark of peripheral blood cell images.
• Experimental Variants: To ensure a fair comparison, the authors constructed four dataset variants, all standardized to 224×224 pixels with identical train/validation/test splits:
  1. Ground Truth (GT): Native 224×224 images.
  2. Bicubic: Native 64×64 images upsampled via standard bicubic interpolation.
  3. SwinIR Classical: Native 64×64 images reconstructed using a SwinIR model trained for high pixel-wise fidelity (SSIM/PSNR optimization).
  4. SwinIR RealGAN: Native 64×64 images reconstructed using a SwinIR model trained with adversarial/perceptual losses (optimizing for visual realism).
• Note: Both SR models performed ×4 upsampling (64→256) followed by resizing to 224 for consistency.

B. Models and Training Protocol

• Classification Models: Two architectures were fine-tuned:
  • ResNet-50: Convolutional Neural Network (CNN).
  • Vision Transformer (ViT-B): Transformer-based model with 16×16 patches.
• Training Strategy: To isolate the effect of image reconstruction rather than model capacity, a lightweight training regimen was used:
  • Pre-trained on ImageNet weights.
  • Fine-tuned for only 5 epochs using the Adam optimizer (learning rate 1e-3).
  • Applied geometric augmentations (flips, rotations, affine transforms) to preserve morphological features.
  • Strict random seeding was enforced to ensure reproducibility.

C. Evaluation Metrics

The study employed a multi-faceted evaluation approach:

1. Image Fidelity: Measured against Ground Truth using SSIM (Structural Similarity Index Measure) and PSNR (Peak Signal-to-Noise Ratio).
2. Classification Performance: Evaluated on the test set using Accuracy and Macro-F1 Score.
3. Confidence-Aware Metric: Introduced AUPR Success (Area Under the Precision-Recall curve for Successful predictions). This metric treats a "correct prediction" as the positive event and uses the model's maximum softmax probability as the confidence score, assessing whether the model is confident when it is right.

3. Key Results

A. Image Fidelity vs. Perceptual Quality

• SwinIR Classical achieved the highest SSIM (0.95) and PSNR (37.7) scores, confirming its optimization for pixel-level accuracy.
• SwinIR RealGAN had the lowest fidelity scores (SSIM 0.83, PSNR 27.9) but produced visually sharper, more realistic textures.
• Bicubic Interpolation fell in the middle regarding fidelity but failed to recover high-frequency details effectively.

B. Downstream Classification Performance

Surprisingly, the results contradicted the assumption that higher pixel fidelity leads to better classification:

• Best Performance: SwinIR RealGAN yielded the highest accuracy and F1 scores for both ResNet-50 and ViT-B, outperforming the original Ground Truth.
  • ResNet-50 (RealGAN): 97.43% Accuracy vs. 96.40% (GT).
  • ViT-B (RealGAN): 89.62% Accuracy vs. 89.01% (GT).
• Worst Performance: Bicubic Interpolation resulted in the lowest classification scores, particularly for the ViT model.
• ResNet vs. ViT: ResNet-50 generally outperformed ViT-B, likely due to convolutional inductive biases aiding faster feature learning in the short training window.

C. Confidence and Prediction Behavior

• Confidence Calibration: The RealGAN variant consistently produced the highest confidence scores for correct predictions (highest AUPR Success).
• Artifact Impact: Bicubic interpolation introduced uncertainty, leading to lower confidence and more incorrect predictions, even when the confidence score appeared high (e.g., ViT-B made errors with >0.70 confidence on bicubic images).
• Key Insight: Perceptual realism (RealGAN) improved the model's internal representation and confidence calibration, even though the images were less "faithful" to the ground truth pixel-wise.

4. Key Contributions

1. Decoupling Fidelity from Performance: The study demonstrates that pixel-level fidelity (SSIM/PSNR) is not a reliable proxy for downstream classification performance in microscopy. In fact, perceptually enhanced images can outperform ground truth.
2. Confidence-Aware Evaluation: The authors advocate for and utilize AUPR Success to reveal nuances in model behavior that standard accuracy metrics miss, showing how reconstruction strategies alter the "confidence landscape."
3. Pipeline Awareness: The work highlights that the choice of upsampling strategy is not merely a preprocessing step but a critical factor that shapes feature representation and learning outcomes in DL-based microscopy.

5. Significance and Implications

• For Microscopy Research: Researchers should be cautious when downsampling images for storage or upsampling for visualization, as simple interpolation (bicubic) can degrade discriminative cues and model confidence.
• For SR Model Selection: For downstream classification tasks, perceptual/realistic SR models (like RealGAN) may be superior to fidelity-optimized models, despite lower PSNR/SSIM scores.
• Reporting Standards: The paper emphasizes the need for unambiguous reporting of reconstruction pipelines in DL studies, as the preprocessing method fundamentally alters the conclusions drawn about model performance.
• Future Directions: The authors suggest fine-tuning SR models specifically on microscopic data and expanding the evaluation to include object localization to ensure models focus on biological features rather than reconstruction artifacts.