Technical Summary: Scaling Vision Models Does Not Consistently Improve Localisation-Based Explanation Quality

Problem Statement

While artificial intelligence models are increasingly scaled to enhance predictive accuracy, it remains unclear whether increasing model depth and parameter count improves the quality of post-hoc explanations. In safety-critical domains such as healthcare and finance, transparency is paramount; however, deep neural networks (CNNs and Vision Transformers) often function as "black boxes." A prevailing assumption suggests that larger models learn features with stronger semantic structure, potentially yielding explanations that align better with human interpretations. This study challenges that assumption by investigating whether scaling architectural depth and parameter count consistently leads to higher-quality visual explanations, specifically regarding their ability to localize relevant image regions.

Methodology

The authors conducted a comprehensive empirical evaluation involving 11 computer vision models across three distinct families: ResNet, DenseNet, and Vision Transformers (ViT). The models varied in depth and complexity, ranging from ResNet-18 to ResNet-101, DenseNet-121 to DenseNet-201, and ViT-B/16 to ViT-L/16. The study included both models trained from scratch and those utilizing pretrained weights (ImageNet-1k, CheXpert, ImageNet-21k).

Datasets:
Experiments were performed on three datasets with ground-truth segmentation masks:

1. COVID-QU-Ex: Chest X-rays for COVID-19 detection.
2. Oxford-IIIT Pet: Animal images for object localization.
3. Chest X Pneumothorax: Chest X-rays for pneumothorax detection.

Explainable AI (XAI) Methods:
Five post-hoc explanation methods were applied to generate saliency maps:

• Saliency Maps
• GradientSHAP
• Integrated Gradients
• Feature Permutation
• Grad-CAM (adapted for ViTs)

Evaluation Metrics:
Explanation quality was quantified by measuring the alignment between generated attribution maps and ground-truth masks using two metrics:

1. Relevance Rank Accuracy (RRA): Measures the fraction of the top-ranked attribution pixels that fall within the ground-truth mask.
2. Dual-Polarity Precision (DPP): A novel metric proposed by the authors. Unlike magnitude-only evaluations, DPP explicitly measures the spatial precision of both positive attributions (supporting evidence inside the mask) and negative attributions (counter-evidence outside the mask). It penalizes contradictory or noisy background attributions.

Statistical Analysis:
The study employed Kruskal-Wallis H-tests and Mann-Whitney U-tests with Benjamini-Hochberg correction to assess statistical significance across model scales and pretraining regimes. Fidelity was further assessed via sanity checks involving layer-wise weight randomization and input pixel permutation.

Key Contributions

1. Systematic Scale Evaluation: The authors examined the relationship between model complexity (parameter count) and classification performance across 11 architectures and three datasets.
2. Quantification of Pretraining Effects: The study quantified how pretraining influences XAI explanation quality, comparing scratch-trained models against their pretrained counterparts.
3. Introduction of Dual-Polarity Precision (DPP): The paper introduces DPP to address limitations in existing metrics by evaluating the spatial precision of both positive and negative attributions, ensuring explanations are penalized for contradictory background noise.
4. Empirical Evidence of Divergence: The study provides evidence that predictive performance and explanation quality can diverge, noting scenarios where models achieve high accuracy but near-zero localization precision.

Results

Model Scale and Explanation Quality:
Across all datasets and XAI methods, increasing architectural depth and parameter count did not consistently improve explanation quality.

• In statistical comparisons, larger models (e.g., ResNet-101, DenseNet-201, ViT-L/16) did not systematically outperform smaller variants (e.g., ResNet-18, DenseNet-121, ViT-B/16).
• In cases where significant differences were found, smaller models matched or exceeded the performance of larger models in 34% of comparisons, while the largest model achieved the highest score in only 12%.
• Effect sizes for scale-related differences were typically negligible to small.

Impact of Pretraining:
Pretraining generally improved predictive performance but did not consistently increase localization scores (RRA or DPP).

• While pretrained models showed stronger dependence on learned weights (indicated by higher degradation under layer randomization) and higher sensitivity to input structure, this did not translate to a systematic gain in localization metrics.
• In many significant comparisons, scratch-trained models performed as well as or better than pretrained models.

Predictive Performance vs. Explanation Quality:
A critical finding was the decoupling of accuracy and interpretability. On the Chest X Pneumothorax dataset, models achieved reasonable classification performance (AUC ~0.77–0.87) while producing localization scores near zero. This suggests that high accuracy can be achieved by relying on spurious correlations or acquisition artifacts rather than the annotated anatomical regions.

Sanity Checks:

• Layer Randomization: Deeper and especially pretrained models showed higher degradation when weights were randomized, indicating a stronger dependence on learned parameters rather than just network structure.
• Input Randomization: Most gradient-based methods showed low sensitivity to pixel shuffling, though Grad-CAM exhibited larger shifts. Pretrained models generally showed higher sensitivity to input structure changes compared to scratch-trained models.

Significance and Claims

The paper concludes that larger models do not reliably provide higher-quality explanations. The assumption that scaling up models inherently improves the semantic alignment of saliency maps is not supported by the data. Instead, smaller models often provide explanations that align as well as, or better than, their deeper counterparts.

The authors argue that explainability should be assessed explicitly during model selection, particularly for safety-sensitive deployments. Relying solely on predictive accuracy is insufficient, as models can achieve high performance while failing to localize the intended regions of interest. The introduction of Dual-Polarity Precision (DPP) offers a more rigorous way to evaluate these explanations by accounting for both supporting and counter-evidence.

The study emphasizes that model selection procedures should consider computational costs and operational requirements alongside performance, as smaller models can offer comparable explanation quality with significantly lower resource demands. The findings suggest that in over-parameterized settings, predictive accuracy and interpretability may diverge, necessitating explicit evaluation of explanation fidelity rather than assuming it improves with scale.