Improved Anomaly Detection in Medical Images via Mean Shift Density Enhancement

1. Problem Statement

Anomaly detection in medical imaging is critical for identifying rare pathological conditions (e.g., tumors, pneumonia, glaucoma) where annotated abnormal samples are scarce, expensive to obtain, or privacy-restricted.

• The Challenge: Traditional supervised learning fails due to the lack of labeled anomaly data. Existing unsupervised or one-class approaches generally fall into three categories:
  1. Reconstruction-based: (e.g., Autoencoders, GANs) Struggle with subtle or localized anomalies that do not significantly impact reconstruction error.
  2. Feature-based: Extract representations using pretrained/self-supervised models but often assume static, Euclidean feature spaces, ignoring the intrinsic manifold structure of medical data.
  3. Density-based: Model normal data distributions but often fail to account for the complex geometric properties of the data manifold.
• The Gap: Current methods typically treat the latent feature space as fixed after extraction. They lack a mechanism to explicitly restructure or refine the latent space to better separate normal from anomalous instances before scoring.

2. Methodology

The authors propose a hybrid framework called Mean Shift Density Enhancement (MSDE) that integrates self-supervised representation learning with manifold-based density estimation. The pipeline operates under a one-class learning paradigm (training only on normal samples) and consists of three stages:

A. Feature Extraction

Medical images are embedded into a high-dimensional latent space (512 dimensions) using pretrained neural network backbones:

• Domain-specific: AnatPaste (self-supervised) is used for chest X-rays.
• Generic: ResNet18 (ImageNet-pretrained) is used for other modalities (MRI, fundus, histopathology).

B. Mean Shift Density Enhancement (MSDE)

This is the core novelty. Instead of using raw features, the framework iteratively refines the latent embeddings to move samples toward regions of higher data likelihood.

• Mechanism: It constructs a manifold-aware similarity graph to estimate sample-specific density weights based on local neighborhoods.
• Iterative Shift: Embeddings are updated via a weighted mean-shift operation:
  $$x_i^{(t+1)} = x_i^{(t)} + \eta (\tilde{x}_i^{(t)} - x_i^{(t)})$$
  Where $\tilde{x}_i$ is the weighted average of neighbors. This process "compacts" normal clusters into high-density topological hubs while pushing anomalous instances toward the periphery.
• Unified Processing: MSDE is applied to the union of training and test sets to ensure geometric consistency and reduce representation mismatch.

C. Density-Based Anomaly Scoring

After refinement, the enhanced features undergo dimensionality reduction and probabilistic scoring:

1. Dimensionality Reduction: PCA reduces the refined features from 512 to 256 dimensions to ensure numerical stability.
2. Gaussian Modeling: A Multivariate Gaussian distribution is fitted to the normal training data in the reduced space.
3. Scoring: Anomaly scores are computed using the Mahalanobis distance, which accounts for feature correlations and variance structure.
   $$s(z) = \sqrt{(z - \mu_z)^\top \Sigma_z^{-1} (z - \mu_z)}$$
   Scores are standardized and passed through a sigmoid function to bound them between [0, 1].

3. Key Contributions

1. Novel Intermediate Step: Introduces Manifold Refinement as a distinct stage between feature extraction and anomaly scoring. This explicitly restructures the latent space to enhance the separability of normal data, a direction previously underexplored in medical imaging.
2. Hybrid Framework: Successfully combines the semantic power of self-supervised feature learning with the statistical rigor of density-based estimation.
3. Modularity: The method is model-agnostic and operates on pre-extracted features, requiring no retraining of the backbone. It can be integrated with any existing feature extractor (CNNs, self-supervised models, foundation models).
4. Efficiency: By operating on embeddings rather than raw images and avoiding iterative image generation (unlike diffusion models), the method is computationally lightweight and scalable.

4. Experimental Results

The framework was evaluated on seven diverse medical imaging datasets (Chest X-ray, Brain MRI, Retinal Fundus, Dermoscopy, Histopathology) following the MedIAnomaly benchmark protocol.

• Overall Performance:
  • Achieved the highest AUC on 4 out of 7 datasets.
  • Achieved the highest Average Precision (AP) on 5 out of 7 datasets.
• Notable Achievements:
  • Brain Tumor Detection: Achieved near-perfect performance with 0.981 AUC and 0.981 AP, significantly outperforming state-of-the-art (SOTA) baselines like MSC (0.973 AUC).
  • Camelyon16 (Histopathology): Surpassed the computationally intensive AutoDDPM diffusion model (0.807 AUC) with 0.812 AUC and 0.820 AP, despite using only lightweight geometric shifting on ResNet18 features.
  • VinDr-CXR: Outperformed the previous best (AE-U) with 0.819 AUC vs. 0.769.
• Limitations:
  • Performance was lower on the ISIC 2018 (Dermoscopy) dataset (0.705 AUC vs. 0.807 for AnatPaste). The authors attribute this to the "garbage in, garbage out" principle: dermoscopic images lack the structured anatomical regions that the AnatPaste backbone exploits, leading to less discriminative initial features that MSDE cannot fully recover.

5. Significance and Impact

• Clinical Utility: The framework offers a scalable, robust tool for early disease detection and screening in low-label settings, particularly where anomaly data is unavailable.
• Paradigm Shift: The results suggest that future improvements in anomaly detection may not solely come from better encoders or scorers, but from better feature-space transformation strategies. It demonstrates that pretrained latent representations contain "untapped" geometric structure that can be revealed through density-guided shifting.
• Deployment: The computational efficiency (avoiding end-to-end retraining and image-space generation) makes it highly suitable for real-world clinical deployment across diverse imaging modalities.

In conclusion, MSDE establishes latent space manifold refinement as a powerful, previously underutilized principle for one-class medical image anomaly detection, bridging the gap between representation learning and probabilistic modeling.