Technical Summary: NucEval – A Robust Evaluation Framework for Nuclear Instance Segmentation

Problem Statement

In computational pathology, nuclear instance segmentation is a foundational task critical for downstream clinical applications such as cancer diagnosis and prognostication. While significant research efforts have focused on improving segmentation models (e.g., CNNs like HoVer-Net and ViTs like CellViT) and pre/post-processing techniques, the evaluation pipeline has received comparatively less attention.

Current evaluation practices suffer from several fundamental limitations that can lead to suboptimal performance estimation and unfair model comparisons. The authors identify four specific issues in the existing literature:

1. Vague Regions: Standard datasets often lack annotations for ambiguous areas (e.g., out-of-focus regions, tissue folds, or indistinguishable boundaries). Including these regions in evaluation can unfairly penalize or reward models for predictions in areas where even human experts cannot provide reliable ground truth.
2. Unnormalized Scores: Metrics are typically averaged across image patches without accounting for the number of nuclei per patch. This introduces a bias where patches with few nuclei disproportionately influence the final score, which does not reflect clinical reality where segmentation is performed on whole-slide images (WSIs).
3. Overlapping Instances: In many datasets, overlapping nuclear regions are arbitrarily assigned to a single instance (often the last encountered in the annotation file). This penalizes models that correctly identify overlapping structures and introduces a logical bias based on annotation order.
4. Border Uncertainty: Manual annotations at object boundaries are inherently uncertain. Penalizing models for minor deviations (a few pixels) at the boundary is often unfair, as human annotators themselves may disagree on the exact boundary location.

Methodology

To address these issues, the authors propose NucEval, a unified, Python-based evaluation framework that integrates four specific modifications into standard metrics (Panoptic Quality [PQ], Aggregated Jaccard Index [AJI], Dice, Detection Quality [DQ], and Segmentation Quality [SQ]).

The Four Modifications

1. Incorporating Vague Areas (Modification #1):

   • Ambiguous regions are explicitly excluded from both the ground truth and model predictions.
   • For instances intersecting these regions, a threshold-based strategy is employed: an instance is removed if the ratio of its overlap with the ambiguous region to its total area exceeds a predefined threshold (e.g., 0.25).
   • Remaining instances are relabeled to ensure contiguous integer identifiers for metric computation.

2. Score Normalization (Modification #2):

   • Instead of a simple arithmetic mean of scores across images, the framework computes a nuclear count-normalized average.
   • Each image's contribution to the final score is weighted by the number of annotated nuclei it contains. This mitigates bias toward sparsely annotated patches.

3. Handling Overlapping Regions (Modification #3):

   • Overlapping regions are assigned to all relevant instances rather than arbitrarily to a single one.
   • This requires ground truth data to be provided as a set of Regions of Interest (ROIs) or a list of binary masks where overlaps are preserved, rather than a single labeled mask where overlaps are merged.

4. Border Uncertainty Handling (Modification #4):

   • A "zone of uncertainty" is created around each instance boundary by applying morphological erosion and dilation.
   • The difference between the dilated and eroded masks defines a zone that is excluded from evaluation (pixels are zeroed out in both ground truth and prediction).
   • The zone width is a configurable hyperparameter (set to 1 pixel in the study) to balance annotation uncertainty with the preservation of meaningful evaluation regions.

Experimental Setup

The framework was evaluated using:

• Datasets:
  • NuInsSeg: The primary dataset, chosen for its unique provision of vague area annotations and dual versions of overlapping area handling (preserved vs. merged).
  • CryoNuSeg: Used to test modifications #2, #3, and #4 (lacks vague area annotations).
  • PCNS: Used to test modifications #2 and #4 (lacks vague area and ROI annotations).
• Models: Three state-of-the-art deep learning models representing different architectures:
  • HoVer-Net (CNN-based)
  • Hover-Next (CNN-based)
  • CellViT (Vision Transformer-based)
• Training: Models were trained using 5-fold cross-validation on a single workstation with an NVIDIA RTX 4090 GPU.

Key Results

The study demonstrates that applying these modifications consistently improves evaluation metrics across all three models and datasets.

• Individual Impact:

  • Modification #4 (Border Uncertainty) showed the most substantial and consistent impact, improving PQ by approximately 5–6% across all models on the NuInsSeg dataset.
  • Modification #1 (Vague Areas) consistently improved all metrics.
  • Modification #2 (Score Normalization) showed significant gains on imbalanced subsets of the data (e.g., PQ improvements of 27–32% on a specific subset), highlighting its importance for datasets with varying nuclear densities.
  • Modification #3 (Overlapping Regions) had the least impact on overall metrics (as overlaps constitute a small portion of the dataset) but demonstrated that merging overlaps can significantly degrade performance (up to ~10% PQ drop in idealized comparisons).

• Cumulative Impact:

  • Applying all modifications simultaneously yielded the highest gains. On the full NuInsSeg dataset, PQ increased by 10.67% to 12.00% across the three models compared to the baseline.
  • On the imbalanced NuInsSeg subset, cumulative PQ improvements were even more pronounced, ranging from 38.45% to 43.19%.
  • Similar positive trends were observed on CryoNuSeg and PCNS, confirming the generalizability of the approach, even when not all modifications could be applied due to missing annotation types.

Significance and Claims

The authors position NucEval not as a new segmentation algorithm, but as a robust evaluation framework essential for the fair comparison and selection of models for clinical applications.

• Model Agnosticism: The modifications are independent of the segmentation model architecture, making them applicable to any CNN or ViT-based approach.
• Fairness and Reproducibility: By addressing biases in score aggregation, ambiguous regions, and boundary uncertainty, NucEval aims to prevent the selection of inferior models for downstream clinical tasks.
• Practical Utility: The authors provide a publicly available Python implementation (NucEval) with complete guidelines, allowing researchers to selectively apply these modifications based on the availability of specific annotations (e.g., vague area masks or ROI sets) in their own datasets.
• Future Direction: The paper notes that while the current implementation uses fixed parameters, the framework is designed to be flexible. It also suggests that future dataset releases should consider preserving overlapping regions and annotating vague areas to fully leverage these evaluation improvements.

In conclusion, the paper argues that rigorous evaluation is as critical as model development in computational pathology. By standardizing and refining the evaluation pipeline through NucEval, the community can achieve more reliable performance estimates and ultimately better clinical outcomes.