💻 computer science

A Multi-Stage Optimization Pipeline for Bethesda Cell Detection in Pap Smear Cytology

This paper presents a second-place winning solution for the Riva Cytology Challenge's Track B, introducing a multi-stage optimization pipeline that combines an ensemble of YOLO and U-Net architectures with overlap removal and binary classification to achieve a mAP50-95 score of 0.5909 in detecting Bethesda cells in Pap smear images.

Original authors: Martin Amster, Camila María Polotto

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Martin Amster, Camila María Polotto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a massive, high-resolution photograph of a crowded city street at night. Your job is to find every single person wearing a specific red hat (the "Bethesda cells") among thousands of other people, streetlights, and shadows. If you miss a person with a red hat, it's a problem. But if you point at a streetlight and say, "That's a person!", that's also a problem.

This paper describes a team of computer scientists who built a super-smart "digital detective squad" to solve this exact problem, but for Pap smear images (microscope slides used to screen for cervical cancer).

Here is how they did it, broken down into simple steps:

1. The Problem: The "One-Size-Fits-All" Trap

In the past, computers were told to look for cells using a rigid, square "magnifying glass" that was always the same size (100x100 pixels).

The Analogy: Imagine trying to find a tiny ant and a huge elephant using the exact same size picture frame. The frame might cut off the elephant's ears or include too much empty space around the ant.
The Fix: The team realized they needed to be flexible. They taught their AI to look for cells in many different sizes (from tiny 20x20 squares to larger 120x120 squares). This way, the AI could zoom in on small cells and zoom out for big ones, just like a human photographer adjusting their lens.

2. The Team: The "YOLO" Scouts and the "U-Net" Sniper

Instead of relying on just one AI, they built a team with different superpowers.

The YOLO Scouts (You Only Look Once):
- Role: These are the fast, aggressive scouts. They scan the whole image quickly and shout out, "I see a cell here!" and "I see another one there!"
- Strength: They are incredibly good at finding things (high "Recall"). They rarely miss a cell.
- Weakness: They are a bit noisy. They sometimes shout "Cell!" at random background specks (False Positives).
- Strategy: They used two different versions of this scout—one looking for tiny details and one for broader shapes—and combined their reports.
The U-Net Sniper:
- Role: This AI is slower but much more careful. Instead of just shouting "Cell!", it creates a "heat map" (like a weather map showing where it's hottest). It highlights the exact center of a cell with high precision.
- Strength: It is very accurate (high "Precision"). It rarely mistakes a speck of dust for a cell.
- Weakness: It is a bit shy and might miss a few cells that the scouts found.

3. The "Refinement Pipeline": The Filter Factory

Once the Scouts and the Sniper sent in their lists of suspects, the team didn't just accept everything. They ran the list through a three-stage filter factory to clean up the noise:

Stage 1: The "Don't Double-Count" Rule (NMS)
- If the Scout and the Sniper both point at the same spot, the system merges them into one confident report. If two reports are too close together, it keeps the one with the higher confidence score and discards the duplicate.
- Analogy: If two people tell you a car is at the corner, you don't count it as two cars. You count it as one car.
Stage 2: The "Crowded Room" Filter (Spatial Density)
- The image is split into 16 small squares.
- In crowded squares: If a square has 30+ "suspects," the system gets suspicious. It assumes the AI is just hallucinating because it's too busy. It raises the bar, only keeping the most confident suspects.
- In empty squares: If a square has very few suspects, the system lowers the bar. It thinks, "Maybe there's a rare, lonely cell here," so it keeps even weak signals to avoid missing anything.
Stage 3: The "Final Judge" (Binary Classifier)
- For the low-confidence suspects that survived the first two stages, a final expert (an AI called EfficientNet) looks at them closely.
- It asks a simple question: "Is this actually a cell, or is it just a weird shadow/artifact?" If it looks like garbage, it gets thrown out.

4. The Result: Second Place!

The team entered this system into a global competition (the Riva Cytology Challenge).

The Score: They got 2nd place in the world with a score of 0.5909.
The Trade-off: The system was tuned to find as many cells as possible (even if it meant a few false alarms) because, in cancer screening, missing a cancer cell is worse than a false alarm.
The Insight: The authors noted that if they wanted to be more "precise" (fewer false alarms), they could just raise the confidence bar a little bit. This would make the system safer for doctors to use in real life, even if the competition score dropped slightly.

Summary

Think of this system as a team of detectives:

Scouts find everyone who might be the suspect.
A Sniper pinpoints the most likely suspects with high accuracy.
A Refinement Team checks the list, removes duplicates, filters out noise in crowded areas, and has a final expert verify the tricky cases.

This approach allows doctors to screen Pap smears faster and more reliably, helping to catch cervical cancer early and save lives.

1. Problem Statement

The paper addresses the critical challenge of automating the detection of Bethesda cells (indicative of cervical cancer or precancerous conditions) in Pap smear images. While Pap smears are a standard screening tool, their efficacy relies heavily on manual interpretation, which is labor-intensive and prone to human error (fatigue, inter-observer variability).

Key challenges identified include:

Data Limitations: A lack of large-scale, high-quality datasets reflecting biological diversity. The authors utilize the RIVA dataset, a new standardized benchmark introduced to bridge this gap.
Detection Complexity: Pap smear images contain complex cytological environments where cells vary significantly in size and morphology.
Metric Constraints: The competition (Riva Cytology Challenge, Track B) evaluates performance using mAP50-95 (mean Average Precision across IoU thresholds from 0.50 to 0.95), requiring models to be robust across varying degrees of localization precision.

2. Methodology

The authors propose a multi-stage optimization pipeline that combines an ensemble of detection models with a refinement stage. The architecture is divided into three main phases:

A. Pre-processing and Model Training

Annotation Resizing: The original dataset used uniform $100\times100$ pixel bounding boxes, which often failed to match actual cell morphology. The authors resized annotations to variable dimensions ( $10\times10$ to $120\times120$ ) while preserving center coordinates to optimize center localization. Empirical testing identified $20\times20$ and $50\times50$ as optimal scales.
YOLO Ensemble: Two YOLOv8n (Nano) models were trained on input resolutions of $1024\times1024$ with different target box sizes ( $20\times20$ and $50\times50$ ). These models were selected for high recall, prioritizing the detection of all potential cells over precision initially.
U-Net Heatmap Regression: A U-Net model with a ResNet34 backbone was trained as a heatmap regressor. Instead of bounding boxes, it predicted 2D Gaussian kernels centered on cell nuclei. This approach yielded lower recall but significantly higher precision compared to YOLO.

B. Ensemble Strategy

The outputs of the models were merged using a distance-based logic:

First Ensemble: Combined the two YOLO models. Detections within a 12-pixel Euclidean distance were averaged (coordinates and confidence). Single-model detections were kept only if confidence $> 0.35$ .
Second Ensemble: Merged the YOLO ensemble output with the U-Net predictions.
- Coincident detections (within 12 pixels) had their scores averaged.
- Crucial Step: The confidence threshold for unmatched detections was set to 0, ensuring that all unique predictions from either the YOLO ensemble or the U-Net were retained to maximize recall.

C. Three-Stage Optimization Pipeline

To refine the raw ensemble output and remove background noise, a post-processing pipeline was applied:

Non-Maximum Suppression (NMS): Applied with a high IoU threshold of 0.75. This was chosen to be permissive to preserve densely packed distinct cells while still removing obvious duplicates, optimizing for the mAP50-95 metric which rewards tight localization across a wide IoU range.
Spatial Density Filtering: Images were divided into a $4\times4$ $4 \times 4$ grid.
- High-density quadrants ( $\ge30$ detections): Applied a strict confidence threshold of 0.1 to filter spurious noise.
- Low-density quadrants ( $<30$ detections): Applied a very relaxed threshold of 0.001 to avoid missing isolated abnormal cells.
Binary Classification Refinement: A binary classifier (EfficientNet-B0) was trained to distinguish legitimate cells from background artifacts. It was trained on "hard negatives" (predictions with IoU $<0.1$ to ground truth). During inference, a threshold of 0.05 was applied to low-confidence ensemble predictions ( $<0.01$ ) to filter false positives.

3. Key Contributions

Novel Ensemble Architecture: Successfully integrated object detection (YOLO) with semantic segmentation/heatmap regression (U-Net) to balance high recall and high precision.
Adaptive Annotation Strategy: Developed a method to dynamically resize bounding boxes during training to better fit cell morphology, improving center localization.
Context-Aware Post-Processing: Introduced a spatial density filtering mechanism that adapts confidence thresholds based on local cell density, addressing the trade-off between noise removal and detecting rare, isolated cells.
Competition Performance: Achieved 2nd place in the Riva Cytology Challenge (Track B) with a mAP50-95 score of 0.5909 on the test set.

4. Results

Evaluation was performed on the RIVA validation set (131 images). Key findings include:

Individual Model Performance:
- YOLOv8n ( $50\times50$ ): Highest recall (0.9937) but very low precision (0.0665) due to high false positives (37,398 FP).
- U-Net: Lower recall (0.7983) but the highest precision (0.694) and F1-score (0.7425) among individual models.
Ensemble Performance:
- The Final Prediction (after the full pipeline) achieved a mAP50-95 of 0.6232 on the validation set.
- It maintained a high recall (0.9485) while significantly reducing false positives compared to standalone YOLO (11,410 FP vs. 37,398 FP).
Trade-off Analysis: The paper notes that optimizing strictly for mAP50-95 requires lower confidence thresholds, which increases false positives. However, a modest increase in the confidence threshold could significantly improve the F1-score (precision/recall balance) with only a marginal drop in mAP, suggesting the model scores are well-calibrated for clinical utility where precision is paramount.

5. Significance

This work demonstrates that a multi-stage pipeline combining diverse deep learning architectures (detection + segmentation) with context-aware post-processing can effectively tackle the complexities of Pap smear cytology.

Clinical Relevance: While the competition metric (mAP) favors high recall, the authors highlight that in clinical screening, a model with higher precision (fewer false alarms) might be more practical for diagnostic assistance. Their framework is flexible enough to allow tuning for this specific clinical need.
Open Source: The authors have made their implementation and source code publicly available, fostering reproducibility and further research in automated cervical cancer screening.