💻 computer science

Unsupervised Source-Free Ranking of Biomedical Segmentation Models Under Distribution Shift

This paper introduces the first black-box-compatible, unsupervised framework for ranking biomedical segmentation models under distribution shift by leveraging prediction consistency under perturbations, enabling reliable model selection for zero-shot reuse or unsupervised domain adaptation without requiring labeled data or source information.

Original authors: Joshua Talks, Kevin Marchesini, Luca Lumetti, Federico Bolelli, Anna Kreshuk

Published 2026-04-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Joshua Talks, Kevin Marchesini, Luca Lumetti, Federico Bolelli, Anna Kreshuk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Model Zoo" Dilemma

Imagine you are a doctor or a biologist who needs to analyze medical images (like looking for cancer cells or counting mitochondria). You don't have time to train a new AI from scratch because that requires thousands of labeled images and weeks of expert work.

Instead, you go to a "Model Zoo" (like the BioImage Model Zoo). This is a massive library where researchers have uploaded thousands of pre-trained AI models. It's like walking into a library with 10,000 different maps of the world.

The Catch: You have a new map of a specific city you've never seen before (your new dataset). You need to pick the single best map from that library to navigate your city. But here's the problem:

You don't have the "answer key" (labeled data) for your new city yet.
You can't ask the model how it feels about the new city (it's a "black box").
Most existing methods to pick the best map require you to have the answer key first, which defeats the purpose.

If you pick the wrong map, your navigation fails. If you pick the right one, you save months of work.

The Solution: The "Stress Test" (CMR)

The authors of this paper introduced a new method called Consistency-Based Model Ranking (CMR). Think of it as a "Stress Test" for AI models.

Instead of asking the model, "How well do you know this city?" (which requires an answer key), they ask: "How steady are you when the world gets a little weird?"

Here is how the stress test works:

The Setup: You take a model and show it a picture from your new dataset. It draws a line around a cell.
The Perturbation (The Stress): You slightly tweak the picture. Maybe you add a little static noise (like a bad TV signal), change the brightness, or blur it slightly.
The Reaction: You ask the model to draw the line again on the tweaked picture.
The Score:
- The Bad Model: If you tweak the picture slightly, the model panics. It draws the line in a completely different place. It's unstable.
- The Good Model: If you tweak the picture, the model stays calm. It draws the line in almost the exact same place. It is consistent.

The Analogy: Imagine two people trying to find a specific tree in a foggy forest.

Person A (Bad Model): If a leaf blows in front of their eyes, they lose their way and point to a rock.
Person B (Good Model): If a leaf blows in front of their eyes, they adjust slightly but still point confidently at the tree.
The Result: Person B is the one you trust to guide you, even though you haven't checked if they are actually right yet.

Why This is a Game-Changer

The paper highlights three major "superpowers" of this method:

1. It Works for "Pixel Painters" and "Object Counters"
Previous methods were good at picking models that just said "This is a cell" (Semantic Segmentation). But they failed at models that say "This is Cell A, and that is Cell B" (Instance Segmentation).

Analogy: Old methods could tell you if a painting was "blue" or "green." They couldn't tell you if the artist did a good job painting individual blue flowers. This new method works for both.

2. It Doesn't Need to See the "Source Code"
You don't need to know how the model was built or have access to its internal brain. You just need to see what it outputs.

Analogy: You don't need to be a mechanic to test a car. You just drive it on a bumpy road. If it shakes apart, it's a bad car. If it rides smoothly, it's a good car.

3. It Works Even After "Fine-Tuning"
Sometimes, you take a model and try to adapt it to your new data without labels (Unsupervised Domain Adaptation). This is risky; sometimes the adaptation makes the model worse. This method can tell you which adapted model is the best, saving you from making a mistake.

The Results: Does it actually work?

The authors tested this on a huge variety of real-world medical tasks:

Microscopy: Counting cells, finding mitochondria, looking at fruit fly wings.
3D Scans: Looking at human jaws and teeth (from a famous dental challenge).

The Verdict:
The "Stress Test" scores correlated almost perfectly with the actual performance of the models.

When the model was consistent under stress, it was accurate on the real task.
When the model was jumpy under stress, it was inaccurate.

They even tested it on the "ToothFairy" challenge (a competition to segment teeth in 3D scans). Without seeing the test answers, their method correctly predicted which models would win the competition.

Summary

In a world where AI models are everywhere but labeled data is scarce, this paper gives us a universal compass.

Instead of needing a map of the destination to pick the right guide, we simply shake the guides. The ones who stay steady and consistent are the ones we trust to lead us to the truth. It's a simple, powerful, and "black-box friendly" way to ensure our medical AI is ready for the real world.

1. Problem Statement

In biomedical imaging, deep learning models are essential for segmentation tasks (e.g., identifying cells, nuclei, or mitochondria). However, acquiring labeled data is expensive and time-consuming. Consequently, practitioners increasingly rely on model reuse, selecting pre-trained models from repositories (like the BioImage Model Zoo) for new target datasets.

The core challenge is model selection: How does one identify the best pre-trained model for a new, unlabeled target dataset without performing costly fine-tuning or having access to ground-truth labels?

Limitations of Existing Methods:
- Supervised Transferability Metrics: Most existing methods (e.g., LEEP, LogME) require target domain labels, which defeats the purpose of zero-shot selection.
- Classification-based Extensions: Methods adapted from image classification often fail in segmentation because segmentation outputs (especially instance segmentation) lack the fixed-dimensional feature correspondence found in classification.
- Instance Segmentation Gap: No reliable unsupervised method exists for ranking instance segmentation models, as instance labels are permutation-invariant and depend on post-processing (e.g., watershed), making feature-space analysis difficult.
- Black-Box Incompatibility: Many methods require access to model weights or feature spaces, which is not possible with black-box models or containerized deployments.

2. Methodology: Consistency Model Ranking (CMR)

The authors propose CMR, the first framework for unsupervised, source-free, and model-agnostic ranking of both semantic and instance segmentation models. The core intuition is that a model well-suited for a target domain will have robust decision boundaries; thus, its predictions should remain consistent even when the input or features are slightly perturbed.

Key Components:

Perturbation Strategy:
- Input Space: Applying test-time augmentations (TTA) such as additive Gaussian noise, brightness/contrast changes, and gamma correction.
- Feature Space: Applying Test-Time Dropout (TTD) to intermediate layers (e.g., bottleneck or skip connections) while keeping weights frozen. This does not require retraining.
Consistency Metrics:
The method compares the original prediction ( $\hat{y}$ ) with the perturbed prediction ( $\tilde{y}$ ) to calculate a consistency score.
- For Semantic Segmentation:
  - CMR-EI (Soft): An extension of Effective Invariance. It measures the overlap between the softmax probabilities of the original and perturbed predictions, restricted to foreground pixels to handle class imbalance.
  - CMR-NHD (Hard): Based on the Normalized Hamming Distance. It counts the number of pixel label flips between the original and perturbed binary masks. This is equivalent to IoU but focuses on stability.
- For Instance Segmentation:
  - CMR-ARS: Since instance labels are arbitrary, pixel-wise comparison is invalid. The authors use the Adapted Rand Score (ARS), which measures the agreement of pixel pairs. If two pixels belong to the same instance in the original prediction, they should also belong to the same instance in the perturbed prediction. This penalizes "split" and "merge" errors.
Ranking Process:
For a set of candidate models, the consistency score is calculated for each image in the target dataset (averaged over multiple perturbations). The final model ranking is determined by the median consistency score across the dataset.

3. Key Contributions

First Unsupervised Source-Free Framework: Introduces a method that ranks models without target labels, source data, or model training assumptions.
Unified Approach for Semantic and Instance Segmentation: Unlike previous works limited to classification or semantic segmentation, CMR handles instance segmentation via the Rand Index, addressing a significant gap in the literature.
Black-Box Compatibility: The method operates solely on model outputs (predictions), making it applicable to black-box models, APIs, and heterogeneous model zoos.
Comprehensive Evaluation: Validated across diverse biomedical tasks (nuclei, cells, mitochondria) and modalities (light/electron microscopy, 3D CBCT), including both direct zero-shot transfer and post-Unsupervised Domain Adaptation (UDA) scenarios.

4. Experimental Results

The authors evaluated CMR against state-of-the-art baselines (e.g., Transfer Score, LEEP, LogME, NuNo, Dispersion) across multiple datasets.

Semantic Segmentation:
- CMR achieved strong correlations (Kendall's $\tau \approx 0.7-0.9$ ) with true target performance (F1-score).
- Failure of Baselines: Supervised metrics (even with labels) like LEEP and LogME failed to correlate reliably with segmentation performance in biomedical domains. Classification-derived methods performed poorly.
- Robustness: CMR remained effective across various perturbation strengths and both input/feature space perturbations.
Instance Segmentation:
- CMR-ARS significantly outperformed the only existing baseline for instance segmentation (SEG), which relies on object centroids and fails to capture shape irregularities.
- CMR successfully ranked diverse architectures, including generalist models (µSAM, Cellpose-SAM) and specialist U-Nets, reproducing ground-truth rankings without labels.
3D Multi-class Segmentation (ToothFairy Challenge):
- On the ToothFairy2 dataset (42 classes), CMR successfully reproduced the challenge's official ranking of submissions without access to test labels.
- CMR was significantly more memory-efficient than competitors, requiring only the predicted class map rather than full softmax probability volumes (critical for 3D data).
Post-UDA Ranking:
- CMR effectively ranked models after Unsupervised Domain Adaptation (using Mean Teacher and AdaBN), outperforming the Transfer Score baseline. It successfully identified models that adapted well versus those that suffered from "mode collapse" or class confusion.

5. Significance and Impact

Practical Deployment: CMR provides a practical tool for biomedical researchers to select the best pre-trained model from a repository for a new dataset without the prohibitive cost of annotation or fine-tuning.
Scalability: By being model-agnostic and source-free, it supports the growing ecosystem of large generalist models and community-driven model zoos.
Theoretical Insight: The work demonstrates that prediction consistency under perturbation is a more reliable proxy for transferability in segmentation than feature-space geometry or classification-based metrics.
Efficiency: The method requires only two inference passes per image (original + one perturbation), making it computationally lightweight compared to ensemble-based or training-heavy alternatives.

In conclusion, this paper establishes a new standard for model selection in biomedical imaging, solving the critical bottleneck of "which model to use" in a zero-shot, unsupervised setting.