Technical Summary: Optimizing Resources for On-the-Fly Label Estimation with Multiple Unknown Medical Experts

Problem Statement

Accurate ground truth estimation in medical screening is hindered by heterogeneity in expert opinions, which arise from differences in experience, training, and subjective judgment. While aggregating annotations from multiple experts can mitigate these inconsistencies, existing algorithms fail to align with the practical constraints of real-time medical workflows. Specifically, current methods often:

1. Assume all data is available upfront (batch processing), preventing on-the-fly labeling.
2. Require prior knowledge of expert performance or pre-labeled training data, which is unavailable in "cold start" scenarios with new expert coalitions.
3. Fail to dynamically allocate resources based on the latent difficulty of individual instances, often treating all cases with uniform annotation effort.

The core challenge is to develop an adaptive, stream-based method that infers ground truth from a continuous stream of data without prior knowledge of expert capabilities, while efficiently distributing annotation resources to minimize cost without sacrificing accuracy.

Methodology

The authors propose an adaptive algorithm designed as a modular backbone (Algorithm 1) that processes a stream of $N$ data points ($X$) using a coalition of $K$ experts ($S_K$). The algorithm operates in three iterative steps for each data point $x_n$:

1. Adaptive Expert Selection and Trust Computation (Function A)

The algorithm ranks experts based on estimated trust levels $\{r^{(k)}\}$, which are derived via Bayesian inference using a Beta distribution posterior. This allows the system to "cold start" with a uniform prior, requiring no external information about expert quality.

• Initialization: The two highest-ranked experts are queried first.
• Selection Strategies: The paper implements three Multi-Armed Bandit (MAB) strategies to balance exploration (querying less-tested experts) and exploitation (querying high-trust experts):
  • AUER (Awake Upper Estimated Reward): Prioritizes experts with high upper bounds on trust estimates.
  • Greedy Sampling: Prioritizes experts with the highest current trust estimates, with a bias toward unqueried experts.
  • Random Sampling: Ensures equal probability of querying to distribute workload evenly.

2. Coalitional Label Inference (Function B)

Based on the collected labels and trust scores, the algorithm infers a coalitional label $\hat{z}^*_n$ and a confidence score $c^*_n$.

• Inference Model: Assuming independent errors, the probability of a label being correct is calculated by weighting expert votes by their estimated accuracy ($r_i$). Incorrect votes are assumed to be uniformly distributed across other classes.
• Stopping Criterion: If the confidence score $c^*_n$ exceeds a predefined threshold $\tau$, the label is accepted. Otherwise, the algorithm queries the next highest-ranked unqueried expert and recalculates the label and confidence. This process continues until the threshold is met or all experts are queried.

3. Parameter Update (Function C)

After processing a data point, the algorithm updates the parameters of the experts (number of queries and successful answers) using the Expectation-Maximization (EM) algorithm. This step retrospectively refines the trust estimates for future decisions, allowing the system to adapt to the coalition's performance over time.

Key Contributions

1. Adaptive Stream-Based Framework: The paper introduces a method that supports on-the-fly labeling of incoming data streams, addressing the continuous nature of medical screening.
2. Cold-Start Capability: The algorithm operates without prior knowledge of expert proficiency or pre-labeled data, utilizing a uniform Bayesian prior to initialize trust estimates.
3. Dynamic Resource Allocation: By querying additional experts only for instances where confidence is low, the method dynamically allocates annotation effort based on the latent difficulty of each case.
4. Modular Architecture: The proposed backbone separates expert ranking, label inference, and parameter updates into abstract functions, allowing for customization to specific task requirements.

Experimental Results

The approach was evaluated on three multi-annotator datasets with distinct modalities:

• Glioma Classification (GC): Artificially annotated histopathological images (6 experts).
• Weather Sentiment (WS): Text data annotated by 110 crowd workers.
• Music Genre (MG): Time-series audio data annotated by 44 crowd workers.

Performance Findings:

• Cost Reduction: The adaptive querying strategy reduced the number of expert queries by up to 50% compared to non-adaptive baselines (which query a fixed number of experts per sample) while achieving comparable accuracy.
• Accuracy-Cost Trade-off: Higher confidence thresholds ($\tau$) increased the number of queries and accuracy, but the adaptive method consistently outperformed fixed-query baselines in efficiency. For instance, on the WS dataset, achieving 0.88 accuracy required 12 experts with a non-adaptive method but only 6 on average with the adaptive approach.
• Expert Workload: The Greedy sampling strategy generally yielded the highest accuracy but resulted in the most unequal workload distribution (highest Gini coefficient). AUER provided a better balance in scenarios with uneven expert availability.
• Convergence: The algorithm demonstrated rapid adaptation; the average number of queried experts dropped quickly as the system accumulated information about expert reliability, stabilizing at lower query counts.

Significance and Claims

The paper claims that existing ground truth inference algorithms do not meet the specific demands of real-time medical annotation, where prior knowledge is scarce and difficult cases require targeted attention. By simultaneously addressing stream-based labeling, cold-start capability, and adaptive resource allocation, the proposed method offers a practical solution for clinical settings.

The authors emphasize that their approach enables a proper distribution of annotation resources, ensuring that more ambiguous or difficult cases receive greater expert attention while minimizing the overhead for straightforward cases. The framework is presented as a foundation for future research, with the code made publicly available to encourage reproducibility and further development in medical ground truth inference. The study acknowledges limitations, such as the reliance on synthetic or crowd-sourced datasets due to the scarcity of large-scale clinician-annotated datasets with known ground truth, and notes that future work will focus on clinical validation and extending the framework to structured prediction tasks.