MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events

1. Problem Statement

The paper addresses critical challenges in applying Machine Learning (ML) to high-stakes healthcare domains, specifically Multi-Label Text Classification (MLTC) for medical device adverse event reporting.

• Complexity of MLTC: The task involves selecting multiple labels from a large, hierarchical, and interdependent set. Real-world data exhibits severe class imbalance (long-tail distribution) and label dependencies, violating standard independence assumptions.
• Benchmark Saturation & Contamination: Existing MLTC benchmarks are often static, saturated, or contaminated by data present in Large Language Model (LLM) pre-training corpora. This makes it difficult to distinguish genuine reasoning from memorization.
• Lack of Uncertainty Quantification (UQ): In high-stakes settings like healthcare, predictive accuracy alone is insufficient. Systems require reliable UQ to flag ambiguous cases for human review. However, current benchmarks rarely evaluate UQ across diverse architectures (encoder vs. decoder) and learning paradigms (fine-tuning vs. prompting).
• The "Reasoning" Gap: It is unclear whether specialized small models or large "reasoning" foundation models are better suited for these tasks, nor is it known how reliable the confidence scores of these models are.

2. Methodology

A. The MADE Benchmark (Data)

The authors introduce MADE, a "living" benchmark derived from FDA Medical Device Adverse Event reports.

• Data Source: Reports from 2015 to mid-2025, mapped to IMDRF (International Medical Device Regulatory Forum) hierarchical codes.
• Structure:
  • Labels: 1,154 unique hierarchical labels across 3 levels (Product Problems and Patient Problems).
  • Distribution: Long-tailed distribution with severe imbalance (Head: >1%, Medium: 0.1–1%, Tail: 0.01–0.1%, Extreme Tail: <0.01%).
  • Temporal Split: Strict temporal separation to prevent data leakage. Training (2015–2023), Validation (early 2024), and Test (mid-2024 to mid-2025).
  • Living Nature: The dataset is continuously updated with new FDA reports, ensuring future models are tested on data post-dating their training, preventing contamination.
• Preprocessing: Deduplication, downsampling via HDBSCAN to preserve rare labels, and up-propagation of labels to include all ancestors in the hierarchy.

B. Experimental Setup

The study evaluates over 20 models across four learning paradigms:

1. Discriminative Fine-Tuning: Encoder-only (Ettin family) and Decoder-only (Llama base) models with a classification head.
2. Generative Fine-Tuning: Decoder-only models trained to generate labels as tokens.
3. Few-Shot Prompting (In-Context Learning): Using instruction-tuned and "reasoning" (thinking) models (e.g., GPT-4.1, GPT-5, DeepSeek-R1, Qwen3) with 10-shot examples.
4. Models Tested: Ranging from 150M parameters (Ettin) to 235B+ parameters (Qwen, GPT-5), including local and API-accessible models.

C. Uncertainty Quantification (UQ) Methods

The paper systematically evaluates three categories of UQ:

1. Information-Based ($U_{info}$): Token-level metrics including Entropy, Perplexity, Average/Max Log-Probability, and Improbability.
2. Consistency-Based ($U_{cons}$): Measures output variability across stochastic forward passes using the sum of eigenvalues of a graph Laplacian.
3. Self-Verbalized ($U_{self}$): Prompting the model to output a confidence score directly.

• Evaluation Metrics:
  • Prediction Rejection Rate (PRR): Measures how effectively high-uncertainty scores identify and allow the rejection of incorrect predictions.
  • Spearman Correlation ($\rho$): Correlation between uncertainty and correctness (negative is desirable).
  • Positive Class Expected Calibration Error ($ECE^+$): Specifically measures under-confidence on true positive instances, addressing issues with standard ECE in imbalanced multi-label settings.

3. Key Contributions

1. MADE Benchmark: A reproducible, contamination-free, living benchmark for MLTC in healthcare with strict temporal splits and hierarchical, long-tailed labels.
2. Comprehensive Baselines: Establishes performance baselines for >20 models across discriminative, generative, and prompting paradigms.
3. UQ Analysis: Provides the first systematic comparison of UQ methods (information, consistency, self-verbalized) specifically for MLTC in a medical context.
4. Critical Trade-off Discovery: Reveals that while large reasoning models improve performance on rare labels, they fail significantly in uncertainty quantification.

4. Key Results

Predictive Performance

• Fine-Tuning vs. Prompting: Discriminative fine-tuning of decoder models (specifically Llama-3.1-8B-Base) achieved the highest overall Macro F1 (0.54) and best head-to-tail accuracy. It consistently outperformed generative fine-tuning of similar-sized models.
• Reasoning Models: Large reasoning models (e.g., GPT-5, Qwen3-235B-Thinking) excelled on Extreme Tail (ET) classes (F1 ~0.34) but lagged behind fine-tuned models on common (Head) classes.
• Prompting: Few-shot prompting generally underperformed compared to fine-tuning, with high variance across models.

Uncertainty Quantification (UQ)

• Generative Fine-Tuning: Surprisingly, generative fine-tuning yielded the most reliable UQ (highest PRR, best $\rho$, and lowest $ECE^+$), outperforming discriminative fine-tuning and prompting.
• Discriminative Fine-Tuning: Provided stable and competitive UQ, particularly for smaller models (e.g., Llama-3.2-1B).
• Reasoning Models: Despite high predictive accuracy on rare labels, reasoning models exhibited surprisingly weak UQ. Their uncertainty scores were near-random (PRR ~0.23), making them unreliable for selective prediction or human-in-the-loop triage without further calibration.
• Self-Verbalized Confidence: Proved to be a poor proxy for actual uncertainty and should be avoided.
• Best UQ Method: Entropy-based token metrics ($U_{info}$) performed best. Combining with consistency metrics ($U_{cons}$) offered marginal gains at high computational cost.

Trade-offs

• Head vs. Tail: A persistent trade-off exists: models optimized for rare labels (reasoning/prompting) sacrifice performance on common labels, while fine-tuned discriminative models balance both but struggle slightly more with the extreme tail than reasoning models.
• Performance vs. UQ: High predictive performance does not guarantee reliable uncertainty estimates. The best UQ came from generative fine-tuning, while the best reasoning models had the worst UQ.

5. Significance and Conclusion

• Practical Guidance: For healthcare applications requiring both accuracy and safety (via UQ), fine-tuned models (especially generative fine-tuning) are the superior choice over prompting or using raw reasoning models.
• Reliability of "Thinking" Models: The study highlights a critical flaw in current "reasoning" models: they can solve difficult classification tasks but cannot reliably signal when they are wrong, posing a risk for automated deployment.
• Future Research: The paper identifies the need for:
  • Investigating why generative fine-tuning improves UQ.
  • Developing methods to calibrate reasoning models.
  • Training smaller encoder models with reinforcement learning to achieve the "best of both worlds" (high accuracy + reliable UQ + low cost).
• Community Resource: By releasing MADE, the authors provide a foundation to prevent benchmark saturation and enable continuous evaluation of future LLMs on fresh, real-world medical data.

In summary, the paper argues that while frontier LLMs show promise in handling complex, rare medical events, task-specific fine-tuning remains the most robust approach for building reliable, safe, and uncertainty-aware ML systems in healthcare.