Technical Summary: Uncertainty-Aware Deep Learning for Genomics Applications

Problem Statement

Deep learning has become the standard for genomic applications, such as identifying functional elements and predicting variant effects. However, the reliability of Uncertainty Quantification (UQ) in these domains remains under-explored. Genomic data presents unique challenges for UQ, including experimental noise, incorrect labels, out-of-distribution (OOD) samples, and severe class imbalance. Furthermore, reliable ground truth is often unavailable, making it difficult to assess uncertainty scores independently of predictive accuracy. Existing benchmarks in computer vision do not transfer well to genomics, and current methods often rely on labeled data for calibration, which is scarce in biological contexts. There is a lack of systematic comparison of UQ methods across the specific dataset conditions characteristic of genomics.

Methodology

The authors present an empirical framework to evaluate three common UQ methods: Deep Ensembles (ENS), Bayesian Neural Networks (BNN), and Monte Carlo Dropout (MC-dropout). The study focuses on two distinct genomic modalities:

1. Sequence-to-Activity Models: Predicting protein binding (Transcription Factors on DNA and RNA-binding proteins on RNA) from sequence data.
2. Single-Cell Expression Analysis: Predicting cell types from single-cell gene expression data.

Experimental Design

To control for specific sources of uncertainty, the authors utilized both simulated and real datasets:

• Simulated TF Data: Generated using SimDNA with 150-nucleotide sequences. Scenarios included clean balanced data, data with missing sequence motifs, and data with 30% label noise (swapping labels between JUN and MEF2A).
• Real RBP Data: Derived from PAR-CLIP experiments (HEK293 cell line) for three RBPs (MBNL1, PUM2, QKI). Settings included balanced vs. imbalanced data and single-label vs. multi-label classification. An OOD test was performed by applying models trained on human data to coronavirus genomes without retraining.
• Single-Cell Data: A subset of the GSE194122 dataset containing four cell types. The authors created balanced and imbalanced versions to test sensitivity to class depletion.

Evaluation Metrics

The study avoids relying solely on predictive accuracy, which is often confounded by UQ method performance. Instead, it employs:

• Agreement Analysis: Measuring the overlap of the top-100 most uncertain samples and calculating Kendall's tau rank correlation between uncertainty scores of different methods.
• Distributional Shifts: Using the Kolmogorov–Smirnov (KS) test to determine if the distribution of uncertainty scores shifts significantly when a specific source of uncertainty (e.g., imbalance, noise) is introduced.
• Directionality: Crucially, the authors assess not just if the uncertainty distribution shifts, but in which direction (e.g., does the model become more uncertain for the under-represented class?).

Key Contributions and Results

1. Superiority of Bayesian Neural Networks (BNN) in Genomic Contexts

The study finds that BNNs are the most reliable method for capturing uncertainty in scenarios common to genomics, specifically class imbalance and out-of-distribution data.

• Class Imbalance: When trained on imbalanced datasets, BNNs consistently increased their uncertainty scores for the depleted (minority) classes. In contrast, Deep Ensembles often became more confident (lower uncertainty) for minority classes, and MC-dropout showed erratic behavior, sometimes becoming more confident for the depleted class.
• Out-of-Distribution (OOD): When models trained on human data were tested on coronavirus genomes, BNNs flagged 36% of viral windows as unusually uncertain (above the 95th percentile of human scores). Deep Ensembles flagged 27%, while MC-dropout flagged only 7%, rendering it ineffective for OOD detection in this context.

2. Limitations of MC-Dropout

MC-dropout demonstrated significant limitations in genomic settings:

• Degenerate Scores: For a large fraction of test samples (up to 68% in some settings), MC-dropout produced an uncertainty score of exactly zero due to softmax saturation, providing no ranking information.
• Low Agreement: MC-dropout showed little to no agreement with BNN or ENS in ranking uncertain samples, particularly in complex or imbalanced settings.

3. Agreement Between Methods

BNN and Deep Ensembles showed higher agreement with each other than either did with MC-dropout. However, the authors note that agreement between methods should not be taken as a proxy for ground truth, as they rely on different posterior approximations.

4. Practical Utility of Uncertainty Scores

In a practical application involving 27 RBPs, filtering predictions based on a low uncertainty threshold (score < 0.1) improved the Average Precision (AP) for 20 of the 27 classes. The study demonstrated that uncertainty scores effectively track class size; smaller training sets resulted in higher median uncertainty scores, allowing for the selection of high-quality predictions in downstream analysis.

Significance and Claims

The paper claims to provide the first systematic comparison of UQ methods across the specific dataset conditions characterizing genomics (imbalance, noise, OOD, multi-label). Its primary significance lies in establishing guidelines for method selection:

• For Imbalanced or OOD Data: BNN is recommended despite its higher computational cost and training complexity, as it is the only method that consistently reflects these uncertainties in a useful direction (increasing uncertainty for under-represented or novel data).
• For MC-Dropout: The authors caution against its use in genomic applications where class imbalance or OOD detection is critical, due to its tendency to produce zero-variance predictions and its failure to track data scarcity.

The study emphasizes that while BNNs require more tuning and computational resources, their ability to provide reliable uncertainty estimates in label-scarce and noisy biological environments makes them a superior choice for critical downstream decision-making in genomics. The authors conclude that their framework offers a novel approach to evaluating UQ without ground-truth labels, which is essential for the adoption of uncertainty-aware deep learning in practical biological research.