Technical Summary: Genomic Model Research Must Move Beyond Anecdotal Evaluation of Interpretability Methods

Problem Statement

While deep learning models have achieved significant predictive success in genomics (e.g., predicting transcription factor binding, gene expression, and enhancer activity), their "black box" nature hinders the elucidation of underlying biological mechanisms. To bridge this gap, researchers increasingly employ Interpretable Machine Learning (IML) techniques. However, the field suffers from a pervasive reliance on anecdotal evaluation. Current practices are characterized by:

1. Single-method reliance: The vast majority of studies use only one IML method without justification or comparison.
2. Subjective validation: Evaluations often rely on visual inspection or cherry-picked "successful" cases that align with prior knowledge, while omitting failures.
3. Lack of rigor: There is no consensus on how to systematically assess the consistency, faithfulness, or biological validity of explanations.

This lack of rigorous evaluation risks fostering unwarranted confidence in IML outputs, potentially leading to misallocation of experimental resources and erroneous biological conclusions.

Methodology

The authors employ a two-pronged approach to quantify and demonstrate the deficiencies of current practices:

1. Large-Scale Mapping Study

The authors conducted a systematic mapping of 3,575 papers published between 2010 and 2025 that utilize IML in genomics.

• Data Extraction: Using a large language model (Gemini-3-flash-preview) with few-shot in-context learning and self-reflection prompts, they extracted structured data regarding the number of IML methods used, validation strategies, the number of reported "successful" instances, and the reporting of failure modes.
• Validation: A manual review of 100 randomly sampled papers confirmed an overall extraction agreement rate of 98%.

2. Empirical Audit via Benchmarking

To move beyond abstract critique, the authors designed a controlled benchmarking study on transcription factor (TF) binding prediction.

• Models: Three divergent genomic foundation models were evaluated: DNABERT-2 (Transformer), HyenaDNA (Convolution/SSM), and Nucleotide Transformer v3 (Hybrid).
• IML Methods: Five common methods were tested: DeepLIFT, Integrated Gradients (IG), In Silico Mutagenesis (ISM), LIME, and Categorical Jacobian (CJ).
• Datasets: A large-scale dataset (~289k training, 11.5k test sequences) was constructed for five TFs (CTCF, MAX, SP1, TBP, GATA1) with distinct characteristics (e.g., high-signal motifs, compositional biases, short/degenerate motifs). Ground truth was established using the UniBind database.
• Evaluation Metrics:
  • Consistency: Spearman rank correlation and Jaccard similarity of top-20 important positions between method pairs.
  • Faithfulness: Perturbation analysis (Sequential Deletion/MoRF and Sequential Insertion) measuring how well explanations reflect the model's decision process via Area Under the Curve (AUC).
  • Biological Validity: "Perception" score, measuring the overlap between the highest attribution regions and experimentally validated motif coordinates.

Key Contributions

1. Systematic Evidence of Anecdotal Reliance: The mapping study reveals that 86% of studies use a single IML method, nearly 50% rely on subjective visual inspection, and 90% report no failure modes.
2. Demonstration of Three Failure Modes: The empirical audit provides concrete evidence that current practices fail to detect:
   • Inconsistency: Different IML methods yield contradictory explanations for the same prediction.
   • Unfaithfulness: Explanations often fail to reflect the model's actual internal decision logic.
   • Biological Misalignment: "Plausible" explanations frequently fail to match causal ground truth, particularly in challenging tasks involving compositional biases or short motifs.
3. Proposed Tiered Framework: The authors propose a practical, tiered framework for rigorous evaluation, ranging from low-effort computational checks to high-effort wet-lab validation, to guide future research.

Key Results

1. Mapping Study Findings

• Method Usage: 86% of studies employed only one IML method.
• Validation: 49.36% relied on visual inspection; only 2.81% conducted wet-lab experiments.
• Reporting: 90.24% of papers reported no failure modes. The majority of papers reported only 1–3 "successful" instances, suggesting cherry-picking.

2. Empirical Audit Findings

• Inconsistency: Different IML methods produced drastically different explanations. The average Spearman rank correlation between method pairs was consistently below 0.4. Even for the most critical positions (top-20), the maximum Jaccard similarity was below 0.5, indicating methods agree on fewer than half of the most important features.
• Unfaithfulness: Faithfulness varied significantly across methods and tasks. While some methods (e.g., ISM, IG) showed better performance in specific contexts, no method consistently excelled. Notably, on tasks with compositional biases (SP1, TBP), all methods showed high AUC values near 1.0 in deletion experiments, indicating they failed to identify the features driving predictions.
• Biological Misalignment:
  • Positive Controls: For CTCF (high signal), methods achieved reasonable alignment (median Perception > 0.5).
  • Challenging Tasks: For GATA1 (resolution challenge) and MAX, median scores dropped to 0.2–0.4.
  • Bias Stress Tests: For SP1 and TBP, the vast majority of sequences yielded Perception scores near zero, indicating that explanations were confounded by background nucleotide composition rather than capturing specific regulatory motifs.
  • Cherry-Picking: The study highlights that while "successful" alignment cases (like Fig 4b) exist and are often reported, failure cases (like Fig 4c) are far more common but rarely disclosed.

Significance and Claims

The paper argues that the credibility of genomic deep learning is currently undermined by a reliance on anecdotal evidence. The authors claim that:

• Theoretical guarantees are insufficient: Methods satisfying mathematical axioms (e.g., IG, SHAP) do not guarantee biological validity or model faithfulness.
• Visual inspection is flawed: It is prone to confirmation bias and cannot detect subtle failures or assess faithfulness.
• Predictive accuracy does not imply interpretability: Models can achieve high accuracy via spurious correlations (e.g., GC content) that IML methods may highlight instead of true biological signals.
• A paradigm shift is necessary: The field must move from subjective "plausibility" to systematic assessment of consistency, faithfulness, and biological validity.

The authors propose that adopting their tiered framework—starting with minimum standards like using multiple methods and reporting holistic statistics—is essential to prevent resource misallocation and ensure that IML-derived insights are reliable enough to guide experimental design and therapeutic target discovery.