Biological rationales from language models enable leakage-resistant forecasts of target-indication success

Technical Summary: PRIORITI – Leakage-Resistant Forecasting of Target-Indication Success

Problem Statement
The translation of biological insights into effective medicines remains slow and failure-prone. A critical bottleneck is the prioritization of target-indication (T-I) hypotheses: determining which biologically plausible relationships warrant substantial translational investment. Existing approaches face significant limitations. Curated resources are labor-intensive and uneven, while many computational models compress complex biological arguments into scalar scores or latent factors. Furthermore, models predicting drug approval or clinical trial outcomes often rely on asset-level features (e.g., molecule identity, sponsor, trial design, regulatory history) that arise after the initial T-I decision, creating a "leakage" problem where models predict program prognosis rather than the validity of the underlying biological hypothesis. Large Language Models (LLMs) offer a potential for synthesizing dispersed knowledge but, when used as direct forecasters, risk conflating evidence retrieval with probability estimation, suffering from uncalibrated confidence and potential data contamination from known development outcomes.

Methodology: The PRIORITI Framework
The authors present PRIORITI (Prospective Rationale-Informed Outcome Reasoning for Integrated Target-indication Intelligence), a framework designed to separate upstream biological evidence synthesis from outcome prediction to prevent data leakage.

1. Evidence Synthesis (LLM as Synthesizer):

   • Input: Only the target gene and disease indication.
   • Process: A domain-instructed LLM (GPT-5) generates structured, drug-agnostic assessments across four prespecified evidence channels:
     1. Human Genetic Causality (HGC)
     2. Biological Coherence (BIO_COH)
     3. Functional and Animal Support (FUNC_ANIMAL)
     4. Cross-Source Consistency (CONSISTENCY)
   • Constraints: The LLM is explicitly prohibited from accessing drug identities, trial outcomes, regulatory status, or external tools. It operates strictly on its internal knowledge cutoff.
   • Output: The LLM produces scalar scores (0–6) for each channel and an integrated causal evidence score (ICES), along with natural language rationales.

2. Feature Construction:

   • Scalar Scores: The five evidence scores (four channels + ICES).
   • Semantic Embeddings: The natural language rationales are embedded to capture the semantic structure of the biological argument (e.g., how conflicting evidence is reconciled).
   • Static Context: Gene functional summaries and disease definitions are embedded independently to encode target and disease identity without relying on the LLM's reasoning step.

3. Supervised Prediction:

   • A supervised model (a weighted AutoGluon ensemble of LightGBM, CatBoost, RealMLP, TabDPT, and RealTabPFN-v2.5) is trained on the combined feature set.
   • Training Data: 6,784 T-I pairs from a historical cohort (outcomes resolved before October 1, 2024), curated from Citeline Pharmaprojects and aligned with the Minikel et al. dataset.
   • Objective: Predict the probability that at least one development programme for a T-I pair will achieve clinical success (Launched, Registered, or Pre-registration).

4. Auditable Explanation:

   • A separate, label-blinded LLM explainer converts the locked model predictions into decision-oriented rationales. This step occurs after prediction is finalized to ensure the explanation does not influence the forecast.

Key Contributions

• Leakage-Resistant Architecture: PRIORITI is the first framework to combine LLM-synthesized, drug-agnostic biological rationales with supervised outcome learning at the T-I level, strictly separating evidence synthesis from probability estimation.
• Large-Scale Dataset: The study utilizes one of the largest assembled cohorts for this task, comprising 23,463 unique T-I pairs, with specific training, held-out, and out-of-time benchmarks.
• Semantic vs. Scalar Evidence: The framework demonstrates that the semantic structure of synthesized rationales (captured via embeddings) provides significantly more predictive signal than scalar evidence scores alone.
• Strict Evaluation Protocols: The study employs rigorous evaluation strategies, including entity-disjoint splits (target-disjoint, indication-disjoint, and doubly-disjoint) and a strictly post-cutoff out-of-time benchmark (765 pairs), to test generalization beyond memorized entities and temporal shifts.

Results

• Historical Performance: On a held-out historical test set (N=1,696), PRIORITI achieved a ROC-AUC of 0.883 and an Expected Calibration Error (ECE) of 0.023. This outperformed a static entity-context baseline (ROC-AUC 0.863) and a human-curated genetic-evidence reference (ROC-AUC 0.530).
• Generalization: In doubly-disjoint evaluations (where neither target nor indication appeared in training), PRIORITI maintained robust performance (ROC-AUC 0.768), significantly outperforming the static baseline (0.658). This indicates the model captures transferable biological relationships rather than just entity identity.
• Out-of-Time Validation: On the strictly post-cutoff benchmark (N=765), PRIORITI retained discrimination (ROC-AUC 0.817) and calibration (ECE 0.044), outperforming the static baseline and direct zero-shot LLM forecasting.
• Zero-Shot Comparison: Direct zero-shot LLM forecasting (even with evidence summaries) performed poorly (Balanced Accuracy ~0.55–0.57) and was poorly calibrated (ECE > 0.10), confirming that the LLM functions better as an evidence synthesizer than a standalone forecaster.
• Prioritization Utility: The model showed strong enrichment for successes in the top-ranked predictions. Among the top 3% of out-of-time predictions, the observed success rate was 0.478 (4.1-fold enrichment over prevalence), compared to 0.261 for the baseline. Conversely, the bottom 20% had a negative predictive value of 1.000.
• Ablation Studies: The LLM-derived rationale embeddings were the strongest individual predictor (ROC-AUC 0.867), outperforming gene or indication embeddings alone.

Significance and Claims
The paper claims that PRIORITI provides a calibrated, biology-first probability for T-I prioritization before downstream translational investment. By decoupling the LLM's role (synthesizing inspectable biological evidence) from the predictive model's role (mapping evidence patterns to outcomes), the framework avoids the pitfalls of direct LLM forecasting, such as latent memory of known outcomes and poor calibration.

The authors emphasize that the framework estimates translational biological support, not the causal probability of regulatory success for a specific asset. It is designed to support target prioritization, indication expansion, and portfolio triage. The study concludes that while clinical success can depend on factors outside the T-I pair (e.g., multi-target pharmacology, patient selection), PRIORITI successfully identifies hypotheses with strong upstream biological justification, offering a scalable, auditable alternative to manual curation and static scoring methods.