DrugAgent: Reliable Multi-Agent Integration of Conflicting Biomedical Evidence for Drug-Target Interaction Assessment

Technical Summary: DrugAgent

Problem Statement
Drug-target interaction (DTI) assessment is a critical task in early-stage drug discovery that requires synthesizing heterogeneous data sources: quantitative predictions from machine learning (ML) models, curated relationships from knowledge graphs (KGs), and experimental findings from scientific literature. These sources are often complementary but imperfect. ML models may lack calibration outside their training distribution; KGs may imply connectivity without direct binding; and literature evidence is unevenly distributed and may contain conflicting or assay-specific findings. Current approaches typically rely on single modalities (e.g., numeric ML predictions) or lack mechanisms to explicitly reconcile agreement, conflict, and uncertainty across these diverse provenances. There is a need for a system that can integrate these sources into an interpretable, evidence-grounded assessment rather than functioning solely as a standalone predictor.

Methodology
The authors introduce DrugAgent, a large language model (LLM)-based multi-agent system designed to integrate heterogeneous biomedical evidence for DTI assessment. The system employs a structured, modular pipeline consisting of four key components:

1. Coordinator Agent: Orchestrates the workflow, invoking specialized agents and enforcing a shared JSON schema for their outputs to ensure consistent downstream processing.
2. Specialized Evidence Agents:
   • ML Agent: Utilizes a pre-trained DeepPurpose model (message passing neural networks for drugs, CNNs for proteins) trained on BindingDB to generate quantitative affinity predictions (e.g., pKd), which are discretized into activity labels (Strong, Moderate, Weak).
   • KG Agent: Constructs a heterogeneous knowledge graph integrating DrugBank, CTD, and BindingDB. It performs multi-hop breadth-first search (max depth 5) to find mechanistic paths. Paths are scored based on length and a penalty for passing through high-degree "hub" nodes to reduce non-specific connectivity. The agent outputs structured evidence including path descriptions and confidence scores.
   • RAG Agent (Retrieval-Augmented Generation): Retrieves full-text articles from PubMed Central (Open Access) using a multi-channel strategy: drug-target pair queries, drug-only queries, and target-only queries. Queries are enriched with assay-specific terminology (e.g., IC50, Kd, inhibition). Retrieved chunks are re-ranked based on section type (favoring Results) and entity co-mentions.
3. Reasoning Agent: This module performs the core integration in two stages:
   • Structured Summarization: Converts agent outputs into a shared representation, explicitly summarizing agreement, conflict, and uncertainty across sources.
   • Rule-Guided Decision-Making: Applies a priority-based logic (majority agreement first, then tie-breaking) to generate a final label. If rules are inconclusive, a constrained LLM fallback is used. This design separates evidence synthesis from the final decision to ensure interpretability.

Key Contributions

• Multi-Agent Integration Framework: DrugAgent is the first system to jointly integrate ML predictions, path-based KG evidence, and RAG-based literature retrieval into a unified, structured reasoning process for DTI.
• Explicit Conflict Modeling: The system explicitly models and summarizes agreement and conflict across evidence sources, rather than treating them as a single input stream.
• Interpretability and Faithfulness: By converting agent outputs into structured representations and using rule-guided decision-making, DrugAgent provides reproducible rationales that are grounded in the specific evidence provided, avoiding the "black box" nature of unconstrained LLM reasoning.
• Evaluation of Evidence Availability: The study systematically analyzes how the availability of retrieved literature impacts the biological plausibility and coherence of the system's outputs.

Results
The system was evaluated on two benchmarks: a kinase inhibition dataset (900 pairs, 178 kinases, 42 inhibitors) and a Tox21 androgen receptor (AR) antagonist screening dataset (500 pairs).

• Faithfulness: In LLM-as-a-Judge (LLMaJ) evaluations, DrugAgent outputs were faithful to the input evidence in 98.8% of kinase cases, with high scores for grounding and low rates of unsupported claims.
• Biological Plausibility: 98.9% of DrugAgent outputs were judged as at least plausible. The system showed high stability, with 98% label agreement across repeated runs.
• Evidence Dynamics:
  • Conflict Resolution: When evidence sources conflicted, DrugAgent tended to assign intermediate labels (e.g., "Moderate") rather than extreme ones. In cases of full conflict, 68% of outputs were assigned "Moderate."
  • Impact of Literature: The presence of direct drug-target literature significantly improved biological plausibility and mechanistic coherence scores. Systems without RAG (e.g., ML+KG) sometimes produced higher plausibility scores but with a higher rate of "contradicted and not plausible" outputs, suggesting that literature acts as a crucial grounding constraint.
  • Stratified Performance: While the ML+KG baseline performed competitively in aggregate, DrugAgent (incorporating RAG) outperformed baselines when direct literature evidence was available. A hybrid strategy (using DrugAgent only when pair-level papers exist, otherwise ML+KG) achieved the best results on the Tox21 dataset (Accuracy: 0.67, F1: 0.67).
• Sanity Checks: Plausibility scores did not directly correlate with ground-truth activity labels, indicating that plausibility reflects the coherence of the argument given the evidence rather than a direct measure of predictive accuracy.

Significance and Claims
The authors position DrugAgent not as a standalone DTI predictor intended to outperform specialized ML models in isolation, but as a complementary tool for evidence integration. The primary significance lies in its ability to:

1. Provide heterogeneous evidence-grounded assessments that are interpretable and reproducible.
2. Offer strategies to model agreement, conflict, and uncertainty in biomedical data, addressing the challenge of reconciling distinct biological modalities and provenances.
3. Demonstrate that retrieved literature is most beneficial when direct evidence is available, highlighting the importance of evidence availability for RAG-based integration.

The paper concludes that while the system improves reliability and interpretability, it remains rule-guided rather than statistically calibrated, and future work is needed to address computational costs, data leakage concerns in pre-trained components, and broader generalization across unseen drug-target pairs and target classes.