Technical Summary: RAGPPI – A Retrieval-Augmented Generation Benchmark for Protein–Protein Interactions in Drug Discovery

1. Problem Statement

In drug discovery, Target Identification (Target ID) is a critical yet resource-intensive phase requiring the identification of protein-protein interactions (PPIs) that lead to therapeutic effects. Specifically, researchers must determine the biological, functional, or physical impacts of a drug binding to a target protein (TP) on an initial protein (IP). Given the vastness of the human proteome (billions of candidates), manually scanning literature to map these pathways is slow and expensive.

While Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) frameworks have been applied to this domain to infer PPI impacts, they face significant challenges:

• Hallucinations: LLMs often generate unreliable results not grounded in scientific fact.
• Lack of Specialized Benchmarks: There is currently no standardized benchmark to evaluate RAG systems specifically on the factual accuracy and reasoning depth required for scientific discovery in Target ID. Existing benchmarks do not adequately address the multi-step reasoning chains (Entity Identification $\to$ Mechanism Elucidation $\to$ Therapeutic Impact) necessary for valid biological inference.

2. Methodology

The authors developed RAGPPI, a benchmark designed to evaluate RAG systems on generating accurate biological impacts of PPIs. The methodology involved three primary phases:

A. Expert-Driven Task Formulation

To ensure the benchmark reflects real-world workflows, the authors conducted interviews with 18 domain experts (including Ph.D. researchers in chemical engineering and drug discovery).

• Query Template: Experts prioritized a unified query: "According to the abstract, what biological, functional, or physical effects result from [PPI]?" This forces models to synthesize interaction partners, functional mechanisms (e.g., inhibition, activation), and therapeutic implications.
• Data Source: The benchmark utilizes BioGRID, the most widely adopted PPI dataset.
• Stratification: To ensure generalizability, the dataset was stratified by literature frequency (High, Medium, Low) and PPI types (e.g., Reaction, Binding, Inhibition), preventing bias toward well-studied diseases like cancer or COVID-19.

B. Dataset Construction

The dataset was built in two stages to balance quality and scale:

1. Gold-Standard Dataset (500 QA pairs):
   • Generated using GPT-4o based on abstracts from BioGRID.
   • Validated by experts through a two-step user study (preliminary and main).
   • Experts categorized responses as Perfect, Acceptable, or Inaccurate. "Perfect" and "Acceptable" were merged into a Correct label, while "Inaccurate" became Incorrect.
   • Experts revised 96.4% of non-perfect answers to ensure the ground truth reflected verified knowledge rather than model biases.
2. Silver-Standard Dataset (3,720 QA pairs):
   • To scale the dataset without requiring further expert annotation, the authors developed an Ensemble Auto-Evaluation LLM.
   • This ensemble consists of three specialized sub-models (all GPT-4o based) that evaluate answers based on expert-derived features:
     • F1 (Average Similarity): The mean cosine similarity between atomic facts of the answer and the source abstract.
     • F2 (Low-Similarity Outliers): The count of atomic facts in the answer that fall below a similarity threshold relative to the abstract.
     • M3 (Global Semantic Alignment): Evaluation of the answer's alignment with the ground-truth text.
   • The ensemble uses majority voting. For the silver dataset, an answer was accepted as Ground Truth (GT) only if both F1 and F2 sub-models rated it as accurate, prioritizing factual grounding over stylistic similarity.

C. Evaluation Metrics

The benchmark evaluates models using:

• Atomic Fact Similarity (F1): Mean cosine similarity of atomic facts.
• Low-Similarity Fact Counts (F2): Number of hallucinated or unsupported facts.
• Accuracy: Determined by the ensemble model (M1, M2, M3, and the final Ensemble vote).

3. Key Contributions

1. Expert-Grounded Benchmark Design: The first benchmark specifically for Target ID, derived from interviews with 18 experts. It establishes a unified query template that demands multi-step reasoning (Entity $\to$ Mechanism $\to$ Impact) and balances PPI types and literature frequencies.
2. RAGPPI Dataset: A factual QA benchmark comprising 4,420 question-answer pairs (500 Gold, 3,720 Silver) focusing on the biological impacts of PPIs.
3. Scalable Auto-Evaluation Method: An ensemble LLM framework that captures expert labeling characteristics (F1 and F2) to automatically label a large-scale "silver" dataset with 93.71% accuracy compared to expert judgments, enabling broader applicability without continuous expert involvement.

4. Experimental Results

The authors evaluated eight systems (general LLMs, open-source biomedical LLMs, and various RAG architectures) on RAGPPI.

• Corpus Sensitivity: RAG models using the authors' curated paper corpus significantly outperformed those using default corpora. For instance, GraPPI (a biomedical RAG system) improved its ensemble accuracy from 13.44% (default corpus) to 74.80% (curated corpus).
• Model Performance:
  • GraPPI with the curated corpus achieved the best overall performance (F1 = 0.76, Ensemble Accuracy = 74.80%), demonstrating the synergy between domain-specific models and domain-specific corpora.
  • General LLMs (e.g., GPT-4.1, Gemini) without retrieval often outperformed RAG systems with default corpora, suggesting that ill-suited retrieval sources can degrade performance. However, LLMs tended to produce answers that were factually aligned but incomplete (high F1/F2 but low M3 alignment with full scope).
• Ensemble Effectiveness: The ensemble evaluation approach (combining atomic fact verification and holistic alignment) proved necessary to distinguish between models that merely guess correct outcomes versus those that correctly synthesize mechanisms.

5. Significance and Claims

The paper claims that RAGPPI fills a critical gap in the NLP and biomedical communities by providing a specialized benchmark for evaluating RAG systems in drug discovery. Its significance lies in:

• Methodological Framework: Providing a robust, expert-driven process for building scientific discovery benchmarks that balance retrieval and generation evaluation.
• Real-World Applicability: Demonstrating that retrieval quality is a primary driver of performance in Target ID tasks, validating the need for specialized corpora in biomedical RAG.
• Community Resource: Offering a publicly available dataset and code to support the development of more reliable, hallucination-resistant AI tools for target identification.

The authors maintain a modest stance, acknowledging limitations such as the static nature of the dataset (lacking temporal evolution of scientific findings), the reliance on abstracts rather than full-text figures, and the potential insufficiency of 500 gold samples to cover all biological PPI impacts. They position RAGPPI as a foundational step toward more complex, multi-hop reasoning tasks in scientific discovery.