Technical Summary: A Benchmark for End-to-End Zero-Shot Biomedical Relation Extraction with LLMs

Problem Statement

Relation extraction (RE) is a critical task in biomedical natural language processing (NLP), essential for converting unstructured scientific literature into structured knowledge graphs to drive hypothesis generation and knowledge discovery. Traditionally, RE relies on supervised learning, which necessitates labor-intensive dataset curation involving specialized annotators to identify entities and their relationships. With the rapid proliferation of biomedical literature (e.g., PubMed indexing nearly 40 million articles), the burden of manual annotation is becoming a bottleneck.

The emergence of Large Language Models (LLMs) has prompted the question of whether zero-shot relation extraction (ZSRE) can replace supervised methods, thereby eliminating the need for training data. However, ZSRE presents unique challenges compared to other NLP tasks. Unlike question answering, which often requires single-token or short-phrase outputs, biomedical RE demands the generation of long, exact, and consistently formatted structured outputs (e.g., tuples of entities and predicates). Furthermore, biomedical texts often contain complex, multi-sentence relations and numerous entity types, making it difficult for LLMs to generate outputs that strictly adhere to annotation guidelines without fine-tuning.

Methodology

The authors propose a comprehensive benchmark to evaluate the end-to-end ZSRE capabilities of frontier LLMs across seven diverse biomedical datasets.

Datasets

The study utilizes seven datasets varying in domain, text length, entity/relation type diversity, and annotation granularity (mention-level vs. entity-level):

• Mention-Level (ML): ADE (Adverse Drug Events), DCE (Drug Combination Extraction), ChemProt (Chemical-Protein interactions), and DDI (Drug-Drug Interactions).
• Entity-Level (EL): CDR (Chemical-Induced Diseases), GDA (Gene-Disease Associations), and BioRED (a complex dataset with 8 relation types and high relation density).
  These datasets range from single-sentence inputs to full abstracts, with relation counts varying from single instances to over 90 per abstract in BioRED.

Models

Three OpenAI models were evaluated:

1. GPT-4: A standard frontier model.
2. OpenAI o1: A model with enhanced test-time chain-of-thought reasoning capabilities.
3. GPT-OSS-120B: An open-weights model.

Experimental Setup

• Prompting Strategy: The models were guided using natural language instructions and output templates. To ensure structured output, the authors utilized OpenAI's API tools for JSON generation, experimenting with both "explicit" (schema-provided) and "inferred" (prompt-derived) modes.
• Evaluation Metrics: The authors developed a specific performance measure to address inconsistencies in prior literature. For Entity-Level (EL) tasks, a predicted relation is considered correct if extracted entity mentions match any gold entity mention, entity types are correct, and the relation type is correct. For Mention-Level (ML) tasks, exact span matching is required. Precision, Recall, and F1-scores were calculated across entire test sets.

Key Contributions

1. Comprehensive Benchmark: Unlike previous studies restricted to one or two datasets, this work evaluates ZSRE across seven datasets covering a spectrum of complexities, including multi-sentence relations and varying numbers of entity/predicate types.
2. End-to-End Focus: The study strictly evaluates end-to-end RE, where the model must identify entities and link them with predicates without pre-annotated entity spans, a significantly harder task than relation classification.
3. Comparative Analysis: The paper provides a direct comparison between zero-shot LLMs and state-of-the-art supervised methods, highlighting the performance gaps and specific failure modes of ZSRE.
4. Error Analysis: The authors conduct a detailed error analysis to identify specific weaknesses, such as partial entity matching, under-prediction in high-density contexts, and hallucination of domain-specific terms.

Results

The experiments reveal that while LLM-based ZSRE is approaching supervised performance on simpler tasks, significant gaps remain on complex datasets.

• Performance Trends:
  • Simple Datasets: On datasets with fewer relation types (1–2) and shorter inputs (ADE, DCE, CDR, GDA), ZSRE performance is competitive, with GPT-OSS-120B achieving the highest F1 scores (e.g., 72.5 on ADE, 73.7 on DCE).
  • Complex Datasets: Performance drops significantly on datasets with higher complexity (4–8 relation types) and longer contexts (ChemProt, DDI, BioRED). For instance, on BioRED, GPT-4 achieved only 9.1 F1, while GPT-OSS-120B reached 21.5.
• Model Comparison:
  • GPT-OSS-120B consistently outperformed GPT-4 across all datasets, often surpassing it by a margin of ~4 F1 points on average.
  • OpenAI o1 generally outperformed GPT-4 on complex datasets (ChemProt, BioRED, DDI) due to its reasoning capabilities but performed similarly to GPT-4 on simpler datasets.
  • GPT-4 generally underperformed relative to the other two models, particularly on tasks requiring complex reasoning or handling multiple relations.
• Comparison to Supervised Methods: Supervised methods using smaller, fine-tuned models generally outperform ZSRE, particularly on ChemProt (where the gap is ~40 F1 points). However, on simpler datasets like ADE, ZSRE approaches the performance of supervised baselines.

Error Analysis

The authors identified several critical failure modes:

• Under-prediction: Models tend to miss relations when an instance contains a high number of gold relations, leading to low recall.
• Partial Matching: A frequent error involves extracting entity mentions that are nearly correct but include extra words or miss necessary words (e.g., extracting "methamphetamine-induced psychosis" instead of "psychosis").
• Boundary Issues: Models struggle with exact entity boundaries, often concatenating discontinuous spans or hallucinating terms not present in the text (e.g., predicting "interleukin-10" when the text only contained "interleukin (IL)-10").
• Complexity Sensitivity: Performance degrades as the number of relation types and the density of relations increase, suggesting current LLMs struggle with the "synergistically demanding" requirements of long, exact extraction.

Significance and Claims

The paper claims that ZSRE via LLMs is a viable alternative to supervised methods for low-complexity biomedical tasks (shorter texts, fewer relation types), offering a path to automate knowledge curation without the heavy burden of annotation. However, the authors emphasize that for complex, high-density biomedical texts, ZSRE currently struggles to match supervised performance, particularly regarding recall and exact entity boundary extraction.

The study concludes that while LLMs are inching closer to supervised methods, they are not yet a complete replacement for specialized, fine-tuned models in high-stakes biomedical knowledge discovery. The authors caution that the "zero-shot" promise is heavily dependent on the specific task's complexity and the consequences of false positives/negatives. They also note the potential (though likely minimal) risk of data contamination in pre-training corpora but argue that the low scores on complex datasets like BioRED suggest that models are not simply memorizing ground truth labels.

Ultimately, the paper serves as a realistic benchmark, demonstrating that while ZSRE reduces annotation costs, it introduces new challenges in output consistency and entity boundary precision that must be addressed before widespread automated deployment in biomedical databases.