💬 NLP

Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning?

This paper demonstrates that prompt-based large language models can serve as a viable, low-resource alternative to supervised learning for biomedical relation extraction, achieving superior macro-F1 performance on rare relation types through a trade-off between the high recall of pairwise classification and the precision of joint generation.

Original authors: Jakob Mraz, Tomaž Curk, Blaž Zupan

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jakob Mraz, Tomaž Curk, Blaž Zupan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of medical research as a massive, ever-growing library. Every year, millions of new books (scientific articles) are added to the shelves. The problem? These books are written in a complex, unstructured language that is hard for computers to read and organize. To make this knowledge useful, scientists need to turn these messy stories into neat, structured databases (like a map of how diseases, drugs, and genes connect).

This paper is about teaching computers to do that organizing job, specifically by finding the "relationships" between things mentioned in the text (e.g., "Drug A treats Disease B").

Here is the breakdown of their experiment, explained simply:

The Old Way vs. The New Way

The Old Way (Supervised Learning):
Imagine you want to teach a student to identify relationships. Traditionally, you would hire a team of expert librarians to read thousands of books, highlight the connections, and write them down on flashcards. You then train a computer to memorize these flashcards.

The Problem: Hiring those expert librarians is incredibly expensive and slow. Also, if the student only studied "heart disease" flashcards, they might get confused when they see a book about "rare tropical viruses."

The New Way (Large Language Models with "Few-Shot" Prompting):
Instead of hiring librarians to make flashcards, the researchers asked a super-smart, well-read AI (a Large Language Model or LLM) to do the job. They didn't train the AI from scratch. Instead, they gave it a "cheat sheet" (a prompt) with a few examples of what to look for and said, "Here is a new book; find the connections like you saw in the examples."

The Benefit: This is much faster and cheaper because the AI already knows a lot of language and facts from its training. It just needs a little nudge to focus on the medical task.

The Two Strategies: "One-on-One" vs. "Group Chat"

The researchers tested two different ways to ask the AI to do the work:

Pairwise Classification (The "One-on-One" Interview):
Imagine you have a list of 100 people in a room. To find out who knows whom, you pull out two people at a time, show them to the AI, and ask, "Do these two know each other?" You repeat this for every single pair.
- Pros: The AI is very thorough and rarely misses a connection (High Recall).
- Cons: It takes a long time because you have to ask the question 100 times. It's like interviewing every pair individually.
Joint Generation (The "Group Chat"):
Instead of pulling people out two at a time, you put the whole group in a room and say, "Look at everyone here and tell me all the connections you see in one go."
- Pros: It's incredibly fast and cheap (High Efficiency). The AI gives you a list of all connections in one answer.
- Cons: The AI might get overwhelmed and miss some subtle connections, or it might be too cautious and only list the ones it is 100% sure about (High Precision, but lower Recall).

The Results: A Trade-Off

The researchers found that neither method was perfect, but they had different strengths:

The "One-on-One" method found more connections, but it was slow and expensive to run.
The "Group Chat" method was much faster and cheaper, and the connections it did find were very accurate.
The "Sweet Spot" (k-constrained generation): They also tried a middle ground where they put small groups of people in the room (e.g., groups of 3) instead of the whole crowd or just pairs. This balanced speed and accuracy.

The Big Surprise:
When they looked at the overall score, the new AI methods were slightly behind the old "expert flashcard" method. However, the paper found a twist:

The old method was really good at the common, easy relationships but struggled with rare or weird ones.
The new AI methods were actually better at finding those rare, difficult connections.
The reason the new methods seemed "worse" overall was because of one specific, confusing category called "Association." It's like a "miscellaneous" bin where the human experts put things they weren't sure about. The AI got confused by this vague label, but when the researchers looked at the specific, clear relationships, the AI performed very well.

The Bottom Line

The paper concludes that using these smart, pre-trained AI models is a viable alternative to the old, expensive way of training computers, especially when you don't have a lot of labeled data or when you need to find rare connections.

If you need to find every possible connection and don't mind the cost, use the "One-on-One" method.
If you need speed, low cost, and high accuracy on the connections you do find, use the "Group Chat" method.

The study suggests that with better definitions of what we are looking for, these AI tools could replace the need for massive, expensive human annotation projects in the future.

Technical Summary: Few-Shot Biomedical Relation Extraction with Large Language Models

Problem Statement
Biomedical relation extraction (BioRE) is critical for converting unstructured literature into structured knowledge graphs. However, existing approaches predominantly rely on supervised learning, which demands costly, expert-annotated datasets and often struggles to generalize to rare relation types or new domains. While Large Language Models (LLMs) offer a potential solution for low-resource settings via prompt-based learning, their performance in BioRE has been inconsistent, with reported results varying from substantial underperformance to competitiveness with supervised baselines. Furthermore, a systematic comparison of different task formulations (pairwise classification vs. joint generation) and their trade-offs in extraction quality versus computational efficiency has been lacking.

Methodology
The authors investigate few-shot BioRE using prompt-based learning with recent open-weight LLMs (Gemma-4 and Qwen-3.5 families, specifically 25–35B parameter variants). The study is conducted on the BioREDirect dataset, which contains PubMed abstracts annotated with six entity types and eight relation types. The evaluation focuses on document-level extraction, a more challenging setting than sentence-level extraction due to long-range dependencies.

The study compares two primary task formulations:

Pairwise Classification: The model processes the text with a single annotated entity pair and predicts the relation type (or "None"). This requires one model call per candidate pair.
Joint Generation: The model processes the text with multiple annotated entities and generates all supported (head, tail, relation) triples in a single call.
- To analyze the precision-recall trade-off, the authors introduce a $k$ -constrained generation setting. Here, entities are divided into overlapping subsets of size $k$ (where $k < N$ , the total number of entities). Predictions from overlapping subsets are aggregated via majority vote.

The experiments utilize deterministic decoding and evaluate performance with and without reasoning capabilities (intermediate reasoning steps). Metrics include Precision, Recall, Micro-F1, and Macro-F1 (to account for class imbalance), alongside a weighted token cost metric to measure computational efficiency.

Key Results

Performance vs. Supervised Baselines: The best-performing few-shot model (Qwen3.5-27B in pairwise classification) achieved a Micro-F1 score of 0.44, substantially outperforming previous few-shot results (0.34) but remaining below the supervised PubMedBERT baseline (0.56).
The "Association" Class Effect: The performance gap with supervised models is largely attributed to the "Association" relation type, which is ambiguously defined and serves as a fallback label in the dataset. When evaluated using Macro-F1 (which better captures performance across imbalanced classes), the prompt-based approaches outperformed the supervised baseline (0.45 vs. 0.38), particularly on rare relation types like Bind, Comparison, and Cotreatment.
Task Formulation Trade-offs:
- Pairwise Classification consistently achieved higher recall, making it suitable for applications requiring maximal relation coverage.
- Joint Generation yielded higher precision but lower recall, likely due to the autoregressive nature of generation omitting uncertain relations.
- $k$ -constrained generation offered a tunable middle ground, allowing practitioners to balance precision, recall, and cost.
Computational Efficiency: Joint generation was significantly more efficient, reducing computational costs by up to 25× compared to pairwise classification. While enabling reasoning improved performance (especially for Mixture-of-Experts models), it incurred substantial computational costs due to increased output token generation. Mixture-of-Experts (MoE) architectures proved more cost-efficient than dense models.

Significance and Claims
The paper claims that few-shot LLMs represent a viable alternative to supervised learning for BioRE, particularly in low-resource settings or when dealing with well-defined, specific relation types. The authors argue that the perceived underperformance of LLMs relative to supervised models is often an artifact of dataset imbalance and annotation ambiguity (specifically the "Association" class) rather than a fundamental limitation of the models.

The study highlights that:

Macro-F1 is a critical metric for evaluating BioRE in imbalanced settings, where LLMs demonstrate superior generalization to rare relations compared to supervised baselines.
Task formulation matters: The choice between pairwise classification and joint generation should be driven by specific application needs (recall vs. precision/cost) rather than a single "best" metric.
Well-defined schemas are essential: The performance of instruction-driven LLMs is highly dependent on clear relation definitions; ambiguity in annotation schemes disproportionately affects LLMs compared to models trained directly on the noisy data.

The authors conclude that while LLMs do not yet fully surpass supervised baselines in raw Micro-F1, their flexibility, ability to generalize to rare classes, and computational efficiency (via joint generation) make them a compelling option for biomedical knowledge extraction.

The Old Way vs. The New Way

The Two Strategies: "One-on-One" vs. "Group Chat"

The Results: A Trade-Off

The Bottom Line

Technical Summary: Few-Shot Biomedical Relation Extraction with Large Language Models

More like this