Technical Summary: Relation Extraction Capabilities of LLMs on Clinical Text

Problem Statement

Clinical Information Extraction (CIE), specifically Relation Extraction (RE), is critical for constructing medical knowledge graphs and supporting clinical decision-making. However, the development of RE systems for non-English languages is severely hindered by a scarcity of annotated datasets. While English benefits from resources like the 2010 i2b2/VA corpus, low-resource languages such as Turkish lack dedicated clinical RE datasets and robust pre-trained biomedical models. Furthermore, the absence of parallel bilingual datasets prevents rigorous cross-lingual evaluation and the development of globally robust clinical AI systems. Existing studies often rely on random in-context example selection or generic retrieval methods, which may not capture the specific semantic nuances required for clinical relation classification.

Methodology

This study presents the first comprehensive bilingual evaluation of Large Language Models (LLMs) for clinical Relation Extraction in English and Turkish. The research employs a systematic approach involving dataset creation, novel retrieval mechanisms, and diverse prompting strategies.

1. Dataset Construction

The authors constructed the first English–Turkish parallel clinical RE dataset derived from the 2010 i2b2/VA relation classification corpus.

• Source: The original corpus contains discharge summaries and progress reports annotated for concept extraction, assertion classification, and relation classification.
• Translation Process: A collaborative effort involving computational linguists and medical experts from Ege University and Dokuz Eylül University translated the dataset. The process included initial machine translation (DeepL), expert validation by senior medical doctors, and final post-editing by a retired medical doctor to ensure terminological precision.
• Subset: A curated subset of 1,500 training and 500 test samples was selected, maintaining proportional representation of relation types (e.g., Treatment-Improves, Test-Reveals, Problem-Indicates).

2. Proposed Method: Relation-Aware Retrieval (RAR)

To address the limitations of generic semantic similarity in example selection, the authors propose Relation-Aware Retrieval (RAR).

• Mechanism: RAR utilizes SimCSE (a contrastive learning framework) fine-tuned from scratch on the target RE task.
• Training Objective: Unlike standard sentence embedding, RAR incorporates entity and relation label information into the training examples. It constructs a weighted similarity score combining sentence-level, subject-entity, object-entity, and relation-type embeddings.
• Goal: To retrieve in-context examples that are not only linguistically similar but also semantically aligned regarding the specific relational structure (e.g., matching "Treatment improves Problem" with similar relation types).

3. Prompting Strategies

The study evaluates several prompting configurations across multiple LLMs (Gemini 1.5/2.0 Flash, DeepSeek-V3, GPT-4o mini):

• In-Context Learning (ICL) Selection: Compares Random sampling, KATE (KNN-Augmented), Fine-Tuned Relation Representation (FT-RR, based on PURE), and the proposed RAR.
• Chain-of-Thought (CoT) Integration:
  • Static CoT: Fixed reasoning templates (Zero-shot and Few-shot).
  • Dynamic CoT: Reasoning generated on-the-fly for retrieved examples using:
    • Gold Label-Induced Reasoning: Prompting the model to explain the rationale for a gold label.
    • Self-Questioning Prompting (SQP): Generating diagnostic questions to uncover relationship nuances.
    • Output Format-Based CoT: A structured output format explicitly stating the relation and a brief justification (adapted from PromptNER).

4. Baselines and Models

• Fine-Tuned Baselines: BERT and PURE (a lightweight architecture using special markers for entities) were fine-tuned on the bilingual dataset.
• LLMs: Evaluation focused on efficiency and reasoning capabilities, selecting models based on MMLU benchmarks, latency, and cost.

Key Results

1. Superiority of Prompting over Fine-Tuning

Prompting-based LLM approaches consistently outperformed traditional fine-tuned models (BERT and PURE), even when the fine-tuned models were trained on significantly larger subsets of the English data.

• Fine-Tuned Performance: PURE achieved a Micro-F1 of 0.740 in English and 0.632 in Turkish on the 1,500-sample subset.
• Prompting Performance: LLMs using RAR achieved Micro-F1 scores of 0.906 (English) and 0.870 (Turkish) with Gemini 1.5 Flash, surpassing the fine-tuned baselines despite using fewer training examples.

2. Impact of Retrieval Strategy

Relation-Aware Retrieval (RAR) consistently outperformed all other selection methods (Random, KATE, FT-RR) across all languages and models.

• In English, RAR with Gemini 1.5 Flash reached 0.906 Micro-F1.
• In Turkish, RAR with Gemini 1.5 Flash reached 0.870 Micro-F1.
• RAR demonstrated high efficiency, achieving peak performance with as few as 5-shot examples, whereas other methods showed diminishing returns or required more examples to approach similar performance.

3. Effectiveness of CoT Strategies

Among CoT methods, Output Format-Based CoT yielded the highest performance, particularly when combined with RAR.

• Best Overall Result: DeepSeek-V3 combined with RAR and Output Format CoT achieved a Micro-F1 of 0.918 in English, matching the state-of-the-art (SOTA) result reported for the i2b2-2010 dataset (which utilized UMLS ontology enrichment).
• Language Disparity: English evaluations generally outperformed Turkish counterparts across all models and techniques. However, the gap narrowed significantly with advanced retrieval and prompting strategies.
• Model Specificity: Gemini 1.5 Flash showed the most consistent performance across both languages. GPT-4o mini struggled significantly in Turkish. DeepSeek-V3 showed strong adaptability, matching Gemini 1.5 in English when using structured reasoning.

4. Error Analysis

Confusion matrix analysis revealed that models struggled most with classes having limited data (e.g., "Treatment Not Administered") and semantic overlaps (e.g., distinguishing between "Test Reveals" vs. "Test Conducted to Investigate"). The Turkish model exhibited higher confusion rates than the English model, attributed to the agglutinative nature of Turkish and less mature language-specific embeddings.

Significance and Claims

The paper claims to provide the first evaluation of Relation Extraction on Turkish clinical text, addressing a critical gap in low-resource medical NLP. Its primary contributions are:

1. Dataset Creation: The introduction of a high-quality, parallel English-Turkish clinical RE dataset derived from the 2010 i2b2/VA corpus, validated by medical experts.
2. Methodological Innovation: The proposal of Relation-Aware Retrieval (RAR), which integrates contrastive learning with entity and relation-specific semantics to improve in-context example selection. The authors demonstrate that RAR consistently outperforms existing retrieval methods.
3. Empirical Evidence: The study establishes that prompting-based LLMs can outperform fine-tuned small language models in clinical RE tasks, even in low-resource bilingual settings. It highlights that high-quality, relation-aware demonstration retrieval is more critical for performance than model size or complex reasoning alone.
4. State-of-the-Art Performance: By combining RAR with structured Output Format CoT, the study achieves a Micro-F1 of 0.918 in English, matching the SOTA for the i2b2-2010 task without relying on external ontologies like UMLS, suggesting a viable path for multilingual clinical AI.

The authors conclude that integrating task-specific retrieval and structured prompting offers a practical pathway to bridge resource gaps in clinical NLP, enabling more equitable and accessible clinical AI systems across different languages.