Structured Semantic Information Helps Retrieve Better Examples for In-Context Learning Applied to Few-Shot Relation Extraction

Technical Summary: Structured Semantic Information Helps Retrieve Better Examples for In-Context Learning Applied to Few-Shot Relation Extraction

Problem Definition
Relation Extraction (RE) is a fundamental NLP task involving the identification of relationships between entity pairs (e.g., subject-object pairs). While traditional supervised RE requires hundreds or thousands of training examples per relation, Few-Shot Relation Extraction (FSRE) aims to perform this task given only a relation name, a brief description, and a handful of examples (typically 1-shot). Existing FSRE approaches often rely on prototype-based classification or embedding-based nearest-neighbor methods, which can lack flexibility under domain or relation shifts. Although In-Context Learning (ICL) with Large Language Models (LLMs) offers a promising alternative, the optimal strategy for selecting additional examples to complement the single "gold" example remains underexplored. Specifically, it is unclear how to balance the need for examples that are semantically similar to the gold example with the need for diverse sentence structures and lexical choices to prevent overfitting to a single pattern.

Methodology
The paper proposes a framework to automatically transform a 1-shot RE setup into a 5-shot or 10-shot setting by generating or retrieving additional examples. The approach evaluates three primary strategies and a hybrid combination:

1. LLM-Based Generation:

   • Paraphrasing: The gold support sentence is paraphrased by an LLM while preserving the subject, object, and relation.
   • Generation: New examples are generated conditioned on the relation name and its natural language definition, encouraging alternative lexicalizations and sentence structures.

2. Retrieval from Unannotated Corpora:

   • The authors retrieve examples from a large subset of the UMBC WebBase corpus (2.3M sentences).
   • SBERT Baseline: Standard sentence embeddings are used to find semantically similar sentences.
   • Syntactic-Semantic Rule Representations (Novelty): Instead of full sentence embeddings, the method extracts lexico-syntactic rules representing the shortest dependency path between the subject and object entities. These rules encode lexical items, dependency relations, directionality, and named entity types.
   • Semantic Matching: A self-supervised model (SoftMatcher) learns semantic representations of these rules. Retrieval is performed based on the cosine similarity between the rule representation of the gold example and candidate sentences. This approach focuses on the specific context indicative of the relation rather than general sentence meaning.

3. Diversity Enhancement:

   • To prevent redundancy, candidate examples are clustered (using k-means or k-means++) based on their semantic representations.
   • Representative examples are selected from clusters to ensure a mix of similarity to the gold example and structural diversity.

4. Hybrid Strategy:

   • The framework combines examples from both LLM generation and corpus retrieval. A prompt is used to select a final set of additional examples (4 or 9) from the combined pool, aiming to balance the high similarity of LLM-generated examples with the realistic diversity of retrieved examples.

5. Task Formulation:

   • The 5-way 1-shot task is reformulated as five independent binary classification tasks (one per relation) using ICL.
   • A Named Entity Recognition (NER) filter is applied prior to ICL to discard candidate relations that are incompatible with the entity types in the query sentence, significantly reducing computational cost.

Key Contributions

• Novel Retrieval Strategy: The introduction of using structured syntactic-semantic rule representations (rather than surface-level sentence embeddings) to retrieve examples for ICL in FSRE. This method captures the core relational structure between entities.
• Diversity Mechanisms: The proposal of explicit clustering and prompt engineering to ensure that selected examples offer diverse word choices and sentence structures while maintaining relation faithfulness.
• Hybrid Framework: The demonstration that combining LLM-generated examples (for similarity) and retrieved examples (for diversity) yields a more holistic representation of the relation than either method alone.
• State-of-the-Art Performance: The establishment of new state-of-the-art results on the FS-TACRED dataset, particularly for smaller language models, without requiring model fine-tuning or additional gold annotations.

Results
The framework was evaluated on two datasets (FS-TACRED and a customized FS-FewRel subset) using four LLMs (Qwen3-4B, Qwen3-14B, Gemma3-4B, and Gemma3-12B).

• Performance: The hybrid approach consistently outperformed baselines, including 1-shot ICL, LLM-only generation, and SBERT-based retrieval. On FS-TACRED, the hybrid method achieved an F1 score of 27.6 with Qwen3-4B (surpassing prior fine-tuned methods like SoftRules which achieved 24.8) and 37.8 with Qwen3-14B.
• Effectiveness of Semantic Rules: Examples retrieved using syntactic-semantic rules consistently yielded better results than those retrieved via SBERT or paraphrased by LLMs. The rule-based examples were found to be more relation-faithful and diverse.
• Model Sensitivity: The benefits of the retrieval strategy were most pronounced in smaller models (4B parameters). Larger models (12B) showed inconsistent gains, with some strategies even degrading performance compared to the 1-shot baseline, suggesting that larger models may already possess sufficient internal knowledge or are sensitive to prompt augmentation.
• Efficiency: The hybrid approach added negligible computational overhead (approx. 0.5% increase in wall-clock time per episode) compared to LLM generation alone, as the retrieval and clustering steps rely on precomputed embeddings and are independent of the LLM size.

Significance and Claims
The paper claims that structured semantic information is critical for effective example selection in In-Context Learning for Few-Shot Relation Extraction. By moving beyond surface-level similarity (SBERT) or simple generation, the proposed method retrieves examples that are both realistic (from unannotated corpora) and structurally faithful to the target relation.

The authors emphasize that their approach:

1. Transfers well across different datasets and LLM families without task-specific fine-tuning.
2. Achieves SOTA on FS-TACRED, demonstrating that adding genuine, retrieved examples can outperform dedicated few-shot models that rely on fine-tuning smaller architectures.
3. Balances similarity and diversity, showing that the optimal ICL performance comes from a "middle ground" where examples are similar enough to define the relation but diverse enough to prevent overfitting to a single syntactic pattern.

The work highlights that while LLMs are powerful, their in-context learning capabilities for RE can be significantly enhanced by grounding example selection in the underlying syntactic-semantic structure of the language, particularly when computational resources or training data are limited.