Technical Summary: Sub-Billion, Super-Frontier Small Language Models for Relation Extraction

Problem Statement

Large Language Models (LLMs) have achieved state-of-the-art performance in Relation Extraction (RE), yet their deployment is hindered by high computational costs, reliance on proprietary APIs, and privacy concerns. While Small Language Models (SLMs) offer a path toward efficient, on-device, and private RE, they traditionally lag behind frontier LLMs in complex reasoning tasks, particularly in domains with implicit semantics and complex narrative structures like literary texts. The central question addressed is whether compact SLMs (ranging from 360M to 3B parameters), when subjected to targeted task adaptation, can close the performance gap with zero-shot frontier LLMs across both general-domain and literary text.

Methodology

The study employs a rigorous, controlled experimental design to evaluate five open-weight SLMs against frontier proprietary models (GPT-5.4 and Claude Sonnet 4.6) and a discriminative RoBERTa baseline.

Models and Scales

• Sub-billion Models: SmolLM2-360M-Instruct and Qwen2.5-0.5B-Instruct.
• Small Models (3B): SmolLM3-3B, Qwen2.5-3B-Instruct, and Llama-3.2-3B-Instruct.
• Baselines: Zero-shot frontier LLMs (via OpenRouter API) and a fully supervised entity-marker RoBERTa classifier.

Experimental Regimes

The authors evaluated 30 distinct configurations derived from five base models, three domain-composition tuning regimes, and two prompt-conditioned tuning styles:

1. Tuning Regimes:
   • GenTune: Fine-tuned exclusively on general-domain datasets (news, Wikipedia, web).
   • LitTune: Fine-tuned exclusively on literary datasets (biographical, fiction).
   • MixTune: Fine-tuned on a balanced mixture of general and literary data.
2. Prompt-Conditioned Styles:
   • 0-shot: No in-context demonstrations during tuning or inference.
   • 2-shot: Two in-context demonstrations included during both tuning and inference.
3. Domain-Adaptive Pretraining (DAPT): A targeted case study on Llama-3.2-3B involving continued pretraining on unannotated literary text (LitBank) before supervised fine-tuning, to isolate the effect of unsupervised domain exposure.

Datasets and Evaluation

The evaluation spans nine benchmarks: seven general-domain (TACRED, SemEval-2010 Task 8, CoNLL04, NYT11, GIDS, Re-DocRED, REBEL) and two literary (Biographical, PG-Fiction).

• Metric: Primary metric is positive-class micro-F1, excluding the "no-relation" class to avoid majority-class bias.
• Protocol: All SLMs were fine-tuned using QLoRA (4-bit NormalFloat quantization with LoRA adapters) on consumer hardware (NVIDIA RTX 4090). Inference used greedy decoding. Frontier models were evaluated zero-shot.
• Prompting: Two conditions were tested: schema-enumerated (listing allowed labels in the system prompt) and generic (no label list). The paper primarily reports schema-enumerated results as a conservative lower bound.

Key Contributions

1. Systematic SLM Evaluation: The most comprehensive controlled study to date on domain-composition and prompt-conditioned tuning for SLMs in RE, covering 30 configurations across five models.
2. Multi-Domain and Cross-Scale Analysis: A controlled comparison of specialist (domain-specific) versus generalist (mixed-domain) models, demonstrating that a single mixed-domain model can cover both general and literary domains with minimal performance trade-off.
3. Training Strategy Analysis: Quantification of how mixed-domain tuning and prompt-conditioned supervision narrow the gap between SLMs and frontier LLMs. Crucially, the study isolates the effect of supervised task adaptation versus unsupervised domain exposure (DAPT).
4. Efficiency-Oriented Assessment: Demonstration that strong RE performance is achievable with models deployable on a single consumer GPU or CPU, offering a practical alternative to API-dependent frontier systems.

Key Results

General-Domain Performance

• SLMs Outperform Frontier LLMs: The best sub-billion model, Qwen2.5-0.5B (fine-tuned on general data with 2-shot prompting), achieved a General Average F1 of 0.828. This significantly outperformed zero-shot GPT-5.4 (0.693) and Claude Sonnet 4.6 (0.662).
• Role of Fine-Tuning: The advantage stems from task-specific adaptation rather than generative decoding or raw scale. A discriminative RoBERTa baseline also exceeded frontier models, confirming that the gains are due to in-domain training.
• Scale vs. Strategy: Within the same model family (Qwen2.5), scaling from 0.5B to 3B yielded negligible gains (+0.004) on general-domain RE, suggesting that for well-tuned tasks, sub-billion models are sufficient.

Literary Domain Performance

• Significant Gaps: Tuned SLMs substantially outperformed frontier models on literary RE. The best 3B model (SmolLM3-3B with LitTune 0-shot) achieved a Literature Average F1 of 0.833, leading GPT-5.4 (0.578) by ~26 points and Claude (0.530) by ~30 points.
• Supervised Adaptation is Key: The DAPT case study revealed that continued unsupervised pretraining on literary text added no practically meaningful gain (≤0.001 F1) over supervised fine-tuning alone. The performance gap is closed by supervised task adaptation, not by additional unsupervised exposure.
• Prompting Effects: For sub-billion models, 2-shot tuning provided massive gains (14–21 F1 points), compensating for limited capacity. For 3B models, the benefit of 2-shot tuning was marginal or negative, suggesting larger models internalize the schema better from supervised data alone.

Prompting and Error Analysis

• Generic vs. Schema-Enumerated: Contrary to expectations, generic prompting (without listing labels) outperformed schema-enumerated prompting by an average of +3.2 F1 points across datasets, with no loss in output well-formedness. This suggests supervised training is sufficient for schema compliance.
• Error Modes: Common failures included near-neighbor label confusion (e.g., confusing specific relations with hypernyms), default-to-negative under-prediction (missing implicit relations), and hallucinated out-of-schema labels. These errors were more prevalent in mixed-domain training and smaller models.

Significance and Claims

The paper claims that targeted training strategies and data-centered optimization can make compact SLMs competitive with, and often superior to, much larger proprietary systems for Relation Extraction.

• Democratization: The results suggest that high-quality RE does not require massive, API-gated models. Compact, task-adapted models can be deployed locally on consumer hardware, ensuring privacy, reducing latency, and lowering energy costs.
• Efficiency vs. Performance: The study demonstrates a favorable performance-efficiency trade-off, where sub-billion models achieve high F1 scores with a fraction of the parameters and storage footprint of frontier LLMs.
• Literary NLP: The work challenges the notion that literary RE is primarily a "scale problem," showing instead that domain-specific supervised adaptation is the critical factor for success in complex narrative domains.

The authors maintain modesty regarding their claims, noting that scale effects are confounded with model families in their setup, that all runs used a single random seed (limiting claims on sub-point differences), and that the evaluation is English-only. They emphasize that their findings highlight the potential of task-specific adaptation over raw scaling or unsupervised domain pretraining for specific extraction tasks.