Small LLMs for Medical NLP: a Systematic Analysis of Few-Shot, Constraint Decoding, Fine-Tuning and Continual Pre-Training in Italian

1. Problem Statement

Large Language Models (LLMs) have demonstrated superior performance in medical Natural Language Processing (NLP) tasks. However, their massive computational requirements (often requiring hundreds of billions of parameters) make them impractical for deployment in real-world healthcare settings, where hospitals and clinics often face strict financial and hardware constraints.

The central research question is: Can "Small" LLMs (SLLMs), specifically those with around 1 billion parameters, effectively perform complex medical NLP tasks in Italian while maintaining competitive accuracy against much larger baselines?

2. Methodology

2.1. Models Evaluated

The authors evaluated three major model families, focusing on models with approximately 1 billion parameters:

• Llama-3.2 (1B and 1.24B variants)
• Gemma-3 (1B variants)
• Qwen3 (1.7B variant)

These were compared against strong baselines: Qwen3-32B and MedGemma-27B.

2.2. Datasets and Tasks

The study utilized a comprehensive, curated collection of Italian medical datasets, covering five main NLP tasks and 20 sub-tasks:

1. Named Entity Recognition (NER): Extracting clinical entities, body parts, drugs, diseases, and symptoms.
2. Relation Extraction (RE): Identifying semantic relationships (e.g., between exams and results).
3. Case Report Form (CRF) Filling: Populating structured fields from unstructured clinical narratives.
4. Medical Question Answering (QA): Multiple-choice questions based on clinical text.
5. Argument Mining: Identifying reasoning elements (claims, premises) in clinical text.

Data Resources Released:

• A new Continual Pre-Training (CPT) corpus of ~406 million words, comprising:
  • Scientific Data (278M words): Med CommonCrawl, drug instructions, medical Wikipedia, E3C corpus, University of Padova theses, and Italian scientific publications (Pensiero Scientifico, Zadig).
  • Clinical Data (126M words): 1.97 million anonymized Electronic Health Records (EHRs) from an Italian hospital's Emergency Department (including anamnesis, discharge letters, lab reports, etc.).

2.3. Adaptation Strategies

The authors systematically compared four adaptation strategies at both inference and training times:

1. Zero-Shot Prompting: Baseline performance using only pre-trained knowledge.
2. Few-Shot Prompting (4-shot): Providing examples to guide the model.
3. Constraint Decoding (CD): Enforcing structured output (e.g., JSON) using a grammar-constrained decoder (via the outlines library) to prevent formatting errors.
4. Supervised Fine-Tuning (FT): Training models on task-specific instruction-output pairs using LoRA (Low-Rank Adaptation).
5. Continual Pre-Training (CPT): Further pre-training on the massive medical corpus before fine-tuning.

3. Key Contributions

1. Systematic Evaluation: The first comprehensive analysis of SLLMs (1B parameters) on Italian medical NLP, comparing inference-time vs. training-time adaptation strategies.
2. Resource Release:
   • The first comprehensive collection of public Italian medical NLP datasets.
   • A new 406M-word medical corpus (Scientific + Clinical) for CPT.
   • Top-performing adapted models released publicly.
3. Codebase: An extensible open-source framework for medical NLP tasks.

4. Key Results

4.1. Performance Comparison

• Fine-Tuning is Superior: Supervised Fine-Tuning (FT) emerged as the most effective strategy, consistently outperforming all inference-time methods (Zero-shot, Few-shot, CD).
• SLLMs vs. Large Models: Small models, when properly adapted, can match or surpass much larger models.
  • The best configuration (Qwen3-1.7B + FT) achieved an average score of 63.9, outperforming the Qwen3-32B (4-shot) baseline by +9.2 points.
  • Gemma-3-1B-it + CPT + FT achieved a score of 59.4, surpassing the 32B baseline by +4.7 points.
• Inference-Time Methods:
  • Few-Shot (4-shot) provided a significant boost (+9.4 points on average over zero-shot) but increased inference time by ~53%.
  • Constraint Decoding alone offered a modest gain (+3.8 points) with negligible impact on inference time.
  • Combining Few-Shot and CD yielded the best inference-time results but was still inferior to Fine-Tuning.

4.2. Task-Specific Insights

• Easiest Task: Question Answering (QA), likely due to exposure to QA-style data during general pre-training.
• Hardest Task: Relation Extraction (RE), indicating SLLMs struggle with complex inter-entity reasoning.
• Most Improved: Argument Mining showed the highest relative gains from adaptation.

4.3. Continual Pre-Training (CPT) Impact

• CPT generally provided marginal or no benefits over Fine-Tuning alone for most models.
• Exception: For Gemma-3-1B-it, CPT provided a noticeable boost, suggesting that specific model architectures may benefit more from domain-specific pre-training before fine-tuning.

4.4. Out-of-Distribution (OOD) Generalization

• While Fine-Tuning improved performance on OOD datasets, the gains were smaller compared to in-distribution results.
• Large baselines (32B) still demonstrated more robust generalization on unseen data, suggesting that while SLLMs are viable, they may require dataset-specific tuning for maximum robustness in new domains.

5. Significance and Conclusion

• Feasibility for Resource-Constrained Settings: The study proves that 1B-parameter models are a viable, cost-effective alternative to massive models for Italian medical NLP, provided they are fine-tuned. This opens the door for deployment in hospitals with limited GPU resources.
• Adaptation Strategy Hierarchy: The paper establishes a clear hierarchy for adapting SLLMs:
  1. Supervised Fine-Tuning (Best performance).
  2. Few-Shot + Constraint Decoding (Best inference-time alternative if training is not possible).
  3. Continual Pre-Training (Generally unnecessary unless specific model architectures suggest otherwise).
• Data Scarcity Solution: By releasing a massive, high-quality Italian medical corpus and datasets, the authors address a critical bottleneck in non-English medical AI research.

In conclusion, the authors demonstrate that with the right adaptation strategies (specifically Fine-Tuning), small, efficient LLMs can outperform significantly larger models on specialized medical tasks, making advanced medical AI accessible to a broader range of healthcare institutions.