A Semi-Automated Annotation Workflow for Paediatric Histopathology Reports Using Small Language Models

1. Problem Statement

Electronic Patient Records (EPRs) contain vast amounts of valuable clinical data, but much of it remains trapped in unstructured free-text formats, particularly in histopathology reports. This limits the utility of this data for quantitative research, secondary analysis, and automated decision-making.

• Specific Challenges: Histopathology reports contain domain-specific terminology, abbreviations, informal language, negations (e.g., "no evidence of"), and uncertainties. Traditional Named Entity Recognition (NER) models (e.g., spaCy, BioBERT) struggle with variable phrasing, discontinuous text spans, and complex implicit relationships.
• Constraints: While Large Language Models (LLMs) offer flexibility, they typically require heavy computational resources (GPUs) and external API access, which is incompatible with the strict privacy and security requirements of sensitive paediatric data that must remain on local, CPU-only infrastructure (e.g., standard NHS laptops with 16GB RAM).
• Goal: Develop a resource-efficient, semi-automated workflow using Small Language Models (SLMs) (1B–5B parameters) to extract structured information from paediatric renal biopsy reports without fine-tuning, while minimizing clinician annotation time.

2. Methodology

The authors developed an iterative, semi-automated workflow centered on a Question-Answering (QA) framework rather than traditional token-based NER.

A. Dataset and Preprocessing

• Data Source: 2,111 paediatric renal biopsy reports from Great Ormond Street Hospital (GOSH), collected between 2018–2023.
• Preprocessing: Reports were parsed to extract specific sections (microscopy and conclusion) using rule-based methods.
• Ground Truth: A subset of 400 reports was manually annotated. 100 reports were used to define the entity schema and guidelines; the remaining 300 were used for evaluation.

B. Entity Schema and Guidelines

• Schema: Based on the Banff classification for renal transplants, adapted for both native and transplant biopsies.
• Entities: Nine key entities were defined, including binary (e.g., cortex present), numerical (e.g., n total glomeruli), and complex string entities (e.g., final diagnosis).
• Guidelines: Clinician-guided rules were created to handle edge cases, negations, and implicit information (e.g., if no abnormalities are mentioned, assume zero sclerosed glomeruli).

C. Model Selection and Configuration

Five instruction-tuned SLMs were evaluated, running locally on CPU via Ollama or llama-cpp-python:

1. Qwen-2.5 (1.5B, FP16)
2. Gemma 2 (2B, FP16)
3. Llama 3.2 (3B, Q8 quantized)
4. Phi-3.5 Mini (3.8B, Q4 and Q8 quantized)

Experimental Settings:
The study tested four prompt configurations:

1. Zero-shot (no guidelines).
2. Two-shot (synthetic examples, no guidelines).
3. Zero-shot with guidelines.
4. Two-shot with guidelines.

D. Evaluation Framework

• Metric: Accuracy calculated as the proportion of correctly extracted entities.
• LM-as-a-Judge (LAAJ): For complex string entities where exact string matching fails, a smaller 1B parameter LLaMA model was used to judge semantic equivalence between the model output and ground truth.
• Disagreement Modelling: A framework comparing two SLMs against ground truth to prioritize reports for clinical review. It categorizes agreements (e.g., "All Agree Correct") and disagreements to flag high-priority errors.

3. Key Contributions

1. SLM Viability in Constrained Environments: Demonstrated that SLMs (≤5B parameters) can achieve clinically useful accuracy (>80%) on specialized medical tasks using only CPU infrastructure, making them accessible for NHS trusts without GPU clusters.
2. Prompt Engineering Strategy: Showed that clinician-guided entity guidelines and few-shot examples significantly boost performance. Guidelines improved accuracy by 7.3–18.6%, while few-shot examples improved it by 5.7–37.9% (depending on the model).
3. Semi-Automated Workflow: Proposed a practical workflow requiring only three clinician meetings over three months. The system uses disagreement modelling to filter out consensus cases, directing clinician attention only to high-risk or ambiguous reports.
4. Open Source Implementation: Released a modular Python codebase (Streamlit-based) for preprocessing, QA prompting, and annotation, designed for non-computational researchers.

4. Results

• Top Performance: Gemma 2 (2B) achieved the highest accuracy at 84.3% using a two-shot prompt with guidelines.
• Comparison with Baselines: The SLM workflow outperformed traditional methods:
  • spaCy: 74.3%
  • BioBERT-SQuAD: 62.3%
  • RoBERTa-SQuAD: 59.7%
  • GLiNER: 60.2%
• Entity-Level Performance:
  • SLMs excelled at complex string entities (e.g., final diagnosis), achieving 88.5–91.8% accuracy compared to <60% for traditional methods.
  • Simple binary entities (e.g., cortex present) were handled well by all models (≥91% accuracy).
  • Challenges: Entities like chronic change and abnormal glomeruli remained difficult (50–79% accuracy) due to high variability and the need for cross-section reasoning.
• Efficiency: Processing times ranged from 26.2 to 72.0 seconds per report. The full dataset of 2,111 reports could be processed in under two days on a standard laptop.
• Failure Modes: Quantized Phi models (Phi-3.5) exhibited catastrophic sequential degradation (incoherent text generation) after processing ~300 reports, likely due to memory/KV cache issues on limited hardware. Gemma 2 and Qwen-2.5 showed minimal JSON parsing errors.
• Disagreement Modelling: The framework identified that 64% of predictions achieved consensus. It successfully prioritized reports with high-priority entity disagreements (e.g., final diagnosis) for clinical review.

5. Significance and Conclusion

• Clinical Impact: This workflow bridges the gap between unstructured medical text and structured research data without requiring expensive infrastructure or massive datasets. It reduces clinician workload by automating the bulk of annotation while retaining human oversight for critical errors.
• Scalability: The approach is highly transferable to other NHS trusts and potentially other medical domains (e.g., adult pathology, genomics) where standardized guidelines exist.
• Strategic Insight: The study suggests that guidelines and few-shot examples function as alternative strategies rather than complementary ones; combining them did not yield significant additional gains over using either alone.
• Future Directions: The authors recommend hybrid approaches (combining fast rule-based extraction for simple entities with SLMs for complex strings) and further refinement of entity schemas to handle granular pathological details.

In summary, the paper proves that Small Language Models, when paired with expert-guided prompting and disagreement-based prioritization, offer a practical, secure, and accurate solution for extracting structured data from complex paediatric histopathology reports.