Real-time structuring of the clinical information section for dopamine transporter SPECT reporting with a small language model pipeline: development and external validation

Technical Summary: Real-time Structuring of DAT-SPECT Clinical Information via a Small Language Model Pipeline

Problem Statement
Accurate interpretation of Dopamine Transporter (DAT)-SPECT imaging relies heavily on contextual clinical information, including patient symptoms, medication history, comorbidities, and structural imaging findings. In routine clinical practice, this data is typically documented as heterogeneous free text. This unstructured format introduces ambiguity, inconsistent terminology, and variable detail levels, which can hinder the consistent consideration of clinically relevant parameters and limit adherence to reporting guidelines. While Natural Language Processing (NLP) offers a solution, existing approaches predominantly rely on Large Language Models (LLMs) with hundreds of billions of parameters. These models require substantial computational resources and cloud-based infrastructure, raising concerns regarding data protection, regulatory compliance, and technical feasibility in clinical settings.

Methodology
The study developed and evaluated a locally deployable pipeline utilizing a Small Language Model (SLM) to transform free-text clinical documentation into standardized, structured reports for DAT-SPECT.

• Datasets: Two retrospective datasets were utilized. The in-house dataset (University Medical Center Hamburg-Eppendorf) contained 2,990 clinical information sections (split into 2,700 for training and 290 for internal testing). An external validation dataset (University Hospital of Freiburg) comprised 595 clinical sections. All texts were in German.
• Structured Dictionary: A reference dictionary comprising 466 entries across 12 categories (e.g., indications, suspected etiologies, symptoms, medication, lateralization) was developed by two experienced DAT-SPECT readers. This dictionary included metadata such as synonyms, abbreviations, and attributes for post-processing (e.g., drug interactions with the tracer).
• Annotation: The training and internal test sets were manually annotated by an expert reader, mapping free-text content to the structured dictionary entries.
• Model Training: A Qwen3 1.7B-parameter model was fine-tuned using supervised learning on the annotated data. Training utilized the 8-bit AdamW optimizer with a learning rate of 1 × 10⁻⁵ over four epochs. During inference, the temperature was set to 0 to ensure deterministic output.
• Rule-Based Postprocessing: A critical component of the pipeline was a rule-based postprocessing tool. Rather than relying solely on the SLM to infer complex clinical logic, this module:
  • Removed redundant entries.
  • Automatically suggested the scan indication based on symptom metadata.
  • Consolidated symptom lateralization and time course.
  • Generated reminders for missing guideline-relevant elements.
  • Cross-referenced medications to distinguish those with relevant interactions with [¹²³I]ioflupane binding from those without.

Key Results

• Internal Performance: In the internal test set (n=290), the postprocessed SLM output provided the same information as the expert manual annotation in 67.6% of cases (196/290). Exact matches occurred in 50.7% (147/290). Interpretation differences due to text ambiguity accounted for 4.5%, and objective SLM errors occurred in 14.1%. Notably, in 13.8% of cases, the SLM output was judged superior to the manual annotation in clarity or completeness. Among the 41 error cases, 87.8% involved only a single erroneous or missing item.
• External Validation: In the external dataset (n=595), 75.0% (446/595) of the postprocessed outputs required no manual correction before inclusion in a report. Of the remaining 25.0% requiring correction, 74.5% needed only a single item corrected. The mean number of corrections per case was 0.34 items, representing 6.0% of all structured items.
• Comparison to Base Model: The fine-tuned SLM significantly outperformed the non-fine-tuned base model (Qwen3-1.7B-Instruct) using three-shot prompting, which achieved exact agreement in only 2.8% of cases compared to 50.7% for the fine-tuned model.

Significance and Claims
The paper claims that a locally deployable SLM combined with rule-based postprocessing can effectively transform unstructured clinical information into standardized, report-ready documentation with high clinical usability and robust external performance.

Key contributions highlighted by the authors include:

1. Local Deployment: The pipeline operates on standard hardware (NVIDIA RTX 6000 Ada GPUs) without external data transfer, addressing data protection and regulatory concerns associated with cloud-based LLMs.
2. Hybrid Architecture: The integration of an SLM with a structured dictionary and rule-based postprocessing combines the flexibility of language models with the reliability and transparency of rule-based systems. This allows for easy adaptation (e.g., adding new symptoms or drugs) without retraining the model.
3. Clinical Utility: The system is designed for real-time use during patient interviews as a "Digital Medical Interview Assistant." It aims to improve consistency, highlight missing guideline-relevant information, and facilitate structured reporting.
4. Generalizability: Despite differences in documentation styles between the internal and external institutions, the pipeline maintained robust performance, suggesting good generalizability across institutions.

The authors acknowledge limitations, including the use of a single reader for annotation (potential subjective bias), the reliance on German-language data, and the fact that the study assessed technical feasibility rather than the direct impact on clinical workflow or diagnostic accuracy. They conclude that while the pipeline demonstrates high accuracy and utility, future prospective studies are needed to evaluate its impact on reporting efficiency and diagnostic outcomes.