Automatic Extraction of Structured Information from Brain MRI Reports Using an Open-Weight Large Language Model

Technical Summary: Automatic Extraction of Structured Information from Brain MRI Reports Using an Open-Weight Large Language Model

Problem Statement
Radiology reports represent a rich resource for large-scale medical research, yet their unstructured, free-text format significantly limits data usability. While structured reporting (SR) is advocated by organizations like the European Society of Radiology (ESR), implementation remains hindered by technical barriers, workflow incompatibilities, and resistance from radiologists. Consequently, extracting structured data currently relies on manual annotation, which is time-consuming and error-prone. Although Large Language Models (LLMs) have shown promise in automating this conversion, most existing studies rely on proprietary models (e.g., GPT-4) that raise privacy concerns due to external data transfer. Furthermore, open-weight models have been underexplored in specialized medical domains and non-English languages, particularly regarding their ability to handle complex, variable-rich neuroradiology reports without extensive fine-tuning.

Methodology
The study evaluated the open-weight LLM LLaMA 3.1 (specifically the 8B and 70B parameter versions) for extracting 30 variables from 947 Dutch brain MRI reports generated at a tertiary memory clinic between 2016 and 2021. The dataset included both unstructured free-text reports and those following a structured template.

• Data Annotation: Thirty variables (24 clinical, 6 quality control) were manually annotated by trained medical students. A subset of 100 reports was double-annotated to establish a consensus reference standard and assess inter-rater reliability (IRR).
• Experimental Design:
  • Zero-Shot Performance: The model's ability to extract data without examples was compared against human annotators.
  • Input Language: The study assessed whether translating Dutch reports to English (using LLaMA 3.1 or OPUS-MT) improved performance, with extracted text back-translated for evaluation.
  • Few-Shot Prompting: To serve as a resource-efficient alternative to fine-tuning, the study tested four example selection strategies for few-shot prompting: Random, Fixed (curated), Structural Similarity (Jaccard distance), and Semantic Similarity (cosine similarity of embeddings).
• Evaluation Metrics: Performance was measured using balanced accuracy for categorical variables, accuracy and Mean Absolute Error (MAE) for counts, and semantic text similarity (via Sentence-BERT) for free-text fields. Statistical significance was assessed using McNemar's test, Friedman tests, and Nemenyi post-hoc tests across 10 random Monte Carlo cross-validation splits.

Key Results

• Human-Level Performance: LLaMA 3.1 demonstrated high zero-shot performance on standardized visual rating scales, achieving balanced accuracies comparable to human raters for Medial Temporal Atrophy (MTA), Global Cortical Atrophy (GCA), and Fazekas scores.
• Variable Challenges: Performance was lower for numerical variables (e.g., infarct counts) and specific lesion subtypes (e.g., splinter infarcts). The model frequently confused "missing" information with explicit negative findings.
• Language Impact: Translating reports to English did not improve performance; in fact, original Dutch reports yielded more reliable results. Translation introduced errors with local terminology (e.g., "splinterinfarcten" was mistranslated as "small infarct," losing morphological specificity).
• Few-Shot Prompting: Few-shot prompting significantly outperformed zero-shot settings, particularly for numerical variables. Structural similarity-based selection emerged as the most effective strategy, improving infarct count accuracy from 66% (zero-shot) to 81% and microbleed count accuracy from 80% to 92%.
• Efficiency: Inference time remained under 30 seconds per report. Few-shot prompting increased inference time by only 7% while substantially reducing the rate of invalid JSON outputs.

Key Contributions

1. Validation of Open-Weight Models: The study provides a systematic assessment of LLaMA 3.1 for structured radiology reporting in Dutch, demonstrating that open-weight models can achieve competitive performance without requiring data transfer to external servers.
2. Few-Shot Optimization: It identifies structural similarity-based example selection as a highly effective, low-resource method to enhance extraction accuracy for numerical variables, offering a viable alternative to fine-tuning in settings with limited annotated data.
3. Language Specificity: The work highlights the critical importance of retaining original language inputs for medical reports, showing that translation can degrade performance by obscuring local medical terminology and morphological distinctions.
4. Comprehensive Benchmarking: The paper establishes a benchmark for 30 diverse variables (categorical, numerical, and free-text) in Dutch neuroradiology, including detailed error analysis and inter-rater reliability comparisons.

Significance and Claims
The authors claim that LLaMA 3.1 shows strong potential for the automated, post-hoc structuring of free-text radiology reports, facilitating large-scale data reuse for dementia research and clinical decision-making. The study emphasizes that few-shot prompting with informed example selection allows these models to approach human-level accuracy for most variables without disrupting existing clinical workflows.

The paper positions this approach as a privacy-preserving solution, as open-weight models can be deployed locally on hospital infrastructure, avoiding the regulatory and security hurdles associated with proprietary cloud-based APIs. While acknowledging limitations—such as challenges with rare or ambiguous findings and the need for prospective validation—the authors conclude that open-weight LLMs represent a scalable, feasible tool for transforming unstructured radiology data into structured research assets.