GS-BrainText: A Multi-Site Brain Imaging Report Dataset from Generation Scotland for Clinical Natural Language Processing Development and Validation

1. Problem Statement

Clinical Natural Language Processing (NLP) holds significant potential for extracting structured data from unstructured electronic health records (EHRs) to support research and clinical decision-making. However, the translation of NLP tools from research to clinical practice is hindered by three critical limitations:

• Lack of Generalizability: Most available clinical text datasets originate from US healthcare systems, which differ in terminology, documentation practices, and healthcare processes from the UK NHS.
• Limited Scope: Existing datasets often represent single institutions or narrow patient populations, failing to capture the linguistic and demographic diversity required for robust system development.
• Data Scarcity: There is a significant shortage of publicly available, expertly annotated brain imaging report datasets, particularly for UK-based settings. While large-scale datasets exist (e.g., MIMIC-CXR), they focus on chest imaging or lack human-curated "gold standard" annotations for specific neurological phenotypes.

2. Methodology

Data Collection and Curation

• Source: The dataset is derived from the Generation Scotland cohort, a population-based study of ~24,000 participants.
• Scope: It includes 8,511 brain radiology reports (CT and MRI) spanning five Scottish NHS health boards (Fife, Lothian, Greater Glasgow and Clyde [GGC], Grampian, and Tayside) from 1994 to 2021.
• Pre-processing: Raw CSV reports were converted to XML. Structural sections (e.g., clinical history vs. radiologist findings) were identified. Clinical history sections were excluded from annotation.
• Annotation Subset: A subset of 2,431 reports was manually annotated. This subset includes a broad age range (mean age 58, median 53) and gender distribution (59% female, 41% male).

Annotation Schema

The team utilized a comprehensive schema originally developed for the EdIE-R system, covering:

• 24 Brain Disease Phenotypes: Categorized into Stroke-Related (11), Tumour-Related (4), and Other Neurological Findings (9).
  • Examples: Ischaemic stroke (deep/cortical, recent/old), Haemorrhagic stroke, Meningioma, Small vessel disease, Atrophy, Microbleeds.
• Annotation Levels:
  1. Text-level: Named entities (findings), location modifiers (deep, cortical, lobar), temporal modifiers (recent, old), negation attributes, and relations.
  2. Document-level: Binary labels indicating the presence/absence of each of the 24 phenotypes for the entire report.
• Process: Annotations were performed by a multidisciplinary team (neurologists, doctors, scientists, students) using the BRAT tool. To ensure efficiency, annotators performed systematic corrections of pre-existing EdIE-R outputs rather than annotating from scratch.
• Quality Assurance: Rigorous protocols included consensus meetings and double-annotation.
  • 100% of Lothian and Fife data was double-annotated.
  • 10–100% of data from other boards was double-annotated.
  • Inter-annotator agreement (IAA) F1-scores ranged from 83.74 to 95.65.

Benchmarking

The authors evaluated the existing rule-based NLP system, EdIE-R, on the full annotated dataset to establish a baseline.

• Metrics: Precision, Recall, and F1-scores were calculated at the phenotype, health board, and age group levels.
• Averaging: Both micro-averaged (weighted by frequency) and macro-averaged (unweighted) scores were reported to account for class imbalance.

3. Key Contributions

1. GS-BrainText Dataset: The creation of the first multi-site, population-based, expertly annotated brain radiology dataset for the UK NHS, addressing the gap in non-US clinical NLP resources.
2. Comprehensive Phenotype Taxonomy: A detailed schema covering 24 clinically significant cerebrovascular and neurological phenotypes with fine-grained modifiers (location, timing).
3. Benchmarking Evidence: A rigorous evaluation demonstrating that NLP performance varies significantly based on geography (health board), phenotype frequency, and patient age, highlighting the challenges of "out-of-the-box" deployment.
4. Linguistic Analysis: Identification of linguistic variations, such as the increased use of "hedging" language (expressing uncertainty) in reports for younger patients (<50 years).

4. Results

Performance by Phenotype

• Overall Performance: EdIE-R achieved a micro-average F1 of 88.82 and a macro-average F1 of 77.47.
• High-Performing Phenotypes: Common conditions like Small Vessel Disease (F1: 95.60) and Atrophy (F1: 95.69) were detected with high accuracy.
• Low-Performing Phenotypes: Rare or complex conditions showed lower performance.
  • Subarachnoid haemorrhage, aneurysmal: F1 = 22.22 (due to low recall of 14.29).
  • Ischaemic stroke, underspecified: F1 = 60.00 (due to low precision).
  • Microbleeds: Achieved perfect scores (F1=100) but are based on very few instances (n=2 and n=7), limiting statistical reliability.

Performance by Health Board

• Significant variation was observed across the five NHS boards, with F1 scores ranging from 86.13 (Tayside) to 98.13 (Fife).
• Notably, performance on Fife (98.13) was higher than on Tayside (86.13), despite Tayside data having been used to refine the system previously. This suggests that local reporting conventions and data heterogeneity heavily influence rule-based system performance.

Performance by Age Group

• Performance improved with age:
  • <50 years: F1 = 77.60
  • 50–69 years: F1 = 86.62
  • 70+ years: F1 = 91.70
• This correlates with the higher prevalence of specific phenotypes in older populations and the use of more definitive language in reports for older patients.

5. Significance and Implications

• Generalizability Challenge: The results underscore that NLP systems trained on single-institution data often fail to generalize even within the same national healthcare system (UK NHS). Local validation is critical before clinical deployment.
• Resource for Research: GS-BrainText provides a "gold standard" resource for developing and testing next-generation NLP tools (including Large Language Models) that must handle linguistic variation, demographic diversity, and multi-site heterogeneity.
• Clinical Impact: The dataset enables large-scale epidemiological studies and cohort identification that were previously infeasible due to the manual effort required to review unstructured radiology reports.
• Future Directions: The authors recommend future work focus on k-fold cross-validation (due to class imbalance) and the development of models that can adapt to local reporting conventions without extensive retraining.

Access: The dataset is available via a controlled access process through Generation Scotland to ensure patient privacy and ethical compliance.