Natural Language Processing Based Solution for Labeling Brain Metastasis Identified in Radiology Reports

Technical Summary: Natural Language Processing Based Solution for Labeling Brain Metastasis Identified in Radiology Reports

Problem Statement
Brain metastases (BM) represent a significant burden in neuro-oncology, far exceeding the volume of primary central nervous system tumors. However, accurate quantification of BMs remains elusive because current cancer registries are not mandated to capture metastatic progression. While registries capture synchronous BMs (present at diagnosis), they miss the majority of asynchronous BMs that develop after the initial cancer diagnosis. Identifying these asynchronous cases currently relies on manual record review by trained registrars, a resource-intensive process that limits population-level surveillance. Existing Natural Language Processing (NLP) attempts often rely on keyword extraction or analyze only the "Impression/Conclusion" section of radiology reports, potentially missing up to 36% of relevant findings documented solely in the "Findings" section. Furthermore, few studies have validated model generalizability across heterogeneous data sources or real-world workflows.

Methodology
The study developed and validated an NLP pipeline using population-based cancer registry data from Alberta, Canada (diagnoses 2012–2019, follow-up to 2022), with external validation in Ontario and British Columbia. The methodology proceeded through four phases:

1. Data Curation and Sampling:

   • Phase 1 (Initial Training): To address the low prevalence of BM, a stratified sampling strategy was employed using Alberta Cancer Registry data. Patients were categorized by primary brain tumor status and synchronous metastasis status to enrich the dataset. 1,200 reports were manually annotated by research assistants under radiologist guidance.
   • Phase 2 (Model-Guided Sampling): To capture asynchronous BMs, the Phase 1 model was applied to 30,000 post-diagnosis reports. Predicted probabilities were used to guide stratified sampling, resulting in 1,011 additional manually annotated reports.
   • Final Training Set: The consolidated training set comprised 1,879 reports (505 BM-positive).

2. Model Architecture:

   • Base Model: Bio_ClinicalBERT (initialized with BioBERT and pre-trained on MIMIC-III) was selected to handle clinical terminology.
   • Section Handling: To respect the model's 512-token limit and avoid truncating 25% of reports, the pipeline isolated the "Findings" and "Impressions" sections using regular expressions.
   • Ensemble Strategy: Two separate models were trained, one for each section. The final report-level prediction was determined by taking the maximum predicted probability from the two component models.
   • Missing Data Handling: In internal validation and Ontario external validation, missing sections were imputed with the full report text. In the British Columbia validation, missing sections were treated as empty strings.

3. Validation Strategy:

   • Internal Validation: Tested on 1,833 reports from 357 patients (Alberta) diagnosed between 2012–2013, with no overlap with training data.
   • External Validation:
     • Ontario: 17,402 reports from St. Michael's Hospital (Toronto) were filtered via stratified sampling.
     • British Columbia: 908 reports from the BC Cancer Registry were manually reviewed.
   • Annotation: All validation sets underwent manual annotation by medical professionals, with discrepancies adjudicated by senior clinicians. Reports labeled as primary brain tumors were excluded from BM performance evaluation.

Key Results

• Internal Validation: At a probability threshold of 0.4, the ensemble model achieved a sensitivity of 0.888 and a precision of 0.499.
• Ensemble Performance: The ensemble approach substantially outperformed single-section models. Single-section models achieved sensitivities of only 67.8% (Findings only) and 74.2% (Impressions only), demonstrating the necessity of processing both sections.
• External Validation: The model demonstrated robustness across diverse data distributions:
  • Ontario: Sensitivity of 0.918.
  • British Columbia: Sensitivity of 0.726.

Significance and Claims
The paper claims that this NLP-based pipeline successfully addresses the operational requirements of cancer registries by enabling scalable surveillance of asynchronous brain metastases. By processing both "Findings" and "Impressions" sections, the system captures a higher volume of relevant cases than single-section approaches. The authors state that the model meets the sensitivity standards required for registry operations and is being implemented into surveillance workflows in Alberta and British Columbia. This implementation provides a foundation for population-level BM surveillance, moving beyond the limitations of current manual review processes and registry mandates that only capture synchronous cases.