Large Language Models to Label CT Radiology Reports for Incidental Findings Found in Lung Cancer Screening: A Comparative Analysis

Technical Summary: Large Language Models to Label CT Radiology Reports for Incidental Findings in Lung Cancer Screening

Problem Statement
The implementation of Lung Cancer Screening (LCS) programs across the European Union faces significant challenges regarding the workload required for radiologists to interpret large volumes of low-dose computed tomography (LDCT) scans. Beyond lung cancer detection, these scans frequently reveal Specific Incidental Findings (SIFs), including aortic valve disease, bronchiectasis, consolidation, coronary artery calcification, emphysema, interstitial lung abnormalities (ILA), mediastinal lymph node abnormalities, mediastinal mass, and thyroid abnormalities. While Artificial Intelligence (AI) has advanced in detecting pulmonary nodules, algorithms for SIFs remain in early development. A primary bottleneck is the scarcity of large, annotated datasets required to train such models. Existing datasets, such as the National Lung Screening Trial (NLST), lack comprehensive SIF annotations. While clinical radiology reports offer a rich source of information for "weak supervision," traditional keyword-based search methods fail to capture medical nuances, negations, and terminology variations. Furthermore, while Large Language Models (LLMs) show promise, there is a lack of validated strategies for using them to label SIFs in LCS reports, particularly regarding the efficacy of different prompting techniques (e.g., Chain-of-Thought) and the performance of lightweight, privacy-preserving open-source models compared to proprietary ones.

Methodology
This retrospective study evaluated the feasibility and optimal strategies for using LLMs to automatically label the presence or absence of nine SIFs in Dutch radiology reports.

• Dataset: The study utilized 300 CT thorax/abdomen reports from a Dutch academic hospital (collected between 2003 and 2020). The dataset was constructed by selecting 200 reports from 2016–2017 and 100 reports from the full range (1997–2021) to ensure diversity. All reports were de-identified.
• Reference Standard: A single general radiologist with 32 years of experience manually labeled the reports for the nine SIFs, serving as the ground truth.
• Models Evaluated:
  • Proprietary: GPT-5 and GPT-4-Turbo (via OpenAI API).
  • Open-Source: Llama-3.1-8B-INST and MedGemma-4B-IT (deployed locally).
  • Baseline: A keyword-based search method (regex).
• Prompting Strategies: Four distinct strategies were tested for each model:
  1. V1: General prompt to label all nine SIFs without specific instructions on determination criteria.
  2. V2: Specific prompts per SIF, including criteria from the European radiological society statement.
  3. V3: Same as V2 but with Chain-of-Thought (CoT) reasoning.
  4. V4: Same as V3 but with few-shot examples (three examples of report, reasoning, and output).
• Evaluation Metrics: Agreement was primarily measured using Cohen's Kappa ($\kappa$), interpreted via Landis and Koch (1977) categories (from slight to almost perfect). Secondary metrics included sensitivity, specificity, and F1 scores. All metrics included 95% confidence intervals calculated via bootstrapping.

Key Results

• Overall Performance: LLMs achieved moderate to almost perfect agreement with the radiologist across most SIFs, validating their utility for automated labeling. However, no single model-prompt combination was optimal for all nine findings.
• Model Performance:
  • GPT-5 achieved substantial or almost perfect agreement for seven of the nine SIFs (including AVD, bronchiectasis, consolidation, CAC, emphysema, MLN, and TA).
  • MedGemma-4B-IT (open-source) demonstrated substantial or higher agreement for most SIFs and notably outperformed the best proprietary model specifically for Interstitial Lung Abnormalities (ILA) ($\kappa = 0.48$ vs. $0.46$).
  • Llama-3.1-8B-INST generally showed lower agreement than the proprietary models and MedGemma.
  • Keyword Search performed best only for Mediastinal Mass (MM), likely due to low linguistic variability for this specific finding, but generally underperformed compared to LLMs due to poor handling of negation and context.
• Prompting Strategy: Contrary to expectations, Chain-of-Thought (CoT) prompting (V3 and V4) generally reduced performance compared to non-CoT strategies (V1 and V2). The highest agreement values were predominantly obtained with V1 or V2. The inclusion of few-shot examples (V4) did not improve agreement and, in some cases, degraded performance, potentially due to models allocating less attention to instructions or focusing on irrelevant parts of longer prompts.
• Error Analysis: Disagreements often stemmed from models misinterpreting context (e.g., labeling AVD based on aortic arch references) or "overthinking" via CoT, leading to nonsensical justifications.

Significance and Claims
The authors conclude that LLMs are a viable tool for the "weak-annotation" of radiology reports to scale training data for future SIF detection algorithms. The study highlights that:

1. No Universal Strategy: There is no single optimal model or prompt for all SIFs; selection must be tailored to the specific finding.
2. Prompt Engineering: For this specific task, concise non-CoT prompts are superior to CoT or few-shot approaches.
3. Open-Source Viability: Lightweight, open-source models (specifically MedGemma-4B-IT) can achieve performance comparable to or exceeding heavier proprietary models for certain findings, offering a privacy-preserving and cost-effective alternative for local deployment.
4. Recommendation: The authors suggest a pipeline for future weakly supervised labeling that begins with concise non-CoT prompts and, where privacy is critical, utilizes lightweight open-source models. They emphasize that model and prompt selection should be validated against an expert-labeled dataset for each specific finding before large-scale deployment.

Limitations
The study acknowledges several constraints: the small dataset size (300 reports) resulted in very few cases for rare findings like mediastinal mass, limiting statistical certainty; the use of a single radiologist as the reference standard prevents assessment of inter-reader variability; the study is limited to Dutch reports from a single center; and the rapid evolution of LLMs means these results may not reflect future model capabilities. Statistical significance testing between models was not performed due to the exploratory nature of the study.