Large Language Model-Assisted Cleaning of Report-Derived Labels in a Large-Scale Chest CT Dataset

Technical Summary: Large Language Model–Assisted Cleaning of Report-Derived Labels in a Large-Scale Chest CT Dataset

Problem Statement
Public medical imaging datasets are critical for benchmarking artificial intelligence models and training new algorithms. However, the quality of structured labels within these datasets is often compromised by reporting styles, negation, uncertainty, ambiguous definitions, and automated extraction errors. Even in carefully constructed datasets like CT-RATE (a large-scale public chest CT dataset with volumetric images, reports, and multi-abnormality labels), label-report discordance can occur. Because these datasets are repeatedly reused for benchmarking, such errors can propagate into model evaluation and comparison, potentially skewing performance estimates. There is a need for scalable methods to identify and correct these inconsistencies without relying solely on labor-intensive manual review.

Methodology
The authors developed an LLM-assisted framework to evaluate and clean report-derived labels in the CT-RATE dataset. The study utilized a retrospective dataset of 24,446 unique radiology reports (after deduplication) paired with 18 predefined abnormality categories.

1. LLM Extraction: Radiology reports (specifically the "findings" and "impression" sections) were processed using GPT-5.4 via Microsoft Azure AI Foundry. The model was prompted to generate binary labels (present/absent) for the 18 categories based strictly on the text, treating uncertain findings (e.g., "possible") as present. Outputs were constrained to a structured JSON format.
2. Comparison and Discordance Identification: The GPT-5.4-derived labels were compared against the original CT-RATE labels at the instance level. Discordant cases were categorized as either existing-label positive/GPT-negative or existing-label negative/GPT-positive.
3. Multi-Model Validation: To assess reliability, the same reports were processed using GPT-5.4 mini and DeepSeek V3.2. A multi-LLM majority-vote strategy was employed, where a label was assigned if at least two of the three models agreed.
4. Radiologist Adjudication: Three review sets were established for manual validation:
   • A random sample of 100 reports (1,800 label instances) annotated by a radiology fellow and finalized by a board-certified radiologist to serve as a reference standard.
   • A general discordance set of 100 instances (50 positive/negative and 50 negative/positive) excluding lymphadenopathy.
   • A targeted discordance set focusing on "lymphadenopathy," the only category with a Cohen's κ ≤ 0.80, sampling up to 50 discordant instances in each direction.
5. Statistical Analysis: Agreement was measured using Cohen's κ and accuracy metrics. Performance against the radiologist-annotated reference set was evaluated using sensitivity, specificity, PPV, NPV, F1 score, and Cohen's κ, calculated via label-macro averaging and micro-averaging.

Key Results

• Overall Agreement: The GPT-5.4-derived labels showed 96.4% overall agreement with the original CT-RATE labels, with a Cohen's κ of 0.884. Agreement varied significantly by category, ranging from 99.6% for hiatal hernia to 79.4% for lymphadenopathy.
• Lymphadenopathy Discrepancy: Lymphadenopathy exhibited the lowest agreement (κ = 0.309). In the targeted review of 99 discordant lymphadenopathy instances, radiologist adjudication supported the GPT-5.4-derived label in 91.9% of cases, suggesting the original CT-RATE labels were frequently incorrect for this category.
• General Discordance: In the general discordance review (excluding lymphadenopathy), radiologists supported the GPT-5.4-derived label in 74.2% of adjudicable instances, compared to 25.8% supporting the original CT-RATE label.
• Reference Standard Performance: Against the radiologist-annotated reference labels, the multi-LLM majority-vote approach achieved the highest performance with a label-macro-averaged F1 score of 96.0% and Cohen's κ of 0.951. This outperformed both the original CT-RATE labels (F1: 93.8%, κ: 0.923) and the single GPT-5.4 model (F1: 95.1%, κ: 0.940). Notably, majority voting achieved "almost perfect" agreement (κ > 0.80) across all 18 categories.

Significance and Claims
The paper claims that LLM-assisted label cleaning is a scalable and effective method for identifying clinically meaningful label-report discordance in large-scale public imaging datasets. The study demonstrates that:

1. Discordance Detection: LLMs can identify errors in existing dataset labels that may not be apparent from aggregate agreement metrics alone.
2. Quality Improvement: Multi-model majority voting can refine report-derived labels to achieve higher reliability than single-model extraction or original dataset labels.
3. Dataset Maintenance: This approach serves as a practical tool for dataset maintenance, helping to prioritize categories with ambiguous definitions (such as lymphadenopathy) for expert review and definition refinement.
4. Future Utility: The authors intend to release the cleaned dataset and the associated code to support future research, aiming to improve the reliability of benchmarks used for AI model development and evaluation.

The authors maintain a modest stance, acknowledging that their LLM-derived labels and adjudications are based on report text rather than image-level ground truth and that the discordance review sets were enriched for disagreement, meaning they do not represent direct estimates of overall dataset accuracy. They emphasize the necessity of human verification in validation subsets to mitigate potential biases introduced by LLM-generated labels.