AI-Assisted Pneumonia Detection, Localisation and Report Generation from Chest X-rays

This paper presents a holistic deep learning pipeline that leverages LLM-driven relabelling of a massive chest X-ray dataset to achieve superior pneumonia detection, localisation, and report generation performance compared to traditional rule-based methods and radiologist benchmarks.

The Gist

Imagine you are a detective trying to solve a mystery: Is this patient sick with pneumonia? Your clues are Chest X-rays (black and white photos of the inside of a chest) and the written reports radiologists (the doctors who read X-rays) have already written about them.

For years, computers have tried to help solve this mystery, but they've been like a student who memorized the wrong answers from a bad textbook. They often get confused, miss the sickness, or get it wrong because the "textbook" (the data they were trained on) had errors.

This paper introduces a new, super-smart detective team that fixes these problems using three main tricks. Here is how they did it, explained simply:

1. The "Translator" Trick: Fixing the Bad Textbook

The Problem:
The biggest datasets of X-rays available to researchers came with labels (answers) generated by simple computer programs called "Rule-Based NLP." Think of these programs like a very literal, slightly confused robot. If a report said, "No pneumonia found," the robot was good. But if a report said, "Pneumonia is possible, but maybe it's just a cold," the robot would get confused and sometimes mark it as "Yes, pneumonia!" or "No, pneumonia!" incorrectly. This is like a teacher grading a test with a broken answer key.

The Solution:
The researchers didn't just use the robot's answers. They hired a super-smart AI Translator (a Large Language Model or LLM).

• The Analogy: Imagine you have a stack of 200,000 messy, handwritten notes from a doctor. A simple robot tries to read them and gets it wrong 25% of the time. Instead, you hire a brilliant, experienced translator (the LLM) who reads the context of the whole sentence.
• The Result: This AI Translator re-read all the reports and corrected the labels. It turned a messy, confusing pile of notes into a clean, accurate textbook. The new labels agreed with human experts 96.5% of the time, compared to only 72.5% for the old robot labels.

2. The "Eagle Eye" Trick: Seeing the Spot

The Problem:
Even if a computer says, "Yes, there is pneumonia," doctors need to know where it is. Is it in the top left? The bottom right? Old AI models were like someone shouting "Fire!" in a building but pointing at the ceiling instead of the kitchen. They often got the diagnosis right but couldn't point to the specific spot.

The Solution:
The team used a technique called Grad-CAM.

• The Analogy: Imagine the AI is looking at the X-ray through a pair of glasses that glow red where it thinks the sickness is. This "heat map" shows exactly which parts of the lung are lighting up in the AI's mind.
• The Result: The AI didn't just guess; it focused its "attention" on the actual white spots (inflammation) in the lungs. While it wasn't perfect at pinpointing the exact zone (it was about 53% accurate at the specific location), it proved the AI was looking at the right parts of the body, not just guessing based on random patterns.

3. The "Reporter" Trick: Writing the Summary

The Problem:
Doctors are busy. They spend 5–10 seconds looking at an X-ray. If a computer could just say "Pneumonia detected," that's helpful, but if it could also write a draft of the report, that would be a game-changer.

The Solution:
Because the AI knew what was wrong and where it was (thanks to the heat map), they fed that information back into the AI Translator.

• The Analogy: It's like the AI acts as a junior doctor. It looks at the X-ray, sees the red glow in the bottom right lung, and then writes a sentence: "There is an opacity in the right lower lung zone suggestive of pneumonia."
• The Result: The system can now generate a structured report draft automatically, saving the human doctor time.

The Big Win: Beating the Humans (Sometimes)

The researchers tested their new detective against:

1. Old AI models (trained on the bad textbook).
2. CheXNet (a famous previous AI model).
3. Human Radiologists (the experts).

The Scoreboard:

• Old AI: Missed many cases.
• CheXNet: Missed about half the cases.
• Human Radiologists: Missed between 22% and 36% of cases (even experts get tired and miss things).
• The New AI: Caught 82% of the pneumonia cases.

Why does this matter?
This isn't about replacing doctors. It's about giving them a super-powered assistant.

• In a busy emergency room, this AI can act as a "second pair of eyes" to catch cases the tired human might miss.
• It can prioritize patients: "Hey, look at this one first, the AI is 99% sure it's pneumonia."
• It can draft the report so the doctor just has to sign it.

In a Nutshell

The researchers took a massive pile of X-rays, used a smart AI to clean up the messy labels (the "textbook"), trained a new model to be a better detective, and gave it the ability to point out exactly where the sickness is and write a report about it. The result is a tool that is more accurate than previous computers and catches more pneumonia cases than even the best human doctors do on their own. It's a step toward a future where AI handles the heavy lifting, letting human doctors focus on the most complex cases and the patients themselves.

Technical Summary

1. Problem Statement

Pneumonia is a leading cause of mortality, yet its diagnosis via Chest X-rays (CXRs) is hindered by:

• High Inter-observer Variability: Radiologists often disagree on interpretations due to the subjective nature of radiographic patterns and low contrast in CXRs.
• Label Noise in Training Data: Existing Deep Learning (DL) models often rely on labels derived from radiology reports using Rule-based Natural Language Processing (rNLP). These methods frequently produce erroneous labels (e.g., misclassifying "no pneumonia" reports as positive due to keyword matching), leading to poor generalizability in real-world settings.
• Lack of Explainability: Many DL models act as "black boxes," lacking the visual or textual justification required for clinical adoption.
• Workflow Bottlenecks: Radiologists spend only 5–10 seconds per image, increasing the risk of oversight.

2. Methodology

The authors propose a holistic Computer-Aided Diagnosis (CAD) pipeline integrating data curation, deep learning, and Large Language Models (LLMs).

A. Data Acquisition and Curation

• Dataset Scale: The study curated the largest composite of publicly available CXRs to date (N = 922,634 images) from six datasets: MIMIC-CXR, VinDr-CXR, CheXpert, PadChest, ChestX-ray14, and RSNA-Pneumonia.
• Training Subset: Approximately 16,000 images were used for training after rigorous filtering (excluding lateral views, poor-quality images, and non-anatomical artifacts).
• Preprocessing:
  • Standardization: DICOM images were normalized, grayscale inversion corrected, and intensity ranges clipped.
  • Segmentation: A pre-trained pyramid scene parsing network (from torchxrayvision) isolated the trunk. Background pixels were clipped to black, and Contrast Limited Adaptive Histogram Equalisation (CLAHE) was applied to enhance contrast.
  • View Classification: A residual network filtered out lateral views, retaining only frontal projections.

B. Label Relabelling via LLM

• Hypothesis: LLM-derived labels would be more accurate than rNLP labels.
• Process: Radiology reports from MIMIC-CXR (214,332 reports) were processed by a locally deployed, open-weight LLM (DeepSeek-R1-Distill-Llama-8B).
• Prompt Engineering: The LLM was instructed to act as an expert radiologist, classifying reports as Positive, Negative, or Uncertain based on context (e.g., distinguishing "resolved pneumonia" from "active pneumonia"). It also extracted positional descriptors.
• Verification: A random sample of 200 reports was manually annotated by three experts. The LLM labels showed 96.5% agreement with human labels, compared to 72.5% for the original rNLP labels.

C. Deep Learning Architecture

• Model: DenseNet-121 initialized with ImageNet weights.
• Configuration: Four training configurations were tested:
  1. MIMIC-CXR (rNLP labels)
  2. MIMIC-CXR (LLM labels)
  3. MIMIC-CXR (rNLP) + VinDr-CXR
  4. MIMIC-CXR (LLM) + VinDr-CXR
• Training Details: Images resized to 480×480; augmented via flipping, rotation, and brightness adjustments. Trained with Adam optimizer, cross-entropy loss, and early stopping based on validation sensitivity.
• Explainability (Grad-CAM): Gradient-weighted Class Activation Mapping was used to generate heatmaps. These were mapped to anatomical lung zones (upper, middle, lower) to localize opacities.
• Report Generation: The LLM was used to generate structured text summaries based on the model's classification and localization.

3. Key Contributions

1. LLM-Driven Label Curation: Demonstrated that using an LLM to relabel radiology reports significantly reduces label noise compared to traditional rNLP, resulting in a more reliable training dataset.
2. Holistic Pipeline: Integrated detection, localization, and structured report generation into a single workflow.
3. Multi-Dataset Generalization: Validated the model across six diverse datasets, proving robustness beyond the training distribution.
4. Explainability: Provided visual (Grad-CAM) and textual (LLM summary) explanations, addressing the "black box" barrier in clinical AI.

4. Results

• Label Quality: LLM relabelling reduced positive cases in MIMIC-CXR by ~47% (correcting over-positivity in rNLP) and achieved near-perfect agreement with human annotators (Cohen's $\kappa$ = 0.93).
• Classification Performance:
  • The best model (MIMIC-CXR (LLM) + VinDr-CXR) achieved 82.08% Sensitivity and 81.97% Precision.
  • This outperformed the reported sensitivity range of human radiologists (64–77.7%) and significantly surpassed CheXNet's F1-score (43.5%).
  • Models trained on LLM labels showed a 4–5% improvement in precision and sensitivity over rNLP-trained counterparts.
• Localization:
  • Grad-CAM localization achieved a moderate F1-score of 52.9% (Sensitivity: 65.7%, Precision: 44.3%).
  • Visual inspection confirmed the model focused on lung fields and pathological opacities, avoiding spurious activations outside the thorax.
• Error Analysis: False negatives were disproportionately associated with left lower lobe pneumonia, likely due to obscuration by the cardiac silhouette.
• Cross-Dataset Performance: The model maintained high performance on VinDr-CXR (F1=85.83%) and MIMIC-CXR (F1=81.35%), while performing lower on datasets with known label noise (e.g., ChestX-ray14, F1=69.85%).

5. Significance and Clinical Impact

• Superior to rNLP: The study proves that LLM-based label curation is a critical step in preparing high-quality medical imaging datasets, directly translating to better model performance.
• Clinical Workflow: The pipeline offers tools for rapid triage, automated report drafting, and real-time surveillance, potentially reducing diagnostic delays and radiologist burnout.
• Generalizability: By training on diverse, high-quality data and validating across multiple independent datasets, the model addresses the common issue of poor generalization in medical AI.
• Future Directions: The authors suggest extending the pipeline to multi-label pathology classification, pediatric datasets, and prospective clinical validation.

Conclusion: The paper establishes that combining Deep Learning with LLM-driven data curation creates a robust, explainable, and high-performing system for pneumonia detection that exceeds current radiologist benchmarks and offers a viable path toward clinical deployment.