RadTimeline: Timeline Summarization for Longitudinal Radiological Lung Findings

1. Problem Statement

Tracking disease progression in longitudinal radiology reports is critical for clinical decision-making but is time-consuming and prone to cognitive overload for clinicians. While AI tools exist for summarizing individual reports, most approaches generate unstructured text summaries. These unstructured formats make it difficult to:

• Verify specific temporal changes (fact-checking).
• Compare findings across different time points.
• Track the evolution of specific pathologies (e.g., a specific nodule) versus general observations.

Existing structured summarization methods often rely on pre-defined, fixed topic lists, which are too rigid for radiology where patients may have multiple distinct nodules in the same lung region that need individual tracking. The paper addresses the need for a structured, verifiable, and dynamic timeline that organizes findings by time and groups related findings together.

2. Methodology

The authors propose a 3-step Large Language Model (LLM) pipeline to generate a structured timeline. The output is a 2D table where:

• Columns represent time-stamped radiology exams.
• Rows represent groups of temporally related findings.
• Cells contain specific, atomized finding facts.

The 3-Step Pipeline:

1. Step 1: Finding Extraction
   • Goal: Extract atomized, lung-related facts from individual reports.
   • Process: An LLM processes a single report to output a Python list of findings.
   • Filtering: An automated filter (based on RadGraph) removes "absent" findings (e.g., "no new nodules") and irrelevant findings (e.g., cardiac issues) to focus on abnormalities.
2. Step 2: Group Name Generation
   • Goal: Generate interpretable natural language labels (row headers) for distinct finding trajectories.
   • Process: Instead of directly clustering findings, the LLM analyzes the longitudinal context (either full reports or just extracted findings) to generate a list of group names (e.g., "Right Upper Lobe Ground Glass Nodule").
   • Rationale: This intermediate step avoids expensive pairwise comparisons of all findings and provides semantic anchors for the next step.
3. Step 3: Group Assignment
   • Goal: Map each extracted finding to a generated group name.
   • Approaches Evaluated:
     • LLM-based: Prompting the LLM to assign findings to groups. Variations included Single vs. Multiple findings per prompt, Zero-shot vs. Few-shot, and Full Context vs. No Context.
     • Embedding-based: Using an embedder model (E5-Mistral) to calculate cosine similarity between findings and group names. Task-specific instructions were added to the embeddings to emphasize anatomical location and finding type.

3. Key Contributions

A. RadTimeline Dataset

The authors created a new evaluation dataset, RadTimeline, consisting of:

• Source: 10 patients from a lung cancer surveillance cohort (6 with future cancer diagnoses, 4 without).
• Scale: 65 chest-related exam reports (CTs and X-rays) spanning 5 years.
• Annotation: A "Gold Standard" timeline was created via a rigorous human-in-the-loop process involving medical students and a physician. This involved:
  • Factuality rating of extracted findings.
  • Group assignment (clustering related findings).
  • Refinement of group names.
• Inter-annotator Agreement: High agreement (86% for factuality, 82 CoNLL F1 for grouping), validating the dataset quality.

B. Structured Summarization Framework

The paper introduces a novel 2D timeline format that facilitates:

• Direct Comparison: Side-by-side viewing of findings across time.
• Traceability: Every fact is linked to its source report, enabling fact-checking.
• Dynamic Grouping: Unlike fixed-topic systems, the group names are generated dynamically based on the patient's specific findings.

C. Evaluation Metrics

The paper proposes specific metrics beyond standard ROUGE:

• Factuality: Correctness of extracted details.
• Group Name Quality: Specificity and relevance of generated row headers.
• Grouping Performance: Measured via CoNLL F1 (standard for coreference resolution tasks).

4. Experimental Results

Finding Extraction

• Performance: GPT-4o outperformed Llama 3.1 (8B) in both ROUGE-L scores and reduction of "unnecessary" (irrelevant/wrong) predictions.
• Filtering: Aggressive filtering of absent findings significantly reduced noise without increasing missed findings.

Group Name Generation

• Context Impact: Using full report contexts yielded better group names than using only extracted findings (finding-only context), as the full context helps merge related details.
• Model Comparison: GPT-4o generated more specific and accurate group names than Llama 3.1. Llama 3.1 tended to generate more generic or duplicate names.

Group Assignment (The Core Task)

• LLM vs. Embeddings:
  • LLM-based methods achieved the highest performance, with the best configuration (GPT-4o, Multiple findings, Zero-shot) reaching 84 CoNLL F1, which is comparable to human annotator agreement (82).
  • Embedding-based methods performed well (up to 83 CoNLL F1) but required task-specific instructions (emphasizing anatomy and finding type) to approach LLM performance.
• Importance of Intermediate Step: The "Group Name Generation" step was critical. Removing this step (assigning groups without pre-generated names) caused a significant performance drop (~70 CoNLL F1), proving that generating interpretable labels first aids the grouping logic.
• Context Length: For Llama 3.1, full context did not always improve grouping performance compared to no context, suggesting potential limitations in utilizing long contexts for this specific task. GPT-4o generally benefited from full context.

5. Significance and Future Work

• Clinical Utility: The RadTimeline format directly addresses the clinical need for tracking disease progression (e.g., nodule growth, resolution of pneumonia) in a way that is auditable and easy to scan.
• Scalability: The approach demonstrates that off-the-shelf LLMs can achieve near-human performance in complex longitudinal summarization tasks without supervised fine-tuning, provided a structured 3-step pipeline is used.
• Limitations:
  • The dataset is small (10 patients) due to the high cost of expert annotation.
  • The study relies on proprietary models (GPT-4o) due to HIPAA compliance requirements, limiting reproducibility for open-source models.
  • The "Gold Standard" was derived from Llama 3.1 outputs corrected by humans, which may introduce bias favoring Llama's style.
• Future Directions: The authors suggest extending this framework to other clinical domains (symptom progression, medication history) and refining the dataset with de-identified data for broader release.

In conclusion, RadTimeline establishes a new benchmark for structured longitudinal summarization in radiology, demonstrating that a pipeline combining extraction, label generation, and assignment can effectively automate the creation of verifiable, clinically useful timelines.