Can Language Models Identify Side Effects of Breast Cancer Radiation Treatments?

Technical Summary: Can Language Models Identify Side Effects of Breast Cancer Radiation Treatments?

Problem Statement
Accurately communicating treatment side effects is critical for cancer survivors, particularly in informed consent and survivorship care planning. However, this task is hindered by fragmented electronic health record (EHR) data and clinical knowledge deficits regarding long-term treatment toxicities. While Large Language Models (LLMs) offer potential to summarize and communicate these effects, their reliability in oncology remains poorly understood. Specifically, it is unclear how LLMs perform when patient documentation varies in specificity (e.g., general mentions of radiation vs. specific anatomical targets), how they handle different prompting strategies, and whether they can reliably capture rare or long-term side effects essential for survivorship care.

Methodology
The authors propose a deployment-oriented stress-testing framework to evaluate LLM reliability in identifying breast cancer radiation side effects. The study utilizes 21 synthetic breast cancer patient profiles derived from public datasets (Chen et al., 2023; Palepu et al., 2025).

• Experimental Design: For each patient profile, the authors created paired versions: a "Base Profile" mentioning radiation generally, and a "Specified Profile" adding specific radiation types or anatomical locations (e.g., "chest wall," "nodes"). All other clinical information remained fixed to isolate the effect of documentation specificity.
• Ground Truth: A clinician-curated reference list was developed by a team of over seven breast radiation oncologists, mapping radiation modalities and locations to established side effects. Each side effect was annotated by frequency (Common to Extremely Rare) and temporal onset (Short-term vs. Long-term).
• Models Evaluated: Seven instruction-tuned LLMs were tested: LLaMA-3.1-8B, Gemma-2-9B, Mistral-7B, Qwen-2.5-7B, Phi-3-Medium, O4-Mini, and GPT-5.3.
• Prompting Regimes: Four regimes were evaluated:
  1. Free-form generation (unconstrained).
  2. Free-form generation with a list-size constraint (20–30 items).
  3. Selection from a predefined, clinician-curated list.
  4. Selection from a predefined list with a list-size constraint.
• Evaluation Metrics: Standard Precision, Recall, and F1 scores were calculated against the clinician reference. Additionally, an Overlap Ratio was computed between outputs from base and specified profiles to measure robustness to minor documentation changes. Semantic matching was performed using an LLM to align generated text with the ground truth, validated against human judgment with 96% agreement.

Key Results

• Precision-Recall Trade-off: In free-form generation, models exhibited significant divergence. Some (e.g., Gemma-2-9B, O4-Mini, GPT-5.3) achieved high precision but low recall, missing many relevant effects. Others (e.g., LLaMA-3.1-8B) produced broad lists with moderate recall but very low precision, generating hallucinated or anatomically irrelevant toxicities.
• Impact of List Size Constraints: Imposing a list-size constraint (20–30 items) in free-form generation consistently reduced precision across all models without reliably improving recall. This suggests that forcing length targets incentivizes the fabrication of incorrect side effects.
• Grounding Improves Reliability: When models were restricted to selecting from a clinician-curated list, precision and recall improved substantially across all models. In this grounded setting, list-size constraints generally increased recall without the severe precision drops seen in free-form generation.
• Sensitivity to Documentation: Models demonstrated low overlap ratios between base and specified profiles, indicating that minor changes in documentation (e.g., specifying a location) led to substantial shifts in the generated side-effect lists.
• Survivorship Gaps: Most models systematically under-recalled rare and long-term side effects, favoring common, short-term toxicities. Notable exceptions were O4-Mini and GPT-5.3, which showed stronger recall for long-term effects.

Key Contributions

1. Stress-Testing Framework: Introduction of a framework for evaluating LLMs in oncology that emphasizes clinically meaningful perturbations (documentation specificity) and survivorship-relevant distinctions (frequency, temporal onset) rather than abstract benchmarks.
2. Structured Evaluation: The first structured experimental analysis of how prompting strategies, documentation specificity, and side-effect characteristics jointly influence LLM performance in breast cancer radiation side-effect generation.
3. Design Recommendation: Demonstration that grounding LLM outputs in expert-curated vocabularies significantly improves reliability, offering a practical design choice for safer applications.
4. Public Resource: Release of a dataset containing 21 patient profiles paired with an oncologist-annotated set of radiation treatments and side effects.

Significance and Claims
The paper argues that while LLMs show promise, their current reliability in oncology is limited by sensitivity to input phrasing, a tendency to hallucinate in unconstrained settings, and a bias toward common, acute side effects over rare, long-term ones. The authors claim that their findings highlight the necessity of grounding LLMs in expert-curated lists for high-stakes clinical tasks like informed consent and survivorship education. They emphasize that application-driven evaluation, which accounts for documentation variability and clinical nuances, is essential before deploying LLMs in patient-facing tools. The work serves as a precursor to more complex workflow integrations, advocating for rigorous stress-testing to ensure safety and utility in cancer survivorship care.