Narrative Feature or Structured Feature? A Study of Large Language Models to Identify Cancer Patients at Risk of Heart Failure

Technical Summary: Narrative Feature or Structured Feature? A Study of Large Language Models to Identify Cancer Patients at Risk of Heart Failure

Problem Statement
Cancer treatments often introduce cardiotoxicity, leading to heart failure (HF) and negatively impacting survivorship. Identifying cancer patients at risk of HF is critical for improving treatment safety. While Electronic Health Records (EHRs) contain rich data for risk prediction, a gap exists in effectively utilizing structured medical codes (ICD and RxNorm) within Large Language Models (LLMs). Previous approaches have relied on traditional machine learning (ML) with sparse one-hot encoding or deep learning models like Time-Aware LSTMs (T-LSTM) that capture temporal sequences but still rely on sparse structured codes. It remains unclear how to best leverage the semantic capabilities of LLMs for structured EHR data to predict disease risk.

Methodology
The study utilized a cohort of 12,806 cancer patients (lung, breast, and colorectal) from the University of Florida Health system, comprising 1,602 cases (HF post-cancer) and 11,204 non-cases. The researchers compared three distinct feature representation strategies across different model architectures:

1. Traditional Machine Learning (One-Hot Encoding):

   • Models: Support Vector Machines (SVM) and XGBoost (XGB).
   • Features: Structured ICD and RxNorm codes were aggregated into PheWAS groups and ingredient-level RxCUI codes to reduce sparsity. These were converted into binary vectors (one-hot encoding).
   • Limitation: This approach simplifies the time-to-event structure and results in high feature sparsity.

2. Deep Learning (Sequence Representation):

   • Model: Time-Aware Long Short-Term Memory (T-LSTM).
   • Features: Patient data was organized into sequences based on encounter time to capture temporal dynamics. Diagnosis and medication codes were treated as sequential inputs with elapsed time tensors.
   • Limitation: While capturing temporal order, the input remains based on sparse structured codes.

3. Large Language Models (Subword Features):

   • Models: BERT, GatorTron-345M, and GatorTron-3.9B.
   • Features (Novel Strategy): The study proposed converting structured EHR data into "narrative features." Official descriptions of ICD and RxNorm codes were organized into a text sequence according to encounter time, with patient demographics converted into a narrative description. This text was tokenized into subwords by the LLM tokenizer.
   • Rationale: This strategy transforms sparse structured codes into dense text, allowing the LLM to leverage pre-trained semantic knowledge (e.g., understanding the relationship between "acute hepatitis B" and "acute myocarditis" via shared subwords like "acute") and increase feature density.

Key Results
The models were evaluated using F1 score, AUC, precision, recall, and accuracy on a 6:2:2 train/validation/test split.

• Feature Density: The "subword" feature representation significantly increased feature density (the number of unique patients per feature) compared to both original ICD codes and PheWAS groupings.
• Model Performance:
  • GatorTron-3.9B achieved the best overall performance with an F1 score of 0.699, outperforming:
    • Traditional SVM (F1 0.302) by 39%.
    • T-LSTM (F1 0.622) by 7%.
    • BERT (F1 0.643) by 5.6%.
  • GatorTron-3.9B also achieved the highest AUC (0.896) and accuracy (0.927).
• Feature Combinations: For both T-LSTM and GatorTron-345M, the best performance was achieved when combining diagnosis, medication, and demographic features.
• Subgroup Analysis: Performance varied by cancer type. GatorTron-3.9B showed the highest F1 score (0.824) and recall (0.913) for patients with "More than one cancer type," while breast cancer patients showed an F1 of 0.735.
• Interpretability: Using LIME, the study visualized attention weights, confirming that the model identified clinically relevant features such as "abnormal electrocardiogram," "rosuvastatin," and "furosemide" as drivers for HF risk prediction.

Significance and Claims
The paper claims that converting structured medical codes into narrative text (subword features) is an effective strategy for leveraging LLMs in disease risk prediction. The primary contribution is demonstrating that LLMs, specifically GatorTron, can outperform both traditional ML and sequence-based deep learning models when applied to structured EHR data, provided the data is transformed into a narrative format.

The authors argue that this approach offers a "straightforward yet efficient strategy" to bridge the gap between structured and unstructured EHRs. Unlike ontological grouping (e.g., PheWAS), which treats related conditions as distinct categories, LLMs using subword features can capture semantic similarities and variations between medical conditions. The study concludes that subword features serve as a superior surrogate for structured codes in LLMs, potentially improving cancer treatment outcomes by better identifying patients at risk of cardiotoxicity-induced heart failure.

Limitations
The authors acknowledge that the GatorTron models are limited by a 512-subword input window, which may truncate longer patient histories. Additionally, the dataset exhibited demographic imbalances, such as a higher proportion of Black patients in the case group and a gender skew in breast cancer cases. Future work is suggested to explore generative LLMs (GPT architecture) and automatic text summarization to further integrate structured and unstructured data.