LLMs for Cardiovascular Risk Prediction from Structured Clinical Data

Technical Summary: LLMs for Cardiovascular Risk Prediction from Structured Clinical Data

Problem Statement
Coronary artery disease (CAD) remains a leading cause of global mortality, necessitating reliable predictive systems for early diagnosis. While traditional machine learning (ML) models perform effectively on structured numerical clinical data (e.g., age, blood pressure, cholesterol), they often fail to leverage the rich, unstructured natural language found in electronic health records (EHRs), physician notes, and diagnostic reports. This creates a gap between the input requirements of standard ML algorithms and the narrative format of real-world clinical documentation. This work addresses the challenge of bridging structured data and natural language to improve CAD risk assessment while exploring the viability of Large Language Models (LLMs) in clinical prediction tasks.

Methodology
The authors propose a hybrid framework that integrates structured clinical data, synthetic narrative generation, and both conventional ML and LLM-based classification. The methodology proceeds through four primary stages:

1. Dataset Construction and Feature Engineering:
   The study utilizes a publicly available CAD dataset comprising 1,190 patient records with 11 clinical attributes (including age, sex, chest pain type, resting blood pressure, cholesterol, fasting blood sugar, ECG results, maximum heart rate, exercise-induced angina, ST depression, and ST slope). Coded variables are mapped to human-readable clinical categories (e.g., converting binary sex indicators to "male"/"female" and ECG results to descriptive terms) to create interpretable feature strings.

2. Synthetic Clinical Narrative Generation:
   Using LLMs (specifically GPT and Gemini), the structured feature strings are transformed into two distinct text formats:

   • Structured Summaries: Comma-separated feature-value pairs.
   • Free-Text Narratives: Detailed clinical reports written in the style of physician notes, describing the patient's condition in full sentences.
     The generation process employs specific temperature settings (0 for deterministic structured output; 0.3 for varied narrative flow) to ensure medical accuracy without inventing data.

3. Narrative Validation and Quality Assurance:
   A rigorous validation pipeline ensures the fidelity of the synthetic data. The LLM acts as a medical data validator to identify contradictions or missing values in the generated narratives. Subsequently, a "reverse extraction" process prompts the LLM to extract clinical values from the free-text narratives back into a structured format. A consistency score is calculated by comparing these extracted values against the original ground-truth records. Categorical fields require exact matches, while numeric fields allow a margin of error (<0.5) to account for text rounding.

4. Predictive Modeling:

   • Conventional ML: Four algorithms (Logistic Regression, Random Forest, Support Vector Machine, and XGBoost) are trained on the original structured data. The data is split 80/20 (stratified), standardized, and evaluated using accuracy, precision, recall, F1-score, and ROC-AUC. Hyperparameter tuning is performed on the best-performing model via GridSearchCV.
   • LLM-Based Classification: The synthetic free-text narratives are used to evaluate LLMs under zero-shot and few-shot prompting conditions (5, 10, and 15 examples). The models are instructed to output binary disease labels (presence/absence of CAD) without traditional weight training.

Key Contributions

• Hybrid Framework: The proposal of an integrated system that combines structured data, LLMs for narrative generation, and ML for prediction.
• Synthetic Narrative Pipeline: A novel method for converting tabular clinical records into natural-language clinical reports suitable for NLP tasks.
• Validation Mechanism: The development of a reverse-extraction consistency scoring system, which achieved an average fidelity of 94.61% across the dataset, ensuring that synthetic narratives accurately reflect original clinical data.
• Comparative Evaluation: A systematic comparison of traditional ML models against LLM-based classification under varying prompting strategies (zero-shot and few-shot).

Results

• Validation: The narrative generation pipeline achieved an average consistency score of 94.61%. Specifically, 45.3% of records achieved a perfect 100% score, and 95.4% scored 90% or higher.
• Machine Learning Performance: Among conventional models, Random Forest achieved the highest performance with a test accuracy of 92.86% and an ROC-AUC of 97.07%. XGBoost followed closely (92.44% accuracy), while Logistic Regression and SVM showed lower but competitive results.
• LLM Performance: LLM-based classification (using GPT and Gemini) demonstrated capability in interpreting natural language but yielded lower accuracy than the best ML models.
  • GPT: Achieved 66.7% accuracy in zero-shot, improving to 73.3% with 15-shot prompting.
  • Gemini: Achieved 60.0% accuracy in zero-shot, improving to 80.0% with 15-shot prompting.
  • Performance generally improved with increased few-shot examples, though it did not surpass the Random Forest baseline.

Significance and Claims
The paper claims that while traditional ensemble models (specifically Random Forest) currently outperform LLMs in raw predictive accuracy on structured data, LLM-based approaches offer distinct advantages for real-world clinical deployment. The primary significance lies in privacy preservation: LLMs can operate directly on natural language patient descriptions, meaning sensitive numerical data (exact lab values, specific blood pressure readings, diagnostic codes) do not need to be explicitly exposed or shared in raw formats. This allows for risk assessment based on narrative text alone.

The authors conclude that combining structured clinical data with LLM-generated narratives enables new directions for hybrid clinical prediction systems. They suggest that future work should focus on testing the framework on larger, diverse datasets, fine-tuning domain-specific models (e.g., ClinicalBERT), and integrating additional modalities like ECG or imaging data into clinical decision support systems.