Regression vs. Medical LLMs: A Comprehensive Study for CVD and Mortality Risk Prediction

This study evaluates the performance of traditional regression models against medical large language models (MedLLMs) for predicting cardiovascular disease and mortality risks using the LURIC dataset, finding that while optimized MedLLMs and boosting techniques achieve competitive AUROCs up to 85%, MedLLMs require calibration adjustments to correct systematic over-prediction.

The Gist

Imagine you are a doctor trying to predict which of your heart patients might face a serious health crisis in the next year. Traditionally, you've relied on a set of mathematical recipes (called regression models) that look at a patient's numbers—like cholesterol levels, blood pressure, and age—to calculate a risk score. These recipes have been the gold standard for decades, but they are rigid; they follow strict rules and can't really "think" about the bigger picture.

Now, a new contender has entered the arena: Medical Large Language Models (MedLLMs). Think of these not as calculators, but as super-smart medical interns who have read every medical textbook, journal, and case study in existence. They don't just crunch numbers; they understand the story behind the numbers.

This paper is a massive "cook-off" to see who wins: the old-school mathematical recipes or the new-school AI interns.

The Setup: The "LURIC" Kitchen

The researchers used a giant, well-stocked pantry of data called the LURIC study. It contains detailed records of over 3,300 heart patients from Germany.

• The Ingredients: Instead of using expensive, hard-to-get doctor's notes (like a long essay written by a physician), they used routine blood test results and basic health stats. Think of these as the standard ingredients you find in any kitchen: cholesterol, kidney function, age, smoking history, etc.
• The Goal: Predict if a patient will pass away within one year (a very serious prediction, but crucial for knowing who needs extra care).

The Contestants

1. The Old Guard (Regression Models)
These are the veteran chefs. They include algorithms like CatBoost and XGBoost.

• How they work: They are like a master chef who has memorized thousands of recipes. If "Cholesterol is high + Age is over 60," they instantly know the risk is high. They are fast, reliable, and very good at handling structured numbers.
• The Result: They performed excellently, hitting a success rate (AUROC) of about 85%. This is the benchmark everyone else had to beat.

2. The New Kids (Medical LLMs)
These are the AI interns. The researchers tested two ways to use them:

• The "Prompting" Method (Zero/Few-Shot): Imagine handing the AI intern a list of patient numbers and saying, "Here is a patient's data. Based on your training, what is their risk?"
  • The Twist: The researchers tried giving the AI a few examples first (like showing it three other patients and their outcomes) before asking about the new one. This is called "few-shot prompting."
  • The Result: The big AI models (like the 70-billion-parameter ones) did surprisingly well, reaching 82%. They were almost as good as the veteran chefs, but they needed a little help (the examples) to get there.
• The "Finetuning" Method: Imagine taking a smart intern and giving them a crash course specifically on this hospital's data.
  • The Result: Smaller AI models (8-billion parameters) that were "finetuned" actually beat the big models and even surpassed some commercial giants like ChatGPT. They reached 82-85%, matching the veteran chefs.

The Big Surprises

1. The "Small but Mighty" Discovery
Usually, in the world of AI, bigger is better. A 70-billion-parameter model is a giant brain; an 8-billion one is a smaller brain. But this study found that if you train the small brain specifically on heart disease data, it can outperform the giant, untrained brain. It's like a specialized mechanic who knows your specific car model better than a generalist who knows everything about cars but nothing about yours.

2. The "Over-Confident" Problem (Calibration)
While the AI interns were good at ranking patients (knowing who is riskier than whom), they were bad at guessing the exact percentage.

• The Metaphor: Imagine the AI says, "There is a 90% chance this patient will be fine." But in reality, the chance was only 80%. The AI was slightly too optimistic (or in some cases, too pessimistic).
• The Fix: The researchers applied a "calibration filter" (called Platt scaling). Think of this as a translator that takes the AI's raw guess and adjusts it to match reality. This simple step fixed the AI's confidence issues, making its predictions much more trustworthy for doctors.

3. No Need for the "Essay"
Previous studies tried to feed these AI models long, messy doctor's notes. This study proved you don't need the essay. You can just feed the AI the raw numbers (the blood test results), and it can still do a fantastic job. This is huge because it means this technology can be used in almost any hospital, even those that don't have perfect digital records.

The Verdict

The paper concludes that Medical AI is ready for the big leagues.

• The Old Guard (Regression) is still the reliable workhorse, especially when you need speed and simplicity.
• The New Guard (MedLLMs) is a powerful new tool. With the right "prompt" (instructions) or a little bit of specific training, they can match or even beat the best traditional methods.

The Takeaway:
We don't have to choose between the old math and the new AI. The future is likely a hybrid kitchen: using the speed of traditional math for quick checks, and the deep understanding of AI for complex cases, all while using the simple, everyday blood test data that hospitals already collect. It's a win for patients, because better prediction means better, earlier care.

Technical Summary

1. Problem Statement

Cardiovascular diseases (CVDs) are the leading cause of global mortality, necessitating robust tools for early risk detection and preventive intervention. While traditional machine learning (ML) methods (e.g., logistic regression, SVMs, gradient boosting) have long been the standard for CVD risk estimation, they often rely on assumptions of linearity and homogeneous effects.

Recent advances in Generative AI and Large Language Models (LLMs) offer new paradigms, but their application in clinical settings faces specific challenges:

• Data Dependency: Most existing LLM studies rely on unstructured textual Electronic Health Records (EHRs) or discharge summaries, which are costly to produce and inconsistently available.
• Performance Gap: Previous studies using open-source LLMs for mortality prediction often lagged behind traditional Artificial Neural Networks (ANNs) and regression models, particularly when used in zero-shot settings without domain adaptation.
• Calibration Issues: LLMs often exhibit poor probability calibration, leading to systematic over- or under-prediction of risk, which is critical in clinical decision-making.

This study aims to bridge these gaps by investigating whether routinely collected, structured biomarkers (rather than unstructured text) can serve as a viable input for Medical LLMs (MedLLMs) to predict 1-year all-cause mortality, comparing them against state-of-the-art regression techniques.

2. Methodology

2.1 Dataset and Cohorts

The study utilizes the LURIC (Ludwigshafen Risk and Cardiovascular Health) study, a prospective investigation involving 3,316 patients who underwent coronary angiography in Germany (1997–2000).

• Full Cohort: 3,316 patients (4.2% 1-year mortality).
• Sub-cohort: 2,112 CVD patients with documented follow-up data (4.1% 1-year mortality).
• Target Variable: 1-Year All-Cause Mortality (1YM).

2.2 Input Features (Biomarker Sets)

Instead of discharge notes, the authors constructed five distinct sets of structured biomarkers and comorbidities:

1. Risk-12: 12 biomarkers from the RiskyCAD project (lipidomics/microRNA).
2. Core-20: 20 routinely measured clinical biomarkers.
3. Lit-21: 21 biomarkers derived from recent literature on LLM mortality prediction.
4. Ext-64: 64 biomarkers from an extended literature review on CVD mortality.
5. Comp-94: A comprehensive aggregation of all above sets (94 features).

2.3 Model Architectures

The study evaluates three categories of models:

• Traditional Regression & Boosting: LinearBoost, CatBoost, XGBoost, Support Vector Machines (SVM), and RealMLP (a Tabular Foundation Model).
• Medical LLMs (Prompting): A wide range of open-source MedLLMs (MedGemma-27B, Med42-70B, MedLlama-8B, Meditron-70b, OpenBioLLM-70B) and commercial models (Gemini-3-Flash, Claude-Sonnet-4.5, ChatGPT-5.2).
  • Zero-Shot: Direct prompting with patient data.
  • Few-Shot: Prompting with 3 synthetic examples to guide the model.
• Supervised Finetuning (SFT): Open-source models (4B and 8B variants) were fine-tuned using LoRA (Low-Rank Adaptation) on the specific 1YM classification task.

2.4 Prompting Strategy

The authors developed a unified prompting pipeline to convert tabular data into natural language.

• Template: A fixed header defining the model as a "medical expert."
• Structure: Patient demographics + a list of biomarkers with full names, units, and values (e.g., "LDL cholesterol (EDTA) (mg/dL): value of 118").
• Few-Shot: Includes synthetic patient cases with risk scores and reasoning to demonstrate the desired output format.

2.5 Evaluation Metrics

• AUROC (Area Under the Receiver Operating Characteristic): Primary metric for discriminative ability, chosen to handle class imbalance.
• ECE (Expected Calibration Error): Measures the reliability of probability estimates.
• Calibration Technique: Platt scaling was applied post-hoc to correct miscalibration.

3. Key Contributions

1. Data Source Innovation: Demonstrated that structured, routinely collected biomarkers are a sufficient and cost-effective input for MedLLMs, removing the dependency on expensive/unavailable discharge summaries.
2. Unified Prompting Framework: Proposed a standardized method to translate tabular patient records into zero- and few-shot prompts suitable for both commercial and open-source LLMs.
3. Comprehensive Benchmarking: Provided the most extensive comparison to date between classic regression/boosting techniques, Tabular Foundation Models, and a wide array of MedLLMs (both open and commercial) on CVD mortality.
4. Calibration Analysis: Identified systematic miscalibration in MedLLMs and demonstrated that Platt scaling can reduce Expected Calibration Error (ECE) by 60–90%.
5. Finetuning Efficacy: Showed that smaller, fine-tuned models (8B parameters) can match or surpass larger (70B) models and commercial counterparts.

4. Key Results

4.1 Predictive Performance (AUROC)

• Top Performers: The best-performing CatBoost (boosting) and the commercial Gemini-3-Flash (via few-shot prompting) both achieved an AUROC of ~85% (0.849) on the CVD sub-cohort using the Ext-64 marker set.
• MedLLM Performance:
  • Large open-source models (70B) with optimized few-shot prompting achieved up to 82% AUROC (e.g., Med42-70B).
  • This performance is competitive with state-of-the-art medical literature baselines like CoroPredict (84% AUROC), SMART (72.9%), and SCORE2 (66.8%).
• Small Models via Finetuning: Smaller models (8B) fine-tuned with LoRA matched or exceeded their larger counterparts. For instance, Meditron3-8B achieved 82.6% AUROC on the full cohort, outperforming many 70B models and commercial APIs.

4.2 Calibration and Stratification

• Miscalibration: Unadjusted MedLLMs exhibited systematic over-prediction of mortality risk (ECE ranging from 0.05 to 0.10).
• Correction: Applying Platt scaling significantly improved calibration, reducing ECE by 60–90%.
• Commercial vs. Open: Commercial models (e.g., Gemini-3-Flash) showed inherently better calibration (lower ECE) than open-source models before post-hoc correction.

4.3 Comparison with Prior Work

The study highlights a significant leap forward compared to earlier research (e.g., [30], [38]), where LLMs lagged significantly behind ANNs and regression models. This study demonstrates that with proper prompting and structured data, LLMs have closed the performance gap.

5. Significance and Conclusion

This paper establishes that Medical LLMs are now viable competitors to traditional regression techniques for CVD mortality risk prediction when utilizing structured biomarker data.

• Clinical Viability: MedLLMs can function as accurate, interpretable tools for risk stratification, either directly on tabular data or in conjunction with textual notes.
• Cost-Effectiveness: The ability to use routine lab data (rather than discharge notes) makes these models scalable across diverse healthcare settings.
• Future Directions: The authors suggest further optimization via Reinforcement Learning from Human Feedback (RLHF) and budget-constrained biomarker selection to enhance clinical deployment.

In summary, the study validates that with the right prompting strategies, few-shot learning, and calibration techniques, Large Language Models can achieve state-of-the-art performance in predicting cardiovascular mortality, offering a powerful alternative or complement to traditional statistical models.