← Latest papers
💻 computer science

Numerical Consistency and Accuracy of Large Language Models in Risk Communication for Cardiovascular Prevention in Older Adults: A Cross-Sectional Multi-Model Paired Study

This cross-sectional multi-model study demonstrates that while current mainstream large language models have achieved near-ceiling accuracy in converting cardiovascular risk metrics for older adults, their performance is primarily differentiated by run-to-run consistency, with DeepSeek showing significantly lower stability than Doubao and Kimi, and inconsistencies arising almost exclusively from benign rounding errors rather than clinically significant errors.

Original authors: Yanhui Jiao, Jiaxin Li, Xiuzhen Xing, Lei Cao, Qing Li

Published 2026-11-03
📖 5 min read🧠 Deep dive

Original authors: Yanhui Jiao, Jiaxin Li, Xiuzhen Xing, Lei Cao, Qing Li

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Heart disease remains the most common cause of illness and death around the world, and its impact falls heaviest on older adults. When doctors try to prevent these problems before they start, they often recommend daily medications like statins. The decision to take such a drug depends on a shared understanding between the patient and the doctor: how much will this help, and what are the risks? This conversation relies on numbers. A doctor might explain that a treatment lowers the chance of a heart attack by a certain percentage, or that for every hundred people who take the drug, a specific number will avoid a bad event. These different ways of stating the same fact are all mathematically equivalent, but they can feel very different to a human mind. Translating one number into another requires a skill called numeracy, which even some medical professionals find difficult. As artificial intelligence tools become more common in healthcare, many people now turn to them to get these explanations. But if a computer gives a patient a number today and a slightly different number tomorrow for the exact same question, trust in that tool begins to crumble.

A team of researchers set out to test how well three popular artificial intelligence systems handle this specific task. They wanted to see if these models could consistently convert complex medical statistics into clear, simple numbers for older adults. The study focused on a scenario where a patient needs to know the benefits and harms of a heart medication. The researchers created eighty different, realistic medical stories involving older patients with various risk factors. In each story, the AI had to calculate four specific values: how much the treatment reduced the risk of a heart event, how much it increased the risk of side effects, how many people would need to take the drug for one person to benefit, and how many would need to take it for one person to be harmed. To ensure a fair test, the researchers asked each of the three AI models to answer every single question three times in a row, and they did this in both English and Chinese. This resulted in thousands of responses that were checked against a locked, perfect answer key created by human experts.

The results revealed a surprising truth about how these systems work. When it came to getting the math right, the models were nearly perfect. They almost always calculated the correct numbers, suggesting that for this type of structured task, their ability to compute is already at the highest possible level. However, the real story was about consistency. When the same question was asked three times to the same model, the answers were not always identical. One of the models, DeepSeek, gave a different set of numbers in more than ten percent of the cases, even though it was set to a mode that should produce the exact same result every time. The other two models, Doubao and Kimi, were much more stable, agreeing with themselves nearly every time. This inconsistency happened even when the models were not trying to be creative; they were simply following instructions to calculate numbers.

When the researchers looked closely at the times when the models did give different answers, they found that the differences were almost always tiny and harmless. In almost every case, the disagreement was just a matter of rounding. For example, if the calculation showed that a patient needed to take the drug twenty-three times to help one person, one run might round that up to twenty-four, while another might round it down to twenty-three. These small shifts did not change the medical advice or the conclusion about whether the drug was worth taking. The models were essentially saying the same thing, just with a slightly different number on the label. Only a tiny fraction of the responses, less than two-tenths of one percent, were truly problematic. These rare instances involved the model giving a completely different value, failing to answer, or changing its mind about whether the answer was correct.

The study also checked if the language used mattered. The models performed just as well in Chinese as they did in English, with no significant difference in accuracy between the two languages. This suggests that the ability to handle these numbers is not tied to a specific language. However, the researchers noted that the model that struggled with consistency in English also struggled in Chinese, indicating that the issue was with the specific software engine rather than the language itself. The findings challenge a common belief that setting an AI to a "zero temperature" mode guarantees it will always give the same answer. In this study, even with those settings, the service provided different numbers at times, likely due to how the system is built and run behind the scenes.

Ultimately, the research shows that while artificial intelligence is excellent at calculating medical statistics, its reliability depends on how consistently it repeats those calculations. For a patient asking a life-or-death question, hearing a different number each time can feel unreliable, even if the numbers are all technically correct. The authors suggest that when evaluating medical AI, experts should look beyond simple accuracy and pay close attention to this stability. They propose a new way to measure reliability by counting how often a model gives answers that are not just different, but dangerously different. In the case of these heart disease scenarios, the models were safe, but the study highlights that for these tools to be trusted in real clinics, they must not only know the right answer but also be able to say it the same way every time.

Drowning in papers in your field?

Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.

Try Digest →