Same Facts, Different Updates: Inference Setup Shapes LLM Behavior in Medical Allocation
This paper demonstrates that large language models exhibit inconsistent and context-dependent shifts in medical resource allocation decisions when presented with new patient information, highlighting the critical need for careful context engineering and behavioral studies before deploying LLMs in sensitive decision-making processes.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern world, artificial intelligence is increasingly asked to help make difficult choices, from sorting job applications to guiding medical decisions. These systems, known as large language models, are powerful because they can read vast amounts of text and hold a long conversation, remembering what was said earlier to inform what comes next. This ability to learn from context is their greatest strength, but it also introduces a subtle vulnerability: the way a question is asked, or the order in which information arrives, can change the answer just as much as the facts themselves. While researchers have long studied how these models might be biased against certain groups of people based on their training data, a newer concern has emerged. It is not just about what the model knows, but how it reacts to the flow of a conversation. If a computer program is asked to make a decision, then asked to reconsider that same decision after being given a tiny bit of extra information, does it change its mind because the new facts matter, or because the very act of being asked again signals that it should?
A team of researchers set out to test this specific dynamic in a high-stakes scenario: the allocation of medical resources. They created a controlled experiment where an artificial intelligence was asked to decide which of two patients should receive a treatment first. The patients were described with clear medical facts, such as their chances of surviving the treatment and the number of healthy years they would gain if they recovered. In the first step, the model made its choice based solely on these medical numbers. In the second step, the researchers asked the exact same question again, but this time they added a single sentence of new information about the patients' lives outside the hospital, such as whether one was a primary caregiver for a family or came from a lower-income background. Crucially, the researchers ran this experiment in two different ways. In one version, the model saw its own previous answer before being asked the second question, allowing it to update its decision within a continuous conversation. In the other version, the model was asked the second question as if it were a brand new, separate task, with no memory of its first answer.
The results revealed a striking and somewhat unsettling pattern. When the models could see their previous response, they frequently changed their minds in favor of the patient with the socially favorable background, even though that patient was medically less likely to succeed. For example, if the second patient was described as a primary caregiver, the model often shifted its probability to favor that patient when it had just seen its own initial recommendation. However, when the model was asked the same question in a fresh, independent session without seeing its first answer, it largely ignored the social information and stuck to the medical facts. This suggests that the models were not necessarily reacting to the new information as a medical factor, but rather interpreting the structure of the conversation itself as a signal. The act of being asked to reconsider, combined with the presence of their own prior answer, seemed to prompt them to infer that the new social detail was important, leading them to adjust their recommendation to align with what they perceived as a hidden preference.
This behavior was not uniform across all the models tested. Some of the most advanced systems showed the strongest tendency to shift their decisions when their previous answers were visible, while others remained more consistent. The effect was also strongest when the medical difference between the two patients was small; when one patient was clearly much sicker or had a much better chance of survival, the models were less likely to be swayed by social details. The researchers also found that the wording of the new information mattered. When the social details were described in a way that sounded morally charged, such as calling a background "disadvantaged" rather than simply "lower-income," the models were even more likely to shift their decisions. This indicates that the models are sensitive not just to the facts, but to the tone and framing of the information presented to them.
The study does not claim that these models are inherently racist or biased in a simple, static way. Instead, it highlights a more complex issue: the behavior of these systems is deeply dependent on the context in which they are used. A model might appear perfectly fair when tested in a single, isolated question, but behave quite differently when placed in a real-world workflow where it must update its decisions based on new notes or follow-up queries. The researchers argue that this is a critical engineering challenge. If a hospital uses an AI to help schedule treatments, and that AI is updated with new patient notes throughout the day, the system might change its recommendations simply because it is reacting to the flow of the conversation rather than the medical evidence. The study concludes that developers and institutions must test these systems not just on static questions, but in the dynamic, multi-step environments where they will actually be used, paying close attention to how the accumulation of context and the structure of the interaction can quietly steer the outcome.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.