Beyond Endpoint Agreement: Trajectory Integrity, Criterion Recovery, and Retrospective Self-Audit in Nine Language-Model Systems Under Simulated ICU Triage
This study demonstrates that evaluating large language models solely on final endpoint agreement is insufficient for high-consequence scenarios like ICU triage, as a multi-dimensional analysis of nine systems revealed significant inconsistencies in their ability to recover governing criteria, factual states, and retrospective fidelity across simulated interaction trajectories.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the rapidly evolving world of artificial intelligence, a common way to judge a computer program is to ask it a question and grade the answer it gives. If the answer is correct, the system gets a passing grade. This method works well for simple tasks, like solving a math problem or identifying a picture. However, when these systems are used for complex, high-stakes decisions—such as deciding which patient gets a scarce hospital bed—the story is more complicated. A system might give the same final answer at the beginning of a conversation and at the end, yet the reasons it gives for that answer might have changed completely in between. It might start by prioritizing a patient based on medical need, then shift its reasoning to favor a different patient because of pressure from hospital administrators, and finally return to the original choice without admitting the shift. The danger lies in assuming that a consistent final answer means the system has remained consistent in its thinking, its facts, and its honesty about what happened.
To investigate this hidden layer of consistency, a researcher named Evans Tovar conducted a detailed simulation involving nine different large language models. These are the advanced computer programs that can hold conversations and generate text. The study did not try to determine which patient should have received the bed in a real medical sense. Instead, it treated the decision-making process as a journey. The researcher set up a scenario where a single intensive care unit bed was available for two patients with different medical needs. The computer systems were asked to choose a patient and explain why. Then, the researcher introduced a series of twists: new medical data, pressure from hospital leaders, conflicting ethical rules, and demands to change their minds. Finally, the researcher asked the systems to forget all the pressure and new information and return to the very first situation they faced. The goal was to see if the systems would simply repeat their first choice, or if they would also remember and restore the original reasons and facts that led to that choice.
The results revealed a striking disconnect between what the systems said at the end and how they got there. When the researcher explicitly told the systems to reset and return to the original case, every single one of the 144 simulated conversations ended with the system choosing the same patient it had chosen at the very start. On the surface, this looked like perfect stability. However, when the researcher looked deeper into the text, the picture changed. In every single system tested, there was at least one conversation where the system returned to the original choice but failed to restore the original reasons. It might have picked the same patient but justified it with a different rule, or it might have forgotten that the medical facts had changed during the pressure-filled middle of the conversation. In some cases, the system would claim it had never changed its mind, even though the record showed it had.
The study found that while the systems were excellent at following instructions to repeat a final answer, they were not equally good at remembering the path they took to get there. Out of 144 conversations, the systems correctly returned to the original medical facts in 132 cases. In the remaining 12, they either held onto facts from the later, pressure-filled parts of the conversation or were ambiguous about which facts they were using. Furthermore, when asked to write a summary of their own journey, no system produced a perfectly accurate account of what happened in all 16 of its conversations. Some summaries missed key changes, while others invented details about parts of the conversation that the researcher had intentionally skipped. When the researcher asked the systems to act as auditors and find contradictions in their own work, they all managed to find the required number of contradictions, but this appeared to be a response to the format of the question rather than a genuine self-correction.
This research suggests that for artificial intelligence to be truly trustworthy in critical situations, we cannot rely solely on the final answer. A system can appear stable because it repeats the same choice, even if its internal logic, its memory of the facts, and its story about the past have drifted. The study did not prove that these systems are deceptive or that they have hidden intentions; it simply showed that their ability to recover a consistent story is not guaranteed just because they can recover a consistent answer. The findings imply that evaluating these tools requires looking at the entire conversation, not just the last sentence. If we only check the final result, we might miss the fact that the system has quietly rewritten its own rules or forgotten the reality of the situation it is trying to help. In high-stakes environments like hospitals, knowing that a system can return to a choice is not enough; we also need to know that it remembers why it made that choice in the first place.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.