Right Diagnoses, Decorative Reasoning:A Perturbation Audit of Medical Chain-of-Thought
This paper introduces a medical perturbation audit framework that reveals a high "Chain-Decoupling Rate" across 14 LLMs, demonstrating that their chain-of-thought rationales often serve as decorative documentation rather than faithful evidence of the actual reasoning driving medical diagnoses.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the high-stakes world of medical care, doctors rely on a specific kind of thinking process to reach a diagnosis. They gather symptoms, weigh evidence, and walk through a logical path to a conclusion. When artificial intelligence systems began to answer medical questions, they adopted a similar method called "chain-of-thought." Instead of jumping straight to an answer, these computer programs generate a step-by-step explanation first, mimicking the way a human doctor might reason. This feature is vital because it allows medical professionals to inspect the logic behind a machine's suggestion, ensuring that a confident answer isn't hiding a dangerous mistake. The hope has been that if the computer writes out its reasoning, that text is a true record of how it reached the decision, acting as a transparent window into its mind.
However, a new study from researchers at Eindhoven University of Technology and the Dana-Farber Cancer Institute challenges this assumption. They asked a fundamental question: does the written explanation actually drive the answer, or is it merely a story the machine tells itself after the fact? To find out, the team treated the reasoning process like a scientific experiment. They took fourteen different medical AI models and subjected them to a rigorous audit. The researchers systematically altered the questions the models were asked—changing a condition from "acute" to "chronic," swapping a patient's age, or flipping a negative statement to a positive one. These changes were designed to be clinically significant, the kind of details a real doctor would notice immediately. If the AI's reasoning were truly faithful, the written explanation should change to reflect the new information, and the final answer should shift if the medical facts demanded it.
What the researchers discovered was a startling disconnect. Across the board, the models behaved as if they were ignoring the changes they were told to make. When the question was altered in a way that should have changed the diagnosis, the written reasoning often remained exactly the same, repeating the original facts as if nothing had happened. In nearly 73 percent of these cases, the model's explanation did not register the edit, and the final answer did not flip, even though the input had been fundamentally changed. It was as if the machine had already decided on an answer and was simply writing a script to match it, regardless of the actual evidence presented. The study tested this by corrupting the reasoning text itself—deleting steps or rearranging sentences—and found that the models' accuracy barely budged. The final answer remained correct not because the reasoning was sound, but because the reasoning was not actually doing the work.
The researchers confirmed that this was not a simple error in reading. They brought in two board-certified clinicians to review the altered questions and the models' responses. The doctors agreed that in 98.5 percent of the cases, the changes made to the questions were clear and significant enough that a faithful reasoning process should have reacted to them. Yet, the models largely failed to do so. In some instances, the models even flipped their answers to incorrect choices when the question was changed, but the written explanation did not reflect this shift, suggesting the reasoning was not the cause of the decision. The study also looked at models that had been specifically trained to reason better, including those from major technology companies, and found the same pattern: the visible chain of thought was largely decorative. It served as a documentation surface, a record of what the model wanted to say, rather than a causal record of how it solved the problem.
This finding suggests that the current reliance on these explanations as proof of safe, logical reasoning may be misplaced. The study does not claim that the models are incapable of giving the right answer, nor does it say the explanations are always wrong. Instead, it reveals that the explanation is often not the engine driving the car. For medical professionals who use these tools to support their decisions, this means they cannot assume that a clear, logical-sounding explanation guarantees that the machine actually used the evidence provided. The research indicates that until the reasoning process is truly tied to the final decision, these AI systems remain a black box where the visible text is a narrative, not a map. The path forward requires building systems where the explanation is not just a story, but the actual mechanism of the answer.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.