When Debate Hurts: Model-Scale-Dependent Effects of Multi-Agent Deliberation in Psychiatric Diagnosis

In the world of mental health care, diagnosing a patient is rarely a simple matter of matching symptoms to a checklist. Even for highly trained human psychiatrists, the process is fraught with uncertainty. Studies have shown that when different doctors look at the same description of a patient, they often disagree on the final diagnosis. Sometimes, the confusion is so deep that even a team of the world's leading experts cannot agree on what is wrong with a person. Because of this, researchers have turned to artificial intelligence, hoping that computer models might offer a more consistent way to identify mental health conditions. One popular idea is to have multiple AI agents, acting like a panel of specialists, debate a case before reaching a conclusion. The theory is that this back-and-forth discussion would catch errors and lead to better answers, much like a human team working through a complex problem together.

A new study from researchers at the Indian Institute of Technology Kharagpur tests this idea in a very practical way, using smaller, open-source computer models that can run on a standard personal computer rather than massive, expensive servers. They set up a system where three different AI "specialists"—one focused on symptoms, one on comparing different possible diagnoses, and one on assessing danger to the patient—would discuss twenty-four different synthetic patient stories. These stories were written to look like real medical case files, covering a range of psychiatric conditions. The researchers wanted to see if this debating process actually helped the AI get the right answer, or if it made things worse. They compared the debating team against a single AI agent that simply gave an answer immediately without talking to anyone else.

The results were surprising and went against the common assumption that more discussion always leads to better results. When the researchers tested the system on two different sizes of computer models—one with one billion parameters and another with three billion—they found that the debating team consistently performed worse than the single agent that just gave a quick answer. This was not a fluke of a single test; the researchers ran the experiment five separate times for each setup to ensure the results were real. In every instance, the AI that debated itself got the diagnosis wrong more often than the one that did not. The gap was significant: at the smaller size, the debating team was correct about 57 percent of the time on average across trials, while the simple, non-debating agent was correct about 73 percent of the time. At the larger size, the debating team improved to about 69 percent on average, but the simple agent still did better, reaching nearly 88 percent accuracy.

The study also looked closely at why this happened, breaking down the process to see which part of the system was causing the trouble. The researchers had assumed that the final step, where a "judge" agent summarized the debate and made the final call, was the main culprit for the poor performance. However, their investigation revealed a more complicated picture. The act of debating itself, even before a judge was involved, already lowered the accuracy compared to the simple approach. Furthermore, the effect of the judge changed depending on the size of the model. On the smaller model, the judge did not make things much worse, but on the larger model, the judge only partially recovered some of the lost accuracy, while its effect on overall accuracy remained inconsistent across the different scales. This suggests that the problem is not just one broken part of the system, but a complex interaction between the size of the computer brain and how it handles group discussion.

Perhaps the most critical finding concerned safety. In mental health, missing a sign that a patient is in danger of harming themselves is a catastrophic failure. The researchers found that the effect of the debate on spotting these dangers flipped depending on the model size. On the smaller model, the debate process actually helped the AI notice a high risk of self-harm that the simple agent had missed. But on the larger model, the opposite happened: the simple agent correctly identified the danger, but the debating team, specifically the judge who summarized the discussion, talked itself out of flagging the risk. This means that a system designed to be safer by having agents argue with each other could actually become less safe, depending entirely on which specific computer model is being used.

The researchers were careful to note that their results do not mean artificial intelligence has already surpassed human doctors. They compared their numbers to published studies of human psychiatrists and found that their AI results were in the same ballpark as human performance on similar tasks, sometimes even higher, but they emphasized that the tests were not perfectly matched. The human studies used different patient stories and different methods for determining the correct answer. The author argues that the most important takeaway is not that AI is better or worse than humans, but that the popular idea of "more debate is better" is not a universal rule. In the specific context of psychiatric diagnosis using these local computer models, the extra step of having agents argue with each other introduced errors that a simple, direct approach avoided. The study serves as a reminder that in complex fields like medicine, adding layers of discussion does not automatically guarantee a better outcome, and sometimes, the most direct path is the most reliable one.