Comparison of performance of an LLM Council of LFM2.5-1.2B, Qwen3-4B-2507 and Phi-4-Mini- reasoning and Meta LLaMa 3.1 8B against MedGemma 4B on extraction and classification of free-text radiology reports
Although the study hypothesized that the domain-specialized MedGemma 4B would outperform a multi-model LLM Council, results showed that MedGemma achieved higher overall accuracy than the Council framework, while the single general-purpose model Qwen3-4B-2507 independently surpassed both, suggesting that strong general reasoning can outperform both consensus-based and specialized medical approaches in clinical radiology report analysis.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the quiet hum of a hospital radiology department, machines capture images of the human body, but the true story of a patient's health often lies hidden within the free-flowing text of the doctor's report. For years, researchers have hoped that artificial intelligence could read these reports as fluently as a human specialist, extracting critical details like tumor sizes or bone fractures to speed up care. Two main paths have emerged to solve this puzzle. One path involves training a computer specifically on medical textbooks and patient records, hoping to build a digital specialist who knows the language of disease by heart. The other path relies on powerful, general-purpose computers that are not taught medicine but are exceptionally good at understanding language, following instructions, and reasoning through complex problems. The big question facing the field is whether a team of these general thinkers, working together to reach a consensus, can outperform a single, highly trained medical expert.
A team of researchers set out to answer this by pitting a specialized medical artificial intelligence against a unique "council" of general-purpose models. They gathered nearly eight hundred real, anonymized radiology reports from a hospital in New Delhi, covering mammograms, hip and knee scans, and brain magnetic resonance images. Their goal was to see which system could better perform two types of tasks: pulling specific numbers and codes out of the text, and deciding whether a scan was normal or required further attention. The specialized model they chose was MedGemma, a system built specifically for healthcare. The opposing force was a council consisting of three smaller, general-purpose models that generated their own answers independently. These answers were then reviewed and combined by a fourth model, acting as a chairman, to produce a single final decision. The researchers hypothesized that the specialized medical model would win, given its deep training in the field.
The results, however, told a different story. When the systems were tested on the mammography reports, which required extracting specific scores, the specialized model performed well, but it was not the clear winner. In fact, one of the general-purpose models in the council, acting alone, achieved the highest accuracy of all. This single model correctly identified the necessary details in nearly ninety-two percent of the mammography cases, edging out the specialized medical model. The council as a whole, with its chairman synthesizing the group's thoughts, performed slightly worse than this single strong model. The gap widened significantly when the task shifted to classifying hip, knee, and ankle reports. Here, the specialized medical model correctly identified whether a patient was pre- or post-surgery in about eighty-six percent of cases, while the council struggled, getting it right only about seventy-two percent of the time. The single general-purpose model, however, soared to nearly ninety-three percent accuracy, correctly interpreting the context of the reports far better than either the specialist or the group effort.
The most surprising finding emerged in the brain scans. The specialized model was very good at spotting abnormalities, but it also made more mistakes by flagging normal scans as abnormal when it saw minor issues elsewhere in the text. The single general-purpose model, by contrast, was perfect at identifying normal scans, never making that specific error, while still catching the vast majority of real problems. When the researchers looked at the council's performance, they found that the process of combining the opinions of multiple models often diluted the correct answers. In several instances, the council ignored the correct answer provided by its best member and instead followed the majority, which turned out to be wrong. The study suggests that for these specific tasks, the ability to reason through a narrative and follow instructions is more valuable than having a database of medical facts memorized. The council's complex architecture, which required four times as much computing power and memory as the single model, did not yield better results; it simply cost more to run.
Ultimately, the research indicates that for reading radiology reports, a single, strong general-purpose model that can reason through text may be more effective than a team of models trying to reach a consensus or a system trained exclusively on medical data. The specialized medical model proved robust, but it was outperformed by the generalist's ability to understand context. The experiment also revealed that adding layers of agreement and synthesis can sometimes introduce errors rather than fix them, as the final decision may reflect a compromise rather than the truth. While the specialized model remains a strong tool, the study suggests that the future of reading medical reports may not depend on building bigger, more complex teams of artificial intelligence, but on finding the single model that understands the story the best.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.