Are Large Multimodal Models Ready for Post-Thrombectomy mTICI Assessment? A Comparison with Physicians
While large multimodal models can achieve physician-level agreement in post-thrombectomy mTICI grading when provided with procedural reports, their performance relies heavily on text input and collapses significantly when assessing images alone, indicating that expert human interpretation remains essential for direct image-based reperfusion evaluation.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Every year, millions of people suffer strokes caused by a blocked artery in the brain. When a large vessel is clogged, the damage can be swift and severe, but doctors have a powerful tool to stop it: a mechanical thrombectomy. In this procedure, a specialist threads a tiny device through the blood vessels to physically pull the clot out. The success of this life-saving operation depends entirely on one thing: whether blood flow is fully restored to the brain. To measure this, doctors use a standard scale that grades how well the vessels have been cleared, ranging from no flow at all to complete restoration. This grading is critical because it determines how patients are treated afterward, how hospitals track their success, and how researchers understand what works best. However, assigning these grades is a human task, and like many human judgments, it can vary from one doctor to another. As artificial intelligence has grown more powerful, a natural question has emerged: can a computer look at the same images and reports a doctor sees and make the same judgment with equal skill?
A team of researchers at the University Hospital Regensburg in Germany set out to answer this question by pitting advanced computer models against human experts. They gathered ninety cases of patients who had undergone the clot-removal procedure for a blockage in the front part of the brain. For each case, they collected the digital angiography images taken before and after the clot was removed, along with the written medical report describing what happened during the surgery. They then asked two different types of artificial intelligence systems to grade the success of the procedure. One system was a general-purpose model known for its broad knowledge, while the other was a model specifically tuned for medical tasks. To ensure a fair test, the researchers also asked four human doctors—two neurologists and two radiologists—to grade the same cases. The doctors and the computers were tested under three different scenarios: looking only at the written report with the final grade hidden, looking only at the angiography images, and looking at both the images and the report together.
The results revealed a sharp divide between what the computers could do with words and what they could do with pictures. When the artificial intelligence models were given the written procedural reports, they performed remarkably well. The general-purpose model, in particular, matched the accuracy of the pooled group of human doctors. It could read the detailed descriptions of the surgery and correctly identify how much blood flow had been restored, achieving a level of agreement with the experts that was virtually indistinguishable. In this text-based setting, the computer was just as reliable as the human specialists. However, the moment the images were removed from the equation and the models were asked to grade the procedure based on the written report alone, they remained strong. But when the tables were turned and the models were forced to look only at the angiography images without any accompanying text, their performance collapsed. The general-purpose model's ability to agree with the human experts dropped significantly, and the medical-tuned model performed even worse, essentially guessing that almost every procedure had been successful regardless of what the images showed.
The human doctors, by contrast, did not suffer from this drop in performance. Whether they were reading the report or looking at the images, their ability to grade the procedure remained consistently high. This suggests that while the artificial intelligence systems are excellent at processing the language doctors use to describe their work, they have not yet learned to truly "see" the complex patterns of blood flow in the images the way a trained specialist does. The study indicates that these models are currently acting more like sophisticated readers who can extract information from a story than as independent observers who can interpret visual data. The researchers found that the models derived almost all of their success from the text; adding the images to the text did not improve their scores, and removing the text left them struggling.
Ultimately, the study concludes that these advanced computer models are ready to assist with the paperwork and data extraction aspects of stroke care, such as automatically pulling success grades from written reports for research or quality checks. They can help standardize how information is recorded. However, they are not yet ready to replace the human eye when it comes to looking at the images themselves. For the direct assessment of whether a blood vessel has been successfully reopened, the visual expertise of a trained doctor remains essential. The technology has reached a point where it can understand the story of the surgery, but it still cannot see the picture.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.