A Reproducible Decomposed Large Language Model Pipeline for Structured Abstraction of Multimodal Neuroimaging Reports: Development and Independent Validation in 1,669 Emergency Department Encounters for Dizziness and Vertigo
This study demonstrates that a decomposed, frozen large language model pipeline can reliably and accurately abstract structured variables from multimodal neuroimaging reports across 1,669 emergency department encounters for dizziness and vertigo, achieving high reproducibility and accuracy comparable to human reviewers to enable scalable, reusable research datasets.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
Hospitals generate a vast ocean of written records every day, from emergency room notes to detailed descriptions of brain scans. For doctors, these narratives are essential for making sense of a patient's condition, but for researchers trying to study large groups of people, they are a difficult barrier. To understand patterns in disease or the effectiveness of treatments, scientists need to turn these free-flowing stories into neat lists of facts, a process called data abstraction. Doing this by hand is slow, expensive, and prone to human error, which often limits studies to small groups of patients. In recent years, powerful computer programs known as large language models have emerged, capable of reading and understanding human language with remarkable skill. These tools offer a potential way to automate the extraction of medical facts from text, but for them to be trusted in serious research, they must prove they are consistent, accurate, and reliable enough to handle the complexity of real-world medical records.
A team of researchers at Staten Island University Hospital set out to test whether such a computer system could reliably translate the complex, multi-part imaging reports of patients suffering from dizziness into structured data. Dizziness and vertigo are among the most common reasons people visit the emergency department, yet they can sometimes signal a serious stroke in the back of the brain. To diagnose this, doctors often order a series of different scans, including CT scans of the head, CT angiography to look at blood vessels, CT perfusion to measure blood flow, and MRI scans. Each of these tests generates its own separate written report, filled with specialized medical terms. The researchers built a specialized pipeline, a sequence of automated steps, to read these reports and pull out twenty-two specific pieces of information, such as whether a stroke was present or how narrow a blood vessel was. They did not rely on a single computer model to do all the work at once. Instead, they broke the task down into smaller, focused jobs, assigning each specific type of scan to a dedicated step in the process, much like a factory assembly line where each station handles one specific part of the product.
The team tested this system on 1,669 emergency department encounters involving patients with dizziness or vertigo. Before trusting the machine, they had to ensure it was reproducible, meaning that if they ran the same reports through the system again, it would produce the exact same results. They ran the system three times in a single session and again nine weeks later, finding that the computer's output was nearly identical every time, with almost perfect agreement on the key variables. They also tested the system using three different, top-tier computer models from different companies to see if the results depended on which specific software was used. The different models agreed with each other at a very high level, suggesting the system was robust and not just a fluke of one particular program. To verify the accuracy of the machine's work, the researchers compared its output against a gold standard created by human experts. Two neurology residents independently read a subset of the reports, and when they disagreed, a third expert resolved the difference. The computer system's ability to match these human experts was comparable to the agreement between the two human readers themselves.
The results showed that the decomposed pipeline was highly accurate, correctly identifying the critical findings in the vast majority of cases. When the researchers looked at the most important variables, such as the presence of an acute stroke or the severity of blockages in major arteries, the computer matched the human experts' conclusions with an accuracy rate between 97 and 99 percent. This high level of performance held true even when the system was tested on cases that were specifically chosen because they contained rare or difficult findings, which are usually the hardest for any system to handle. The study also highlighted that the system was stable over time; running the same data through the system months later yielded the same results, a crucial requirement for any tool used in long-term medical research. The researchers noted that while the system was excellent at extracting facts that were clearly stated in the reports, it was less perfect at interpreting more ambiguous descriptions, which is a known limitation of any tool that reads text rather than looking at the actual images.
This work demonstrates that it is possible to build a reliable, automated system for turning complex medical narratives into structured data at a scale that was previously impossible. By breaking the task into smaller, manageable pieces and using multiple models to cross-check the work, the researchers created a pipeline that is both reproducible and accurate. The study does not claim that computers have replaced the need for human doctors or that the system is perfect for every possible medical question. Instead, it provides a practical path forward for researchers who need to analyze large datasets to understand disease patterns. The quantified error rates identified in the study give future researchers a clear picture of the system's limitations, allowing them to use the data with confidence while knowing exactly where to apply additional human review. Ultimately, this approach offers a way to unlock the value hidden in millions of medical reports, turning unstructured text into a resource that can drive large-scale discoveries in neurology and emergency medicine.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.