A Configurable LLM System for Automated Data Extraction in Meta-Analysis Using a Minimal Atomic Unit Framework

In the world of modern medicine, doctors and policymakers rely on a rigorous process called a systematic review to decide which treatments work best. Imagine trying to understand the effectiveness of a new drug by reading just one study; the picture would be incomplete and potentially misleading. Instead, experts gather every relevant study on a topic, extract the specific numbers from each one, and combine them to find a clear answer. This process, known as a meta-analysis, is the gold standard for evidence-based decisions. However, the work is incredibly labor-intensive. Researchers must manually read through dozens of dense medical reports, hunting for tiny details like how many patients improved, what their starting health was, and exactly when the measurements were taken. One small mistake in copying a number or misreading a table can skew the entire final result, leading to incorrect conclusions about patient care.

For years, scientists have hoped that artificial intelligence could handle this tedious data-hunting. Large language models, the same type of technology that powers advanced chatbots, are excellent at understanding human language. Yet, when asked to pull specific numbers from complex medical reports, these models often stumble. They might confuse a result from one group of patients with another, miss a crucial detail buried in a paragraph, or worse, invent a plausible-sounding number that never appeared in the original text. This tendency to "hallucinate" data is dangerous in medicine, where accuracy is non-negotiable. A recent study by researchers at Xi'an Jiaotong-Liverpool University and their colleagues set out to solve this problem by teaching the artificial intelligence a new way to think about the task, not just as a general reader, but as a precise data extractor.

The team developed a system that breaks down the chaotic information in a medical report into tiny, manageable pieces they call "minimal atomic units." Instead of asking the computer to read a whole page and guess what numbers belong where, the system forces the computer to focus on one specific combination of facts at a time: a specific group of patients, a specific time when they were checked, and a specific health outcome being measured. Think of it as the difference between asking someone to describe the contents of a messy room in a single sentence versus asking them to list the items on the floor, then the items on the table, and then the items on the shelves, one category at a time. By forcing the artificial intelligence to tackle the extraction in these small, structured steps, the researchers ensured that every number the computer pulled out was tied directly to the specific sentence or table where it was found.

To test if this method worked, the researchers first tried it on a small set of three medical reports. They compared the new, step-by-step approach against a more traditional method where the computer was asked to do everything at once without the structured breakdown. The results were stark. When the computer used the new method, it correctly identified nearly every single piece of data it was supposed to find, and it never made up a number that wasn't there. In contrast, the traditional approach missed more than three-quarters of the important details and struggled to keep the different groups of patients straight. The researchers noted, however, that this first test was just a preliminary look, using the same reports to both teach the system and test it, so it was not a final proof of success.

To get a true measure of performance, the team then applied their finalized system to a much larger and independent set of 88 medical reports covering five very different areas of medicine: breast cancer surgery, schizophrenia treatment, heart disease prevention, type 2 diabetes management, and orthopedic rehabilitation. These reports contained complex tables, multiple groups of patients, and various ways of measuring success. The system processed these documents and produced a structured list of data points, with every single number linked back to its original source text so a human could verify it instantly. The results were remarkably strong. Out of nearly 26,000 individual data points the system was asked to extract, it got the right answer almost every time. It missed very few pieces of information, and when it did make a mistake, it was almost always a case of leaving a blank space rather than inventing a false one. The rate of making up data was vanishingly small, occurring in only a tiny fraction of a percent of cases.

The study also looked at how the system performed across the different medical fields. It worked exceptionally well for breast cancer and heart disease studies, where it found almost every required number. It performed slightly less perfectly for diabetes studies, where it occasionally missed repeated numbers that appeared in multiple rows of a table, but even then, it remained highly accurate. The researchers found that the most common errors were not wild guesses, but rather omissions of information that was repeated in the text or confusion between similar-sounding medical terms. Crucially, the system never produced a result without attaching the specific text from the original report that supported it, ensuring that a human reviewer could always check the work.

Despite these impressive results, the authors are careful not to claim that the problem is completely solved. They emphasize that their system is a powerful tool to assist human experts, not a replacement for them. The comparison between the new method and the old one was based on a small set of reports used during the development phase, so while the results strongly suggest the new method is superior, it has not yet been proven in a completely independent, head-to-head trial against other systems. Furthermore, the system still requires human oversight to catch the rare instances where it might miss a detail or struggle with unusual formatting. The researchers conclude that this approach offers a promising path forward for evidence synthesis, turning a slow, error-prone manual process into a fast, traceable, and highly accurate workflow, provided that human experts remain in the loop to verify the final output.