Diagnostic Performance of Agentic AI for Rare Disease Diagnosis: A Systematic Review, Meta-analysis, Workflow Development, and Benchmark-based Validation

For millions of people around the world, a diagnosis is not a starting point but a destination that remains out of reach. Rare diseases, by their very nature, are elusive. They present with symptoms that are often vague, overlapping, or unique to a single individual, and the medical knowledge required to connect these dots is scattered across specialized fields that few doctors encounter in their daily practice. This leads to a frustrating journey known as the "diagnostic odyssey," where patients and families spend years searching for answers that might be hidden in plain sight within vast amounts of medical data. In recent years, artificial intelligence has offered a new hope for solving these puzzles. Specifically, a new generation of intelligent systems has emerged that does not just read text but actively thinks through problems. These systems, often called "agent" systems, are designed to mimic the way a human specialist works: they break a complex problem into smaller steps, look up information in external databases, and refine their guesses as they gather more evidence, rather than simply guessing based on a single prompt.

Researchers from Zhejiang Chinese Medical University and its affiliated hospital set out to understand how well these advanced thinking machines perform when faced with the specific challenge of rare diseases. They did not build a new machine themselves; instead, they acted as investigators, gathering and analyzing the results from seven different studies that had already been published or shared as preprints between 2025 and 2026. These studies tested various versions of these intelligent agents on thousands of rare disease cases, asking a simple but critical question: how often does the system correctly identify the right disease as its very first guess? By combining the data from these separate investigations, the researchers created a large-scale picture of the current state of the technology. They found that across more than 33,000 cases, these intelligent systems got the correct diagnosis right on the first try about half the time. While this is a significant improvement over older methods that relied on single, static models, the results were far from perfect. The performance varied wildly depending on the specific system used and the type of data it was tested on, with some setups succeeding nearly 80% of the time and others struggling to reach 30%. This inconsistency suggests that while the technology is promising, it is not yet a uniform solution.

To understand why these systems succeed or fail, the team looked closely at the internal "workflows" or step-by-step processes the different agents used. They discovered that the most successful systems shared a common set of behaviors. Almost every effective agent used a strategy of breaking the diagnostic task down into smaller, manageable steps and then reasoning through those steps to refine its initial ideas. They also frequently reached out to external medical knowledge bases, such as databases of genetic information or rare disease registries, to verify their hypotheses. The researchers took these common, successful patterns and built a simplified, lightweight version of this workflow. They then tested this new workflow against a standard set of 50 rare disease cases, pitting the "thinking" agent against the same underlying computer model running in a "standalone" mode, where it had to guess without the help of the step-by-step process or external lookups.

The results of this head-to-head comparison revealed a nuanced story. When the model operated alone, it correctly identified the top diagnosis in only 5 out of the 50 cases. When the same model was guided by the structured workflow, it improved its performance, getting the right answer first in 11 out of the 50 cases. While this represents a clear improvement, the difference was not statistically large enough to be considered a definitive victory in this small sample size. However, the researchers noticed something even more encouraging when they looked at the top five guesses rather than just the top one. The workflow-assisted model placed the correct diagnosis within its top five choices 50% of the time, a substantial jump from the baseline. This suggests that while the system might not always land on the perfect answer immediately, the structured thinking process helps it keep the correct disease in the running, narrowing down the field of possibilities significantly.

The study concludes that these intelligent agents have demonstrated a genuine ability to assist in diagnosing rare diseases, but they are not yet a finished product. The wide variation in performance across different systems indicates that the specific design of the workflow and the quality of the data it accesses matter deeply. The researchers emphasize that their findings are a proof of concept rather than a final solution. The workflow they constructed serves as a reproducible method for improving how these systems think, but it requires further testing on a much larger scale and in real-world clinical settings to prove its reliability. Ultimately, the goal is not to replace human doctors with machines, but to create a collaborative partnership where the machine's ability to process vast amounts of information and follow a rigorous reasoning process supports the clinician's judgment, potentially shortening the long and difficult journey for patients waiting for answers.