💻 computer science

Agentic clinical reasoning over longitudinal myeloma records: a retrospective evaluation against expert consensus

This retrospective study demonstrates that an agentic clinical reasoning system outperforms standard RAG and full-context approaches in synthesizing longitudinal multiple myeloma records to match expert consensus, particularly on complex cases, though its higher severity of residual errors necessitates prospective validation before clinical deployment.

Original authors: Johannes Moll, Jannik Lübberstedt, Christoph Nuernbergk, Jacob Stroh, Luisa Mertens, Anna Purcarea, Christopher Zirn, Zeineb Benchaaben, Fabian Drexel, Hartmut Häntze, Anirudh Narayanan, Friedrich Put

Published 2026-04-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Johannes Moll, Jannik Lübberstedt, Christoph Nuernbergk, Jacob Stroh, Luisa Mertens, Anna Purcarea, Christopher Zirn, Zeineb Benchaaben, Fabian Drexel, Hartmut Häntze, Anirudh Narayanan, Friedrich Puttkammer, Andrei Zhukov, Jacqueline Lammert, Sebastian Ziegelmayer, Markus Graf, Marion Högner, Marcus Makowski, Florian Bassermann, Lisa C. Adams, Jiazhen Pan, Daniel Rueckert, Krischan Braitsch, Keno K. Bressem

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Overwhelmed Doctor" vs. The "Smart Assistant"

Imagine a doctor treating a patient with a complex, long-term illness like Multiple Myeloma. This isn't a one-time visit; it's a journey that can last decades. Over that time, the patient accumulates a massive "backlog" of paperwork: hundreds of lab reports, imaging scans, discharge summaries, and handwritten notes.

The Problem:
To make a single decision today (like "Is this patient ready for a new therapy?"), the doctor has to read through years of history, connect dots between a lab result from 2018 and a side effect from 2022, and ignore outdated info. It's like trying to find a specific needle in a haystack that keeps growing every day. Doctors are already drowning in paperwork, and this task is exhausting and prone to human error.

The Question:
Can Artificial Intelligence (AI) act as a super-assistant that reads all these years of records instantly and gives the doctor a reliable answer?

The Experiment: Four Different "AI Assistants"

The researchers built four different types of AI systems to test this. They gave them the same 469 complex questions about real patients and asked them to answer based only on the patient's records. They then compared the AI answers to the "Gold Standard" answer provided by a panel of four expert human doctors.

Here are the four "assistants":

The "Single-Pass" Reader (Simple RAG): Imagine a librarian who hears a question, runs to the shelves, grabs the first few books that look relevant, and immediately writes an answer. It's fast, but it might miss the most important book hidden on a different shelf.
The "Looping" Reader (Iterative RAG): This librarian is smarter. If the first books don't have the answer, they go back, ask a follow-up question, and grab more books. They keep looping until they feel they have enough info.
The "Mega-Reader" (Full Context): This assistant tries to read every single page of the patient's entire medical history at once, stuffing it all into its brain before answering. It's like trying to drink from a firehose; it has all the info, but it might get overwhelmed and forget the middle parts.
The "Agentic" Assistant (The Star of the Show): This is a detective. Instead of just reading or looping, it plans.
- It breaks the big question down into small steps.
- It decides which specific tools to use (e.g., "First, check the lab results for kidney function. Then, look for the specific drug name in the 2023 notes.").
- It keeps a "sticky note" (memory) of what it found and what it still needs.
- It only stops when it has gathered all the specific evidence required to solve the puzzle.

The Results: Who Won?

The researchers found that the "Mega-Reader" and the "Looping Reader" hit a performance ceiling. They both got about 75% of the answers right. They were good, but they couldn't get any better, no matter how much data they were given.

The "Agentic" Assistant was the only one to break through that ceiling, reaching 79.6% accuracy.

Where did the magic happen?
The AI didn't just get slightly better; it got much better on the hardest tasks.

Simple Questions: (e.g., "Did the patient take this drug last week?") All the AIs did about the same.
Complex Questions: (e.g., "Based on all the labs, scans, and past treatments from the last 5 years, is this patient eligible for a specific new therapy?") The Agentic Assistant pulled ahead significantly. It was 9.4% more accurate than the others on these tough cases.
The Longest Records: The advantage grew even larger for patients with the longest medical histories (the "haystacks" with the most needles).

The Catch: The "Severity" of Mistakes

The paper notes a crucial, slightly scary detail. While the AI made mistakes at a rate similar to human doctors (about 12% vs. 13.6% disagreement), the type of mistakes was different.

Human Doctors: When they disagreed, it was often a minor difference (e.g., "I think the date is Tuesday" vs. "I think it's Wednesday").
The AI: When it was wrong, it was more likely to be a clinically significant error (e.g., missing a critical rule that would make a treatment dangerous).

Think of it this way: If two humans argue about a recipe, they might argue about whether to add a pinch of salt. If the AI argues, it might accidentally tell you to add a cup of salt. The AI is generally accurate, but its "bad days" are more dangerous than a human's "bad days."

The Conclusion

The study concludes that AI can be a powerful tool for doctors, but it needs to be the "Agentic" kind—the one that plans and uses tools step-by-step, rather than just reading everything at once.

However, because the AI's mistakes can be more severe than human disagreements, the researchers say we cannot just hand this system to doctors tomorrow. It needs more testing in real-world clinics to ensure it doesn't accidentally harm patients before it becomes a standard part of medical care.

In short: The "Smart Detective" AI is the best at solving complex medical puzzles, but until we are sure it won't make dangerous mistakes, it should remain a helper, not the boss.

1. Problem Statement

Multiple Myeloma (MM) management involves sequential therapy lines over years or decades, generating vast amounts of heterogeneous clinical data (lab results, pathology, imaging, free-text notes).

The Challenge: Clinical decisions require synthesizing information across dozens to hundreds of documents and timepoints. No single document contains the full picture.
The Gap: While Large Language Models (LLMs) have been applied to clinical tasks, their ability to perform longitudinal reasoning (synthesizing evidence over time) against an expert-annotated standard has not been rigorously established.
Limitations of Current Approaches:
- Single-pass RAG: Often fails to retrieve all necessary evidence or lacks iterative refinement.
- Full-Context Input: Feeding all records into the model often leads to the "lost in the middle" phenomenon, where relevant information is ignored due to context window limits or attention dilution.
- Agentic Systems: Previous evaluations focused on simulated patients or structured databases, not real-world, unstructured, longitudinal EHRs with contradictory data.

2. Methodology

Study Design & Data

Cohort: 811 patients with Multiple Myeloma treated at TUM University Hospital (2001–2026).
Data Volume: 44,962 clinical documents and 1.33 million laboratory values per patient (median 55 docs/patient).
External Validation: 716 patients from the MIMIC-IV critical care database (English), used without adaptation to test transferability.
Question Bank: 48 clinical question templates stratified into three complexity levels:
- Level 1: Single-record lookup.
- Level 2: Temporal reasoning across multiple sources.
- Level 3: Criteria-based synthesis (e.g., eligibility for CAR-T) across document types and timepoints.
Reference Standard: Independent double annotation by four oncologists, with a senior hematologist adjudicating disagreements. 469 evaluable patient-question pairs were retained for the primary analysis.

System Architectures Compared

All systems used the same locally deployed 120B-parameter open-weight LLM (gpt-oss-120b) to ensure fair comparison:

Simple RAG: Single-pass dense retrieval without query rewriting.
Iterative RAG: Includes subquery rewriting, hybrid retrieval (BM25 + dense), cross-encoder reranking, and a sufficiency loop.
Full Context: All documents concatenated in reverse chronological order until the context window is filled.
Agentic System (Proposed): A multi-step reasoning pipeline with four distinct phases:
- Skill Selection: Retrieves specific clinical reasoning protocols from an indexed library based on the query.
- Planning: Constructs an ordered tool-use plan with explicit stopping conditions.
- Execution: Iteratively executes tools (report retrieval, lab value retrieval, deterministic calculators) and updates a structured memory state.
- Synthesis: Generates a final answer with inline citations, adhering to a schema-defined format.

Evaluation Metrics

Primary: Concordance with expert consensus (binary for single-choice, F1-score for list-type answers).
Secondary: Clinically significant error rate, citation sufficiency, and performance stratified by question complexity and record length.

3. Key Contributions

First Agentic Evaluation on Real Longitudinal EHRs: This is the first study to evaluate agentic reasoning against an expert-adjudicated standard on real-world, multi-decade MM records, moving beyond simulated or structured data.
Demonstration of a Performance Ceiling: The study establishes that Iterative RAG and Full Context converge at a specific performance ceiling (~75%), suggesting that simply adding more context or basic iteration is insufficient for complex synthesis.
Agentic Superiority in Complexity: The agentic system is the only approach shown to exceed this ceiling, with gains scaling directly with task complexity and record length.
Error Profile Analysis: The study provides a granular breakdown of errors, revealing that while the system's total error rate matches human disagreement, the severity of system errors is higher (more clinically significant), highlighting a critical safety gap.

4. Key Results

Overall Performance

Agentic System: 79.6% concordance (95% CI: 76.4–82.8).
Full Context: 75.8% concordance.
Iterative RAG: 75.4% concordance.
Simple RAG: 71.5% concordance.
Significance: The agentic system significantly outperformed Full Context (+3.8 pp, $p=0.006$ ) and Iterative RAG (+4.2 pp, $p=0.007$ ).

Performance by Complexity

The advantage of the agentic system grew monotonically with difficulty:

Level 1 (Simple): +1.0 pp advantage (not significant).
Level 2 (Medium): +3.9 pp advantage.
Level 3 (Complex): +9.4 pp advantage ( $p=0.032$ $p = 0.032$ ).
- At Level 3, non-agentic systems ranged from 48.9% to 55.7%, while the agentic system reached 65.1%.

Performance by Record Length

For patients with shorter records (bottom 90%), all systems performed similarly.
For the top decile of record length (>541k characters, ~245k tokens), the agentic system achieved 75.3% concordance, significantly outperforming Iterative RAG (63.9%) and Full Context (61.8%).
This indicates the agentic architecture is uniquely capable of navigating extremely long, complex histories where other methods degrade.

Error Analysis & Safety

Error Rates: The system's total error rate (12.2%) was comparable to expert disagreement (13.6%).
Severity Distribution (Critical Finding):
- System Errors: 57.8% were classified as clinically significant.
- Expert Disagreements: Only 18.8% were clinically significant.
- Interpretation: While the system makes as many mistakes as humans, its mistakes are more likely to lead to incorrect clinical decisions.
Failure Modes:
- Simple tasks failed due to incomplete retrieval.
- Complex tasks failed due to integration errors (retrieving evidence but failing to synthesize it correctly), consistent with the "lost in the middle" phenomenon in long-context models.

External Validation

On the MIMIC-IV cohort (English, different institution), the system ranking was preserved (Agentic > Iterative RAG > Full Context), demonstrating robustness across languages and documentation structures without fine-tuning.

5. Significance and Conclusion

Clinical Impact: The agentic approach offers a viable path for AI-assisted clinical decision support in complex, longitudinal diseases like Multiple Myeloma, specifically for patients with the most complex histories where manual chart review is most burdensome.
Architectural Insight: The study proves that structured reasoning (planning, iterative retrieval, deterministic tools) is superior to passive context ingestion for complex synthesis tasks. The "skill library" was identified as the primary driver of performance gains.
Safety Warning: Despite high concordance, the higher severity of system errors compared to human disagreements indicates that prospective evaluation in routine care is mandatory before deployment. The system is not yet ready to replace human oversight but shows promise as a decision-support tool.
Future Work: The authors suggest that larger models with better long-context reasoning capabilities could further narrow the gap in integration failures, but the structural agentic approach remains the critical differentiator.