🤖 AI

RealICU: Do LLM Agents Understand Long-Context ICU Data? A Benchmark Beyond Behavior Imitation

The paper introduces RealICU, a hindsight-annotated benchmark that evaluates large language models on long-context ICU data using physician-reviewed ground truth, revealing that current models struggle with recall-safety tradeoffs and anchoring biases despite improved memory architectures.

Original authors: Chengzhi Shen (Cherise), Weixiang Shen (Cherise), Tobias Susetzky (Cherise), Chen (Cherise), Chen (Cherise), Jun Li, Yuyuan Liu, Xuepeng Zhang, Zhenyu Gong, Daniel Rueckert, Jiazhen Pan

Published 2026-05-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Chengzhi Shen (Cherise), Weixiang Shen (Cherise), Tobias Susetzky (Cherise), Chen (Cherise), Chen (Cherise), Jun Li, Yuyuan Liu, Xuepeng Zhang, Zhenyu Gong, Daniel Rueckert, Jiazhen Pan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Hindsight" Problem

Imagine you are watching a high-stakes chess game where the clock is ticking down, and the board is changing every second. The players (doctors) have to make moves based on only the pieces they can see right now.

In the past, researchers tried to teach AI to play this game by showing it the moves the human players actually made and saying, "If the AI does what the human did, it's correct."

The problem: The paper argues this is a bad way to teach AI. Why? Because the human players were making moves with incomplete information. Sometimes, a doctor makes a move that seems right at the moment, but looking back later (with hindsight), we realize it was actually a mistake because we didn't know a crucial piece of information that would appear an hour later.

The Solution: The authors created a new test called RealICU. Instead of asking the AI, "Did you copy the doctor's move?", they ask, "Knowing everything that happened to the patient from start to finish, what should the doctor have done at this specific moment?"

The Setting: The ICU as a "Data Storm"

Think of an Intensive Care Unit (ICU) not as a quiet room, but as a hurricane of data.

Every 30 minutes, a patient generates a flood of information: heart rates, blood tests, nurse notes, medication logs, and imaging reports.
Doctors are like captains steering a ship through this storm. They have to constantly reassess: Is the patient getting better? Is there a new danger? What should we do next? What should we absolutely avoid?

The Four Tasks: The "Co-Pilot" Checklist

The paper tests AI on four specific jobs, acting like a clinical co-pilot sitting next to the doctor:

Patient Status: Is the patient getting better, staying the same, or getting worse?
Acute Problems: What are the immediate fires we need to put out? (e.g., "The patient is developing an infection.")
Recommended Actions: What should we do in the next hour to help? (e.g., "Give this specific fluid.")
Red Flags: What should we never do? (e.g., "Do not give this drug; it will kill the patient.")

The Datasets: The "Gold" and the "Scale"

To test the AI, they built two libraries of patient stories:

RealICU-Gold: A small, precious collection of 930 time-slots. These were carefully reviewed and labeled by a panel of senior human doctors who looked at the entire patient story before deciding what the right move was at each step. This is the "Gold Standard."
RealICU-Scale: A massive library of nearly 12,000 time-slots. Since humans can't label that much data, they trained a super-smart AI (called Oracle) to do the labeling. They checked that Oracle agreed with the human doctors, then let Oracle label the rest.

The Results: Where AI Stumbles

When they tested the smartest AI models available, the results were surprising and concerning. The AI struggled in two main ways:

1. The "Recall-Safety Tradeoff" (The "Shotgun" Problem)

The Issue: When the AI tried to be helpful and suggest many possible treatments (high recall), it started suggesting dangerous things.
The Analogy: Imagine a mechanic who, when asked to fix a car, suggests 10 different repairs. To make sure they don't miss anything, they suggest replacing the engine, even though the car just needs an oil change. In the ICU, this means the AI suggests treatments that could actually harm the patient.
The Stat: In some cases, nearly 47% of the AI's "helpful" suggestions were flagged as potentially dangerous.

2. The "Anchoring Bias" (The "First Impression" Problem)

The Issue: The AI gets stuck on its first guess about the patient and refuses to change its mind, even when new evidence proves it wrong.
The Analogy: Imagine a detective who decides a suspect is guilty in the first 5 minutes of an investigation. Even when the suspect is proven to be in a different city three hours later, the detective keeps trying to build a case against them.
The Result: The AI would keep recommending treatments for a condition the patient used to have, ignoring the fact that the patient had already recovered from it.

The Attempted Fix: ICU-Evo (The "Structured Memory" Agent)

The researchers tried to fix this by giving the AI a better memory system, called ICU-Evo. Instead of just reading the whole history like a book, they organized the memory into five specific "notebooks":

Working Memory: What happened right now.
Trend Memory: How vital signs are moving up or down over time.
Event Memory: A log of major "plot twists" (like a surgery or a sudden crash).
Trajectory Memory: A summary of the whole story so far.
Insight Memory: A special notebook for "hunches" about this specific patient (e.g., "This patient reacts strangely to painkillers").

Did it work?

Yes and No. The AI got much better at understanding the story and spotting problems (like the "Acute Problems" task).
But... It still made dangerous mistakes. The "structured memory" helped the AI reason better, but it didn't stop the AI from making unsafe suggestions. The paper concludes that memory alone isn't enough to make a safe ICU co-pilot.

The Bottom Line

The paper introduces RealICU as a new, stricter test for medical AI. It shows that while AI is getting smarter at reading medical data, it is still terrible at safety and updating its beliefs when new information arrives.

The authors warn that we cannot just trust AI to copy what doctors do in the past. We need to build systems that can reason through the entire patient journey and, most importantly, know when not to act. Until then, AI in the ICU is like a very knowledgeable but reckless navigator: it knows the map, but it might steer the ship into a rock if we aren't careful.

Technical Summary: RealICU – A Benchmark Beyond Behavior Imitation for LLM Agents in ICU Decision Support

1. Problem Statement

Intensive Care Units (ICUs) generate dense, evolving streams of clinical data requiring physicians to reassess patient states under significant time pressure. While Large Language Models (LLMs) and agentic systems have advanced, existing clinical benchmarks fail to adequately evaluate their capabilities in this specific domain.

The primary limitations of current benchmarks are twofold:

Static vs. Sequential: Most benchmarks reduce clinical reasoning to static question answering, diagnosis, or single-endpoint prediction (e.g., mortality), failing to assess reasoning across a changing patient trajectory.
Behavioral Imitation vs. Clinical Correctness: Existing benchmarks using Electronic Health Record (EHR) databases like MIMIC-IV typically treat recorded clinician actions as ground truth. However, these actions are made under incomplete information and limited temporal context. Consequently, they may be suboptimal. Evaluating AI against these labels rewards the imitation of potentially flawed human behavior rather than true clinical correctness.

There is a critical need for a benchmark that evaluates AI systems on clinical correctness using hindsight judgment, where the "optimal" action is determined after reviewing the full patient trajectory, rather than relying on what was recorded at the bedside.

2. Methodology

2.1 The RealICU Benchmark

The authors introduce RealICU, a hindsight-annotated benchmark designed to evaluate LLM-based clinical decision support in the ICU. The benchmark is built upon the MIMIC-IV dataset and focuses on four physician-motivated tasks that constitute the core capabilities of an ICU "co-pilot":

Patient Status: Classifying whether the patient is improving, stable, or deteriorating.
Acute Problems: Identifying free-text sets of acute problems or emerging risks requiring active management.
Recommended Actions: Proposing free-text sets of actions likely to benefit the patient within the next hour.
Red Flag Actions: Identifying high-risk actions that should be avoided to prevent unsafe outcomes.

Data Construction:

Windowing: ICU trajectories are partitioned into 30-minute windows with a 2-hour stride.
Partial Observation vs. Hindsight Labeling: At inference time, the model observes only data available up to time $t$ . However, the ground truth labels are generated by senior physicians (or a validated LLM) reviewing the entire patient trajectory up to discharge. This creates an asymmetry that mirrors the gap between real-time decision-making and retrospective review.
Datasets:
- RealICU-Gold: Contains 930 windows from 94 distinct ICU stays, labeled by consensus among five senior ICU physicians. Inter-rater reliability (IRR) ranges from 0.826 to 0.985.
- RealICU-Scale: Extends the evaluation to 11,862 windows using Oracle, an LLM-based hindsight evaluator validated against RealICU-Gold (achieving >0.895 F1 score across tasks).

2.2 Evaluation Framework

Models are evaluated on their ability to predict the four tasks based on partial observations.

Metrics: Accuracy and Macro-F1 for Patient Status; Hit@k and Recall@k (k=5) for Acute Problems and Recommended Actions (using semantic matching via PubMedBERT); and Harmful Recommendation Rate (HRR) for Red Flags, which measures the fraction of recommended actions that are flagged as unsafe.
Baselines: The study compares various LLM backbones (Gemini-3.1-pro, GPT-5.4, Qwen3-235B) across different context configurations: Full-context, Local-window, Retrieval-Augmented Generation (RAG), and a structured memory agent.

2.3 ICU-Evo: Structured Memory Agent

To address the challenges of long-horizon reasoning, the authors propose ICU-Evo, a memory-augmented agent framework. Unlike standard agents that treat all context equally, ICU-Evo maintains a structured memory state $M_t$ decomposed into five components:

Working Memory ( $M^{work}$ ): Recent raw observations at high resolution.
Trend Memory ( $M^{trend}$ ): Signal trends of vital signs and lab values.
Critical Event Memory ( $M^{event}$ ): An append-only log of clinically critical events (e.g., interventions, turning points).
Trajectory Memory ( $M^{traj}$ ): Compressed narrative summaries of the stay at periodic intervals.
Insight Memory ( $M^{insight}$ ): Patient-specific hypotheses regarding deviations from population-level expectations.

ICU-Evo utilizes three specialized agents (Observation, Assessment, Insight) operating at different temporal scales to update these memory components, aiming to improve long-horizon reasoning.

3. Key Results

3.1 Current LLMs Struggle with RealICU

Despite using frontier LLMs, performance on RealICU remains low, indicating the benchmark is unsolved.

Performance Gaps: Even the best-performing system (ICU-Evo with Gemini-3.1-pro) achieved only 0.459 accuracy on Patient Status and 0.534 Recall@5 on Action Recommendation.
Oracle vs. Agent Gap: There is a massive performance gap between the Oracle (which has full hindsight) and the Agent (real-time). For Patient Status, Oracle achieves an F1 of 0.987, while the best agent achieves only 0.365. This suggests the bottleneck is not medical knowledge but the ability to integrate evidence over time under partial observability.

3.2 Structured Memory Improves Reasoning

ICU-Evo consistently outperforms baselines (Full-context, Local-window, RAG) across all four tasks.

Long-Horizon Stability: ICU-Evo maintains high Hit@5 scores (near 0.8) for Acute Problems even in stays extending to 1,800 hours, whereas non-memory baselines degrade significantly.
Token Efficiency: While ICU-Evo consumes more tokens per prediction than local-window baselines, it provides significantly denser longitudinal context per input token, making it more efficient for timeline-aware reasoning.

3.3 Identified Failure Modes

The study identifies two critical failure modes in current LLM agents:

Recall-Safety Tradeoff: Agents that increase the recall of recommended actions often simultaneously increase the rate of harmful suggestions. For instance, ICU-Evo with GPT-5.4 improved Action Recommendation Hit@5 by over 20 points compared to RAG, but the HRR (Harmful Recommendation Rate) doubled from 0.234 to 0.473. The harmful recommendations frequently involve hemodynamic management, volume/diuresis, anticoagulation, and ventilation.
Anchoring Bias: Agents tend to over-commit to early interpretations of the patient and fail to update these hypotheses even when later evidence contradicts them. Case studies show agents generating "insights" based on early trajectory data that persist and mislead subsequent recommendations, even when current window data suggests a different state (e.g., maintaining a stable status despite new deterioration signals).

3.4 Ablation Studies

Removing specific memory components reveals their distinct roles:

Working Memory: Crucial for local reasoning; its removal degrades all tasks.
Trajectory Memory: Essential for temporal understanding; its removal hurts Acute Problems and Action Recommendation.
Insight Memory: Removing insight memory sometimes improves performance (e.g., reducing HRR or improving Action Recommendation Hit@5), suggesting that current LLMs are not yet capable of reliably identifying personalized clinical patterns and that their generated insights can actively mislead the agent.

4. Significance and Claims

The paper claims that RealICU provides a clinically grounded testbed for measuring and improving AI sequential decision support in high-stakes care. Its significance lies in:

Shifting the Evaluation Paradigm: Moving from "behavioral imitation" (matching recorded clinician actions) to "clinical correctness" (matching hindsight expert judgment). This exposes the limitations of current models in reasoning under uncertainty and partial observability.
Revealing Safety Gaps: The benchmark highlights that current LLM agents, even with advanced memory structures, struggle with safety in dynamic environments. The identified "recall-safety tradeoff" and "anchoring bias" suggest that simply scaling context or adding memory is insufficient for reliable ICU co-pilots.
Methodological Contribution: RealICU offers a framework for evaluating AI in high-stakes sequential decision environments where recorded human actions are imperfect and the optimal action is only visible in hindsight.

The authors conclude that while structured memory (ICU-Evo) improves long-horizon reasoning, it does not fully eliminate safety failures. Reliable ICU co-pilots will require advances in long-context clinical reasoning architectures alongside better safety mechanisms, and the adoption of hindsight evaluation to properly assess these systems.