🤖 machine learning

Learning Dynamic Representations and Policies from Multimodal Clinical Time-Series with Informative Missingness

This paper proposes a novel framework for learning dynamic patient representations from multimodal clinical time series that explicitly leverages informative missingness through a multimodal encoder and Bayesian filtering, demonstrating improved performance in offline treatment policy learning and mortality prediction on large-scale ICU datasets.

Original authors: Zihan Liang, Ziwen Pan, Ruoxuan Xiong

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zihan Liang, Ziwen Pan, Ruoxuan Xiong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: How do you keep a patient alive when they are critically ill?

Usually, you'd look at the clues: their heart rate, blood tests, and the doctor's notes. But in a hospital, the clues are messy. Sometimes a doctor checks a patient every 5 minutes because they look sick. Other times, they check them once an hour because the patient looks stable. Sometimes, a nurse writes a long, detailed note about a sudden change; other times, the notes are short or missing.

The problem is that the missing clues are actually clues themselves.

If a doctor stops checking a patient's blood pressure, it might mean the patient is fine. But if they start checking it every minute, it means the patient is crashing. Most computer programs ignore these "missing" moments or treat them as errors. This paper introduces a new AI detective that doesn't just look at the data; it looks at how the data was collected to understand the patient's true condition.

Here is a simple breakdown of how their new system, called OPL-MT-MNAR, works:

1. The "Missing Clues" are Informative (The Detective's Intuition)

In the real world, doctors don't check patients randomly. They check them based on how sick the patient is.

The Analogy: Imagine a teacher grading students. If a student is struggling, the teacher might check their homework every day. If the student is doing great, the teacher might only check once a week.
The Mistake: Old AI models think, "Oh, the teacher only checked once a week, so the student must be fine."
The New AI: This model realizes, "Wait, the pattern of checking tells me something. If the teacher suddenly started checking every day, the student is in trouble, even if I don't have the homework yet."

The authors call this "Informative Missingness." They teach the AI to pay attention to when and why data is missing, not just the data itself.

2. Reading Between the Lines (Multimodal Fusion)

Hospitals have two main types of clues:

Structured Data: Numbers like heart rate, temperature, and lab results (like a spreadsheet).
Unstructured Text: Doctors' and nurses' notes (like a diary).

The Problem: These two types of clues arrive at different speeds. A heart rate monitor updates automatically. A nurse's note might only happen when something changes.
The Solution: The AI uses a special "translator" that combines the numbers and the words. But it doesn't just mash them together. It asks: "Is this nurse writing a note because the patient is getting worse, or just because it's shift change?" It uses the timing of the note to decide how much weight to give it.

3. The "Mental Model" (Bayesian Filtering)

This is the smartest part. The AI builds a hidden mental model of the patient's health.

The Analogy: Think of a chess player. They don't just look at the pieces on the board right now (the current data). They remember the history of the game and the moves the opponent made (past treatments) to guess what the opponent is thinking (the hidden state).
How it works: The AI tracks a "belief state." If a patient was given a strong drug and didn't get better, the AI updates its belief: "This patient is very sick, even if their current numbers look okay." It learns that actions change the patient, and it uses that history to predict the future.

4. Learning from the Past (Offline Policy Learning)

The AI learns by studying millions of past patient records (like a medical student reading old case files). It tries to figure out: "If the doctor had done X instead of Y, would the patient have lived?"

The Safety Net: Since the AI can't experiment on real patients, it uses a special math trick (called Implicit Q-Learning) to learn safely. It only suggests actions that are similar to what real doctors did, but slightly better, ensuring it doesn't suggest dangerous, crazy treatments.

5. The Results: Why It Matters

The team tested this AI on data from three huge hospital databases (MIMIC-III, MIMIC-IV, and eICU).

The Score: They measured how good the AI was at predicting who would survive and what treatment would work best.
The Winner: The new AI beat the average human doctor's decision-making style significantly.
- Prediction: It predicted death after 72 hours with 88.6% accuracy (much better than previous methods).
- Treatment: It found treatment strategies that were 28% better than standard care, especially for the sickest patients.

The Big Takeaway

This paper teaches us that silence is loud. In a hospital, the fact that a test wasn't done, or a note wasn't written, tells a story just as important as the test results themselves. By teaching AI to listen to the "silence" and the "rhythm" of medical records, we can build systems that understand patients more deeply, predict crises earlier, and save more lives.

In short: They built an AI that doesn't just read the medical chart; it reads the story behind the chart.

1. Problem Formulation

The paper addresses the challenge of learning optimal treatment policies and predicting patient outcomes from Electronic Health Records (EHRs) that are multimodal (structured measurements like vitals/labs and unstructured clinical text) and irregularly sampled.

Informative Missingness (MNAR): A core problem is that data is often missing not at random. The decision to record a measurement or write a note is driven by the patient's latent health state (e.g., sicker patients are monitored more frequently) and clinician behavior. Ignoring these "observation patterns" leads to biased representations.
Modality Differences: Structured data (vitals) and text (notes) follow different recording mechanisms. Structured data may be routine, while text updates depend heavily on documentation behavior, creating distinct temporal patterns of missingness across modalities.
Partial Observability: The true patient health state is latent. Standard models often fail to capture the cumulative effect of past treatments or the information contained within the absence of data.
Goal: To learn a dynamic patient representation that explicitly leverages these informative missingness patterns to optimize offline treatment policies (Off-Policy Learning) and predict adverse outcomes.

2. Methodology: OPL-MT-MNAR

The authors propose OPL-MT-MNAR (Off-Policy Learning under Multimodal Observations with Temporal Missing-Not-At-Random Patterns), a two-stage framework.

Stage 1: Patient State Representation Learning

This stage learns a dynamic latent state $s_h$ from the history of observations $I_h$ . It combines Bayesian filtering with multimodal encoding.

MNAR-Aware Multimodal Encoder:
- Structured Data: Uses an extension of GRU-D (Gated Recurrent Units-Decay). It incorporates explicit decay factors based on the time since the last observation. Crucially, it augments inputs with explicit MNAR features: cumulative observation counts, missing rates, and windowed observation frequencies.
- Clinical Text: Introduces a Documentation-Process Factor ( $F^{doc}_h$ ). This is a recurrent state updated at each step based on note presence, recency, and density, independent of the text content. It captures the "behavior" of documentation.
- Fusion: Uses a gating mechanism to fuse structured embeddings and text embeddings. The gate is modulated by the Documentation-Process Factor, allowing the model to weigh text based on its semantic content and the context of how it was documented (e.g., distinguishing a burst of new notes from stale data).
Latent Belief State (Variational Inference):
- To address partial observability and the need to track treatment history, the model maintains a latent belief state $z_h$ .
- Action-Conditioned Dynamics: The transition $z_h \to z_{h+1}$ is conditioned on the treatment action $a_h$ . The paper provides a theoretical proof (Theorem 1) that without action-conditioning, the policy gradient for future rewards vanishes in offline settings because the observation encoding is deterministic given the history.
- Implemented via a Variational Autoencoder (VAE) where the prior is conditioned on $(z_h, \phi_h, a_h)$ and the posterior is inferred from future observations.
State Construction: The final patient state $s_h$ is a combination of the unified observation embedding $\phi_h$ and the latent belief $z_h$ . A reconstruction loss ensures the state preserves sufficient information about the original observations and missingness patterns.

Stage 2: Downstream Tasks

The learned state $s_h$ is used for two joint tasks:

Offline Policy Optimization: Uses Implicit Q-Learning (IQL). It learns a value function via expectile regression (to avoid overestimation) and extracts a policy via advantage-weighted behavioral cloning. This avoids querying out-of-distribution actions.
Outcome Prediction: Predicts post-72-hour in-hospital mortality. Jointly training this task provides a direct, low-noise supervisory signal that improves the quality of the learned state representation (positive transfer).

3. Key Contributions

Explicit Modeling of Informative Missingness: Unlike prior work that treats missingness as noise or only handles it via imputation, this framework treats the observation process itself (timing, frequency, and burstiness of data) as a rich signal for patient state estimation.
Multimodal Integration with Process Awareness: It uniquely integrates clinical text by modeling the documentation process (when and how often notes are written) separately from the text content, using a dedicated recurrent factor to guide fusion.
Theoretical Insight on Action-Conditioning: The paper proves that for offline RL with terminal rewards, latent dynamics must be conditioned on actions to enable learning of long-term treatment effects; otherwise, the policy gradient is zero.
Joint Learning Framework: It demonstrates that jointly learning policy optimization and outcome prediction yields better representations than learning them separately.

4. Experimental Results

The framework was evaluated on three large-scale ICU datasets: MIMIC-III, MIMIC-IV, and eICU, focusing on sepsis treatment with 4-hour decision steps.

Policy Performance (Fitted Q-Evaluation - FQE):
- MIMIC-III: OPL-MT-MNAR achieved an FQE of 0.679, significantly outperforming clinician behavior (0.528) and strong baselines like MedDreamer (0.583) and SBCQ (0.501).
- Generalization: It maintained superior performance on MIMIC-IV (0.634 vs. 0.521) and eICU (0.604 vs. 0.534), demonstrating cross-institutional robustness.
- High-Acuity Patients: The largest gains were observed in high-severity (high SOFA score) patients, where observation patterns are most informative.
Outcome Prediction:
- Achieved an AUROC of 0.886 for post-72-hour mortality prediction on MIMIC-III, outperforming GRU-D (0.844), BRITS (0.852), and MedDreamer (0.867).
Ablation Studies:
- Removing the explicit MNAR and Documentation-Process modeling caused a significant drop in performance (FQE dropped from 0.679 to 0.507), confirming that modeling the observation process is the primary driver of improvement.
- Text integration provided substantial value, with nursing notes contributing the largest single-modality gain.

5. Significance

This work represents a significant advance in Clinical AI and Offline Reinforcement Learning:

Paradigm Shift: It moves beyond treating missing data as a technical nuisance to be imputed, recognizing that why and when data is missing is a critical clinical signal reflecting patient severity and clinician urgency.
Multimodal Utility: It effectively leverages the often-underutilized clinical text (notes) by modeling the documentation behavior, which is crucial for capturing the dynamic evolution of patient care.
Safety and Reliability: By using offline RL with conservative evaluation (IQL) and demonstrating robustness across different hospital systems, it offers a pathway toward safer, data-driven treatment recommendations that respect the complex, non-random nature of real-world medical data.
Theoretical Rigor: The proof regarding the necessity of action-conditioned latent dynamics provides a foundational theoretical justification for modeling treatment history in sequential decision-making under partial observability.