💬 NLP

Text Knows What, Tables Know When: Clinical Timeline Reconstruction via Retrieval-Augmented Multimodal Alignment

This paper introduces a retrieval-augmented multimodal alignment framework that reconstructs precise clinical timelines by integrating semantically rich but temporally ambiguous unstructured narratives with structurally precise but incomplete EHR data, thereby significantly improving timestamp accuracy and capturing events missed by tabular records alone.

Original authors: Sayantan Kumar, Shahriar Noroozizadeh, Juyong Kim, Jeremy C. Weiss

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sayantan Kumar, Shahriar Noroozizadeh, Juyong Kim, Jeremy C. Weiss

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Text vs. The Spreadsheet

Imagine a patient's hospital stay is like a movie. To understand the story, you need two things:

The Script (The Doctor's Notes): This is the unstructured text where doctors write down what happened. It's full of rich details, feelings, and context (e.g., "The patient seemed lethargic and had trouble speaking"). However, the script is messy. It doesn't always say exactly when things happened. It might say "a few hours later" or "the next morning," leaving the timing fuzzy.
The Production Schedule (The Electronic Health Record): This is the structured data—tables of lab results, vital signs, and medication times. It is incredibly precise. It tells you exactly that a heart rate was 110 at 2:00 PM. But, it's dry and incomplete. It misses the "story." It might record a heart rate but miss the fact that the patient was complaining of dizziness right before it.

The Problem: If you try to reconstruct the patient's timeline using only the script, the timing is vague. If you use only the schedule, you miss huge chunks of the story.

The Solution: The authors built a "Time-Travel Assistant" (a computer system) that combines both. It uses the script to find the story events and the schedule to pin them down to the exact minute.

How the System Works: Building a House

The authors didn't just mash the two data sources together randomly. They built a four-step construction process, like building a house:

Step 1: Pouring the Foundation (Finding the "Anchor" Events)
First, the system reads the doctor's notes and picks out the most important, "central" events. These are the pillars of the story, like "Patient arrived at the ER" or "Patient was diagnosed with sepsis." These are the anchors that hold the timeline together.

Step 2: Rough Drafting (The Initial Sketch)
The system creates a rough timeline based only on the text. It guesses the order of events. For example, it knows "fatigue" happened before "weakness," but it might be unsure if that was 2 hours before or 10 hours before. The timeline is there, but the clock is fuzzy.

Step 3: Installing the Clocks (Retrieval-Augmented Calibration)
This is the magic step. The system looks at the structured data (the spreadsheet) to find precise timestamps for those "anchor" events.

Analogy: Imagine you are trying to remember when you had breakfast. You know it was "before work." That's vague. But then you check your calendar and see you left for work at 8:00 AM. Suddenly, you know breakfast was likely around 7:00 AM.
The system uses the precise data from the spreadsheet to "calibrate" the fuzzy text timeline. It sharpens the clock on the anchor events.

Step 4: Filling in the Walls (The Full Timeline)
Once the anchors are set to the correct time, the system goes back to the doctor's notes to find all the smaller, less important details (like "patient felt a bit cold" or "family visited"). It places these smaller events relative to the now-precise anchors. Finally, it does one last check against the spreadsheet to make sure everything still lines up perfectly.

What They Found: The "Text Knows What, Tables Know When" Rule

The researchers tested this system on real hospital data (specifically for sepsis, a life-threatening reaction to infection) and found some surprising things:

1. The Text is the Storyteller; The Table is the Timekeeper
The system didn't find more events by looking at the spreadsheet. The text alone was actually very good at finding what happened. The spreadsheet's real superpower was telling the system when it happened.

The Result: Adding the spreadsheet data didn't change the list of events much, but it made the timing of those events much more accurate.

2. The "Middle" is Where the Magic Happens
In a text-only timeline, the beginning and end are usually clear (like "Admitted" and "Discharged"), but the middle is a blur. The system found that the spreadsheet data was most helpful for fixing the "middle" of the story, turning a blurry sequence of events into a clear, step-by-step progression.

3. The Spreadsheet is Missing Half the Movie
The authors did a "gap analysis" and found something critical: 34.8% of the events found in the doctor's notes were completely missing from the spreadsheet.

Analogy: The spreadsheet is like a list of ingredients, but the doctor's note is the actual cooking process. The list might say "flour, eggs, sugar," but it won't tell you that the baker "burnt the first batch" or "forgot to stir."
The Finding: Important things like "patient seemed confused," "pain got worse," or "family was worried" often never make it into the structured tables. If you only look at the tables, you miss these crucial clues.

4. The Multi-Step Approach Wins
They tried different ways of building the timeline. They found that doing it all in one big jump (reading the text and guessing the time all at once) was the worst method. The best method was the "scaffold" approach: build the foundation first, calibrate it with the spreadsheet, and then build the rest of the house.

The Bottom Line

This paper proves that to understand a patient's journey, you can't just rely on the numbers, and you can't just rely on the stories.

Text gives you the rich, complete picture of what happened.
Tables give you the precise clock to tell you when it happened.

By using a smart system to combine them, the researchers created a timeline that is both complete (it has all the details) and precise (it knows exactly when things happened). This is a big step forward for understanding how diseases like sepsis progress over time.

Technical Summary: Text Knows What, Tables Know When

Problem Statement
Reconstructing precise clinical timelines is critical for modeling patient trajectories and forecasting risk in complex conditions like sepsis. However, existing data sources present a complementary but fragmented view of patient history. Unstructured clinical narratives (text) offer semantically rich, contextually complete descriptions of a patient's course, including early indicators of deterioration and functional status, but often lack temporal precision and contain ambiguous event timing. Conversely, structured electronic health record (EHR) data (tables) provides precise temporal anchors through laboratory values, vitals, and medications but frequently misses a substantial portion of clinically meaningful events, such as symptom progression, patient-reported outcomes, and causal context. Prior work has shown that relying on either modality alone results in either temporally imprecise trajectories or incomplete event coverage.

Methodology
The authors propose a retrieval-augmented multimodal alignment framework that bridges these gaps to improve the temporal precision of absolute clinical timelines extracted from text. The approach formulates timeline reconstruction as a graph-based multistep process rather than a single-pass extraction task. The pipeline operates in four phases:

Central Scaffold Construction: The system first extracts a set of temporally informative "central events" from the narrative to serve as temporal anchors. It then estimates pairwise temporal relations between these central events to build an initial relative temporal graph.
Scaffold Calibration (RAG): The initial central scaffold is calibrated using retrieved structured EHR rows. The system retrieves the top- $k$ structured rows most relevant to each central event via embedding-based similarity and uses this evidence to refine the absolute timestamps of the central anchors.
Full Timeline Assembly: Non-central events are extracted from the text and attached to the central scaffold with relative time offsets. These are combined with the calibrated central timeline to form an initial complete timeline.
Final Refinement: A second retrieval-augmented update is performed on the full set of events, using structured EHR evidence to further refine the timestamps of the entire trajectory.

The framework utilizes instruction-tuned large language models (LLMs) for all extraction, reasoning, and calibration steps, orchestrated via a graph-based workflow (LangChain/LangGraph). Evaluation is conducted on the i2m4 benchmark, derived from MIMIC-III and MIMIC-IV discharge summaries, comparing the multimodal pipeline against unimodal text-only baselines.

Key Contributions
The paper makes five primary contributions:

RAG-based Multimodal Integration: Development of a framework that aligns text-derived event timelines with structured EHR rows, demonstrating improved temporal precision over unimodal text-only reconstruction.
Multistep Graph-Based Pipeline: Introduction of a decomposition strategy that uses central events as temporal anchors to scaffold the reconstruction, followed by non-central event attachment and refinement, outperforming single-pass formulations.
Stage-Specific Integration Analysis: Empirical comparison of calibration strategies (updating only the central scaffold, only the final timeline, both, or neither), isolating the specific value of structured evidence at different stages.
Gap Analysis of Tabular Data: An empirical characterization of clinically relevant events present in text but absent, delayed, or semantically compressed in tabular data, quantifying the necessity of unstructured text for complete trajectory reconstruction.
Performance Characterization: Analysis showing that multimodal alignment yields the most significant gains when event timing cannot be directly inferred from text, particularly for events supported by structured evidence.

Results
Evaluated across seven instruction-tuned LLMs (including DeepSeek, GLM, and Qwen variants), the multimodal pipeline consistently improved temporal concordance and Absolute Timestamp Accuracy (AULTC) compared to unimodal text-only baselines, without compromising event match rates.

Temporal Precision vs. Event Recovery: The addition of structured evidence had little effect on the number of events recovered (event match rate remained stable), but significantly improved the accuracy of their timestamps. For instance, GLM5 achieved the highest concordance (0.812) and AULTC (0.829) in the multimodal setting.
Ablation Studies: The multistep graph decomposition proved superior to single-step generation. The best performance was achieved when structured evidence was used to calibrate both the central scaffold and the final full timeline.
Sensitivity Analysis: Events supported by structured EHR evidence showed uniform improvements in AULTC. Central events demonstrated significant gains in temporal ordering (concordance), though absolute timestamp accuracy gains were sometimes trade-offs for enforcing coherent chronology.
Information Gap: The gap analysis revealed that 34.8% of text-derived events were entirely absent from tabular records. Missing events included presenting symptoms, functional status, and causal factors. When tabular counterparts existed, they often provided useful temporal anchors (median discrepancy of 2.6 hours), but they failed to capture the full semantic content of the narrative.

Significance and Claims
The paper claims that the primary benefit of multimodal alignment in clinical timeline reconstruction is temporal calibration rather than event discovery. Structured EHR data serves most effectively as an external temporal anchor to sharpen the timing of events identified in narrative text, rather than as a substitute for the narrative's event content.

The authors argue that clinical timeline construction should be viewed as an alignment problem between complementary sources of temporal evidence. While narrative text provides the most complete account of the patient course (including early symptoms and context), structured records provide sharper temporal anchors. By reconstructing explicit patient timelines through this multimodal alignment, retrospective records can be transformed into longitudinally structured representations better suited for risk forecasting, causal analysis, and temporal decision support. The study suggests that even in text-only settings, decomposing the task around temporally informative anchor events may improve timeline quality, but the integration of structured data offers a distinct advantage in temporal fidelity.

The Big Picture: Text vs. The Spreadsheet

How the System Works: Building a House

What They Found: The "Text Knows What, Tables Know When" Rule

The Bottom Line

Technical Summary: Text Knows What, Tables Know When

More like this