🤖 machine learning

Tokenization Tradeoffs in Structured EHR Foundation Models

This paper demonstrates that optimizing tokenization strategies—specifically by combining event and positional time encodings into single tokens—significantly improves both the predictive performance and computational efficiency of electronic health record foundation models across diverse clinical tasks.

Original authors: Lin Lawrence Guo, Santiago Eduardo Arciniegas, Joseph Jihyung Lee, Adam Paul Yan, George Tomlinson, Jason Fries, Lillian Sung

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lin Lawrence Guo, Santiago Eduardo Arciniegas, Joseph Jihyung Lee, Adam Paul Yan, George Tomlinson, Jason Fries, Lillian Sung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but very young student (the AI) how to understand the complex story of a patient's life, written in a medical record. This record isn't a novel; it's a chaotic timeline of thousands of events: "Patient had a fever," "Blood test showed high sugar," "Doctor ordered an X-ray," "Patient took medicine."

The big question this paper asks is: How should we translate this messy medical timeline into a language the AI can actually read and learn from?

The authors call this process Tokenization. Think of tokenization as the "translation dictionary" or the "Lego instructions" you give the AI before it starts building its understanding.

Here is the breakdown of their findings using simple analogies:

1. The Three Big Choices

The researchers tested three different ways to build this "dictionary":

Choice A: How to describe an event (The "Combo Meal" vs. "Separate Plates")
- The Problem: A medical event usually has a name (e.g., "Glucose") and a value (e.g., "High").
- Option 1 (Factorized): You give the AI two separate cards: one says "Glucose," the other says "High." The AI has to figure out that these two cards belong together.
- Option 2 (Joint): You glue them together into one card that says "High Glucose."
- The Finding: Gluing them together (Joint) worked better. It's like giving a student a pre-assembled Lego brick instead of two loose pieces. The AI doesn't have to waste brainpower figuring out that "High" and "Glucose" go together; it just sees the concept immediately. This made the AI smarter and faster to train.
Choice B: How to handle time (The "Calendar" vs. "The Clock")
- The Problem: Medical events happen at specific times. Does the AI need to know exactly how many days passed between events?
- Option 1 (Time-Tokens): You insert special "time cards" between events, like a card that says "Waited 3 days."
- Option 2 (Time-Positions): You don't add extra cards. Instead, you just tell the AI, "This event happened on Day 50 of the patient's life," and let the AI's internal clock handle the math.
- The Finding: The internal clock (Time-Positions) won. Adding extra "time cards" actually cluttered the AI's view and slowed it down. Just knowing the order and the patient's age was enough.
Choice C: How much detail to show (The "Movie" vs. The "Highlight Reel")
- The Problem: A lab test isn't just one moment; it's a process: Doctor orders it -> Nurse takes blood -> Lab processes it -> Result comes back.
- Option 1 (With Workflow): Show the AI every single step of that process.
- Option 2 (Without Workflow): Just show the final result.
- The Finding: Showing the whole "movie" (Workflow) helped the AI learn better at the hospital where the data came from. However, this didn't help when they tried to use the AI at a different hospital, because every hospital does the paperwork differently.

2. The "Local Binding" Secret

Why did the "Combo Meal" (Joint Encoding) work so much better? The authors discovered a concept they call Local Binding Efficiency.

Imagine you are teaching a child to recognize a "Red Apple."

The Hard Way (Factorized): You show them a card that says "Red" and a card that says "Apple." You have to teach them that when these two appear next to each other, they mean "Red Apple." If you show them "Red" and "Car," they have to learn that means something else. This is hard and confusing.
The Easy Way (Joint): You just show them a picture of a "Red Apple." The connection is built-in.

In medical data, "High" means something totally different for "Glucose" (bad) than it does for "Hemoglobin" (maybe good). If you split them up, the AI has to learn these complex rules from scratch. If you glue them together, the AI gets the answer immediately. This saved the AI a massive amount of computing power and made it learn faster.

3. The "Traveling" Test

The researchers trained their AI on data from a Pediatric Hospital (kids) and then tried to use it on data from an Adult ICU.

The Vocabulary Clash: The AI was confused because the adult hospital used different codes (like a different dialect). About 70% of the words the AI knew were "out of vocabulary" for the adults.
The Result: Even with this huge language barrier, the AI that used the "Combo Meal" (Joint Encoding) strategy still performed the best. It proved that understanding the structure of the data (gluing the right things together) is more important than memorizing every specific word.

The Bottom Line

This paper is a guide for building better medical AIs. It tells us:

Don't over-complicate the input. Glue related medical facts together into single, meaningful chunks.
Don't clutter the timeline. Let the AI use its internal sense of time rather than forcing it to read extra "time cards."
Structure matters more than size. By organizing the data smarter (Joint Encoding), we can build models that are not only more accurate but also cheaper and faster to train.

In short: Give the AI a clear, pre-assembled puzzle, not a box of loose, confusing pieces.

1. Problem Statement

Foundation models for structured Electronic Health Records (EHRs) are trained on longitudinal sequences of timestamped clinical events (e.g., diagnoses, labs, medications) to learn adaptable patient representations. A critical, yet understudied, component of training these models is tokenization: the process of converting raw clinical timelines into discrete model inputs.

Current EHR foundation models adopt various tokenization strategies without controlled comparison, leading to uncertainty regarding:

Information Preservation: What clinical structure (code-value associations, temporal intervals) is preserved vs. lost?
Efficiency: How do design choices affect sequence length, vocabulary size, and computational cost (FLOPs)?
Learning Burden: Which relationships must the model learn from scratch (e.g., associating a value with a code) versus which should be pre-computed?

The authors argue that tokenization choices are fixed at pretraining time and propagate to all downstream applications, making the optimization of this step crucial for performance and efficiency, especially given that EHR data is orders of magnitude smaller than language corpora.

2. Methodology

The study employed a rigorous $2 \times 2 \times 2$ factorial experimental design to isolate the effects of three tokenization axes.

Datasets

Primary Development: SickKids (The Hospital for Sick Children, Toronto), a pediatric dataset with ~2 million patients and 169 million events (2018–2025).
External Validation: MIMIC-IV (Beth Israel Deaconess Medical Center), an adult ICU dataset with ~340k patients and 179 million events.

Tokenization Design Axes

Event Encoding:
- Joint Encoding: A single token fuses the clinical code and its attribute (e.g., Glucose_High).
- Factorized Encoding: Separate tokens for the code and attributes (e.g., Glucose + High), requiring the model to learn the association.
Time Encoding:
- Time-Positions: Uses Rotary Positional Embeddings (RoPE) where the token's position corresponds to the patient's age in days.
- Time-Tokens: Inserts discrete tokens representing time intervals (e.g., "2h-6h") between events.
Workflow Annotation:
- With Workflow: Represents multi-step clinical actions (e.g., Order $\to$ Collection $\to$ Result) as distinct events.
- Without Workflow: Represents actions as a single event at the final timestamp.

Model Architecture & Training

Architecture: Decoder-only Transformer (28 layers, 768 hidden dimension, 116M backbone parameters).
Pretraining: Autoregressive next-token prediction on SickKids data.
Evaluation:
- Local: 74 clinical prediction tasks (AUROC) across 6 domains (e.g., lab abnormalities, mortality, readmission).
- External: 13 tasks on MIMIC-IV to test cross-institutional generalization.
- Ablation Studies: Targeted experiments to disentangle "token efficiency" (sequence length) from "local binding efficiency" (learning code-value associations).

3. Key Contributions

Systematic Factorial Analysis: The first large-scale, controlled comparison of tokenization strategies in EHR foundation models, quantifying independent effects on performance and compute.
Identification of "Local Binding Efficiency": The paper introduces the concept that in data-limited settings, pre-computing code-attribute associations (Joint Encoding) is superior to forcing the model to learn these associations from scratch (Factorized Encoding).
Efficiency-Performance Trade-off: Demonstrates that strategies improving performance (Joint Encoding, Time-Positions) simultaneously reduce computational cost, contrary to the intuition that more complex encoding yields better results.
Generalizability Insights: Shows that structural encoding advantages (Joint) transfer across institutions despite vocabulary mismatches, while temporal and workflow benefits are institution-specific.

4. Key Results

Performance and Efficiency

Joint Event Encoding: Outperformed Factorized encoding in 73/74 tasks. Crucially, it required 39.5% fewer pretraining FLOPs because it produced shorter sequences.
Time-Positions: Outperformed Time-Tokens in 71/74 tasks and required 9.6% fewer FLOPs. Time-Tokens degraded performance, likely by fragmenting sequences and consuming context window space without proportional information gain.
Workflow Annotations: Improved local performance (64/74 tasks) but increased compute costs by 35.6% (due to longer sequences) and showed no transfer benefit to the external adult cohort.

Mechanism Analysis (Ablations)

Binding Problem: The advantage of Joint Encoding was traced to local binding efficiency. In Factorized encoding, the model must learn that a specific value token (e.g., "High") means different things depending on the preceding code (e.g., "Glucose" vs. "Creatinine"). In data-limited EHR settings, the model struggles to learn these context-dependent associations. Joint encoding hard-codes this structure, resolving the binding problem at tokenization time.
Token Efficiency: When sequence lengths were equalized (Fixed-Event regime), the performance gap between Joint and Factorized encoding vanished, confirming that the primary driver was not just sequence length but the efficiency of information representation.

External Evaluation (MIMIC)

Vocabulary Mismatch: Applying SickKids-trained tokenizers to MIMIC resulted in a 69.8% out-of-vocabulary (OOV) rate.
Transferability: Despite the high OOV rate, the Joint Encoding advantage generalized (similar effect size to local evaluation). Conversely, Time-Positions and Workflow benefits did not transfer, likely due to demographic shifts (pediatric vs. adult age distributions) and institutional workflow differences.

5. Significance and Implications

Design Guidelines: The study establishes Joint Event Encoding and Positional Time Encoding as the recommended defaults for structured EHR foundation models. These choices maximize predictive performance while minimizing computational cost.
Data-Limited Learning: The findings highlight a fundamental trade-off in data-limited domains: pre-computing structural associations at tokenization time is often more effective than relying on the model to learn complex compositional structures from limited data.
Cross-Institutional Deployment: While vocabulary alignment remains a challenge for cross-institutional transfer, the structural benefits of joint encoding are robust. This suggests that future models should prioritize encoding clinically meaningful units (code-value pairs) rather than raw codes.
Efficiency: By reducing pretraining FLOPs by nearly 40% without sacrificing accuracy, these tokenization strategies make the development of large-scale EHR foundation models more feasible and sustainable.

In conclusion, the paper argues that tokenization is a "tractable lever" for improving EHR foundation models. By resolving the local binding problem through joint encoding, developers can achieve superior performance with significantly lower computational overhead.