💻 computer science

OphthaDT: Generative Digital Twins for Forecasting Visual Acuity Trajectories in Ophthalmology

OphthaDT is a novel LLM-based generative digital twin that serializes fragmented multimodal clinical data into structured narratives to accurately forecast visual acuity trajectories in ophthalmology, demonstrating superior performance over traditional baselines—particularly for complex neovascular age-related macular degeneration cases—while effectively handling irregular sampling without imputation.

Original authors: Pietro Belligoli, Nikita Makarov, Sayedali Shetab Boushehri, Fabian Schmich, Raul Rodriguez-Esteban, Michael Menden

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pietro Belligoli, Nikita Makarov, Sayedali Shetab Boushehri, Fabian Schmich, Raul Rodriguez-Esteban, Michael Menden

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict how a specific plant will grow over the next year. Usually, scientists look at the plant's history, measure its height every few weeks, and draw a straight line to guess where it will be in 12 months. This works great if the plant grows steadily. But what if the plant is erratic—sometimes shooting up, sometimes wilting, reacting wildly to rain or sun? A straight line fails there.

This paper introduces OphthaDT, a new kind of "digital plant" (or in this case, a Digital Twin) designed specifically for eye doctors. It uses a powerful type of AI called a Large Language Model (LLM) to predict how a patient's vision will change over time.

Here is how it works, broken down into simple concepts:

1. The Problem: Messy Medical Data

In real life, patient records are messy. People miss appointments, doctors measure things at different times, and data comes in different formats (numbers, images, notes). Traditional computer models often struggle with this "messiness." They usually require data to be perfectly organized in a grid, and if a patient misses a visit, the model has to guess (impute) the missing numbers, which can lead to errors.

2. The Solution: Turning Data into a Story

OphthaDT takes a different approach. Instead of forcing data into a spreadsheet, it turns a patient's entire medical history into a structured story.

The Analogy: Imagine taking a patient's scattered medical records—age, diagnosis, every eye exam, every injection they received, and every measurement—and writing them out as a chronological diary entry.
The Process: The AI reads this "diary" just like a human would. It sees the beginning (the patient's baseline health), the middle (what happened at each visit), and then it is asked to write the next chapter: "What will the patient's vision score be in 8 weeks? 24 weeks? 100 weeks?"

Because the AI is trained to understand language, it naturally handles missing visits or irregular timing without needing to guess the missing numbers first. It just reads the story as it is.

3. The Test: Two Different Eye Diseases

The researchers tested this "storytelling AI" on two very different eye conditions using data from 3,220 patients across four major clinical trials:

nAMD (Wet Macular Degeneration): This condition is like a wild rollercoaster. Vision can fluctuate wildly and unpredictably.
- The Result: OphthaDT was the clear winner here. It predicted the vision trajectory better than traditional math models (like Linear Regression) or standard machine learning tools (like Random Forests). It reduced prediction errors by about 6% compared to the others. The paper suggests this is because the AI is better at capturing the complex, non-linear "ups and downs" of this disease.
DME (Diabetic Macular Edema): This condition is more like a steady ramp. Once treated, patients' vision tends to stabilize and follow a predictable path.
- The Result: Here, the results were a tie. The simple, straight-line math models worked just as well as the fancy AI. In fact, for this specific disease, the simple models were sometimes slightly better at the very beginning. OphthaDT was still competitive, beating some older models, but it didn't show a massive advantage because the disease itself wasn't very chaotic.

4. The Takeaway

The paper concludes that complexity matters.

If the disease trajectory is simple and stable (like DME), simple tools work fine.
If the disease trajectory is complex and volatile (like nAMD), the "storytelling" power of the AI (OphthaDT) shines because it can understand the nuances that simple math misses.

What the paper does NOT claim:

It does not claim this is currently being used in hospitals to treat patients today.
It does not claim it can predict any eye disease (it was only tested on nAMD and DME).
It does not claim it replaces doctors; it is a tool for forecasting based on past data.

In short, OphthaDT is a new way to teach computers to read patient histories like stories, proving that this method is especially powerful when predicting the unpredictable.

Technical Summary: OphthaDT – Generative Digital Twins for Forecasting Visual Acuity Trajectories

Problem Statement

Precision medicine in ophthalmology requires accurate longitudinal predictions of treatment outcomes, yet this is hindered by the fragmented nature of multimodal clinical data. Traditional forecasting methods often struggle with irregular sampling, missing data, and the complexity of heterogeneous clinical histories. While Digital Twins (DTs) offer a pathway to create virtual patient replicas for personalized treatment and trial design, existing foundation models in ophthalmology (e.g., RETFound, VisionFM) are primarily limited to single-timepoint image-based classification or diagnosis. They generally fail to address longitudinal trajectory forecasting from structured clinical time series. Furthermore, while generative serialization methods have shown promise in oncology, their applicability to ophthalmic endpoints with distinct disease dynamics remains unexplored.

Methodology

The authors introduce OphthaDT, a Large Language Model (LLM)-based digital twin designed to forecast Best Corrected Visual Acuity (BCVA) trajectories. The core innovation lies in transforming fragmented, multimodal clinical records into structured textual narratives, allowing the LLM to process complex temporal dependencies without explicit imputation.

1. Data and Serialization

The model was trained on longitudinal data from 3,220 patients across four Phase III clinical trials involving neovascular age-related macular degeneration (nAMD) and diabetic macular edema (DME).

Input Data: The dataset includes static baseline characteristics (demographics, medical history) and time-stamped longitudinal events (visual acuity scores, imaging biomarkers, treatment history, vitals, and adverse events).
Text Serialization: The framework converts patient histories into a tokenizable narrative organized into three hierarchical modules:
1. Static Baseline: Demographics and pre-trial history.
2. Longitudinal History: A chronological sequence of visits detailing BCVA, administered drugs, and adverse events.
3. Forecasting Instructions: A structured prompt specifying the target variables and future timepoints (e.g., weeks 8, 12, 16... up to 100).
Handling Irregularity: This serialization approach naturally accommodates missingness and irregular sampling intervals, a common challenge in clinical trials, without requiring mean imputation or feature engineering.

2. Model Architecture and Training

Backbone: The system utilizes MedGemma 4B, a biomedical pre-trained LLM, selected for its instruction-tuning capabilities and open-source availability.
Training Objective: The model is fine-tuned using supervised learning on instruction-completion pairs. The training employs standard cross-entropy loss, masked to compute gradients only on the target completion tokens (the future BCVA values).
Inference: To generate predictions, the model samples multiple independent completions ( $M=10$ ). These text-based outputs are decoded into numerical trajectories by a structured parser, and the final forecast is derived as the element-wise mean of these trajectories to mitigate sampling variance.

Key Contributions

The paper outlines three primary contributions:

Serialization Method: A novel approach that maps multimodal ophthalmological events into a tokenizable narrative, enabling LLMs to perform forecasting tasks while handling irregular clinical data.
High-Accuracy Benchmarking: The establishment of a new benchmark for BCVA forecasting over extended horizons (up to 100 weeks), demonstrating superior performance in complex disease trajectories.
Generalizability Evidence: Empirical validation that LLM-based trajectory modeling, previously successful in oncology, generalizes to ophthalmic endpoints, despite differences in disease dynamics and treatment modalities (localized intravitreal injections vs. systemic therapies).

Results

OphthaDT was benchmarked against a Linear Model, Random Forest (RF), and XGBoost across four timepoints (Weeks 8, 24, 52, and 100). Performance was measured using Mean Absolute Error (MAE) in ETDRS letters and the coefficient of determination ( $R^2$ ).

nAMD (High Variability): OphthaDT achieved the lowest prediction error (MAE) across all timepoints compared to all baselines.
- It demonstrated an average MAE reduction of 6.0% compared to all baselines.
- Specifically, it outperformed Random Forest by 5.7% and the Linear Model by 2.1% on average over the 100-week horizon.
- The authors attribute this success to the model's capacity to capture the high longitudinal variability and non-linear biomarker interactions characteristic of nAMD.
DME (Stable Trajectories): In the DME cohort, treatment responses were more stable, converging toward a visual acuity plateau.
- The Linear Model remained the strongest baseline, achieving the lowest MAE and highest $R^2$ in most instances, particularly during the initial induction phase.
- However, OphthaDT maintained competitive performance, outperforming XGBoost by an average MAE reduction of 6.9% and Random Forest by 2.6%.
Subgroup Analysis: The model demonstrated consistent predictive precision across demographic (age, gender) and treatment subgroups, suggesting the serialization method effectively captures underlying clinical trajectories regardless of specific study protocols.

Significance and Claims

The paper claims that OphthaDT demonstrates the viability of generative digital twins for clinical trajectory forecasting in ophthalmology. Its significance lies in:

Methodological Shift: Moving away from manual feature engineering and imputation toward a holistic, text-based representation of patient history that leverages the inductive biases of foundation models.
Complexity-Dependent Advantage: The results suggest that the predictive advantage of LLM-based modeling scales with trajectory complexity. While linear models suffice for stable conditions like DME, OphthaDT's non-linear modeling capacity is essential for capturing the volatile, fluctuating nature of nAMD.
Practical Utility: By handling irregular sampling and heterogeneous data natively, this approach offers a practical methodology to reduce patient burden (by potentially reducing the need for dense sampling) and accelerate drug development through more accurate patient simulation.

The authors conclude modestly that while the current implementation is limited to a specific subset of clinical variables, the architecture's multimodal capabilities suggest that integrating imaging data (such as OCT scans) is a logical next step to further enhance the model's utility.