🤖 AI

Large language models as synthetic clinical experts to inform longitudinal rare-disease modeling

This paper proposes using large language models as synthetic clinical experts to supervise a variational autoencoder, thereby integrating domain knowledge into the representation learning of longitudinal rare-disease data to improve clinical faithfulness and prediction accuracy.

Original authors: Clemens Schächter, Astrid Pechmann, Janbernd Kirschner, Jan Hasenauer, Harald Binder

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Clemens Schächter, Astrid Pechmann, Janbernd Kirschner, Jan Hasenauer, Harald Binder

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of medicine, some conditions are so rare that doctors struggle to gather enough information to understand how they change over time. When a disease affects only a handful of people, the data collected from patient visits is often sparse, messy, and difficult to fit into standard statistical models. To build a useful picture of how these diseases progress, researchers usually need to combine the numbers they have with deep clinical knowledge—the kind of intuition and experience that a specialist doctor gains over years of practice. However, turning that human expertise into a mathematical rule is incredibly difficult. Doctors often rely on subtle context, like a child's age or the specific combination of symptoms, to make a diagnosis, and these nuances are hard to write down as a simple formula. Without a way to include this expert judgment, computer models might learn patterns that look mathematically correct but make no sense to a human clinician, potentially leading to misleading predictions about a patient's future.

A team of researchers has developed a new way to bridge this gap by using artificial intelligence to act as a stand-in for a human specialist. They tested their method on data from children with spinal muscular atrophy, a rare neuromuscular disorder that causes progressive muscle weakness. In this condition, doctors classify patients into different types based on how severe their symptoms are and which motor milestones, such as sitting or walking, they are likely to reach. These classifications are not just labels; they guide treatment and help families understand what to expect. The challenge is that two children with very similar muscle strength scores might be classified differently if one is much younger than the other, because the same physical ability means something different at different ages. The researchers wanted to build a computer model that could learn the complex patterns of this disease while respecting these age-dependent clinical rules, without needing a human expert to sit down and write out every single rule by hand.

To solve this, the team created a system where a large language model, a type of artificial intelligence trained on vast amounts of medical text, acts as a "synthetic clinical expert." Instead of trying to force the AI to follow a rigid set of instructions, the researchers asked it to look at descriptions of a child's motor skills and decide which of two possible disease categories the child fits into best. They did this for thousands of patient records, asking the AI to make these judgments repeatedly to ensure it was consistent. Once the AI had provided its opinions, the researchers trained a smaller, faster computer program to mimic those judgments. This smaller program became a "surrogate" that could instantly evaluate any new data and tell the researchers whether a specific pattern of symptoms matched a particular disease type.

The real innovation happened when the researchers used this surrogate to guide the training of a larger model designed to understand the disease's progression. This larger model works by taking complex, high-dimensional data from patient visits and compressing it into a simpler, low-dimensional summary. Think of this summary as a compact map of the patient's condition that captures the most important details while ignoring the noise. Usually, these models are trained just to make sure the map can be expanded back into the original data with high accuracy. However, the researchers added a new rule to the training process: the model was penalized if it created a summary that, when expanded back into a patient profile, changed the disease classification. For example, if the original data suggested a child had a moderate form of the disease, the model was not allowed to create a summary that, when reconstructed, looked like a severe form, even if the numbers were very close. The surrogate expert acted as a referee, checking every attempt the model made to ensure the clinical meaning remained intact.

When the researchers applied this method to real-world data from a registry of spinal muscular atrophy patients, the results showed that the approach worked. The model trained with the help of the synthetic expert made fewer mistakes in preserving the correct disease classification compared to models trained without this guidance. Specifically, the disagreement between the original patient records and the model's reconstructed versions dropped from about 11 percent to 7 percent. This meant the model was better at keeping the clinical story of the patient consistent, even while simplifying the data. Furthermore, the summaries created by this improved model were better at predicting future milestones, such as when a child would be able to sit up or walk, than the summaries from models that ignored the expert knowledge. The study suggests that by using artificial intelligence to simulate the judgment of a clinical expert, researchers can build statistical models that are not only mathematically sound but also clinically faithful, capturing the subtle, context-dependent realities of rare diseases that traditional methods often miss.

Technical Summary: Large Language Models as Synthetic Clinical Experts for Longitudinal Rare-Disease Modeling

1. Problem Statement

Modeling longitudinal data for rare diseases presents a unique challenge: the scarcity of patient data necessitates the integration of clinical knowledge to guide statistical models. However, eliciting and formalizing this expert knowledge into explicit mathematical constraints, priors, or formulas is often difficult. Clinical distinctions in rare diseases (e.g., Spinal Muscular Atrophy, SMA) are frequently based on experience, contextual factors like age, and nuanced interpretations of ordinal test scores, making them hard to encode as rigid rules. Furthermore, the sustained collaboration required between clinical and statistical experts is time-consuming and prone to miscommunication. Consequently, standard statistical models often optimize for easily formalizable objectives (e.g., likelihood, reconstruction loss) that may fail to preserve clinically relevant structures, such as disease severity boundaries.

2. Methodology

The authors propose a framework that utilizes Large Language Models (LLMs) as "synthetic clinical experts" to supervise the training of a generative representation model. The approach consists of three main stages:

A. Synthetic Expert Label Generation (Offline)

Input: Clinical observations (e.g., HINE-2 motor function profiles) and contextual data (e.g., patient age) are rendered as natural language text.
LLM Querying: LLMs are queried via pairwise comparison tasks. For a given profile, the LLM is asked to choose between two candidate clinical labels (e.g., "SMA Type 1" vs. "SMA Type 2").
Consistency Auditing: The pairwise responses are aggregated to form a final synthetic label. This process includes checks for:
- Ordinal Consistency: Ensuring choices respect the ordered nature of disease severity (e.g., Type 1 < Type 2 < Type 3).
- Stability: Verifying that the choice remains consistent when the order of the candidate labels is reversed.
- Uncertainty: LLMs are queried to return an uncertainty score for each answer on a low, medium, or high scale, which is examined to see if it is elevated for samples where the LLM failed to recover the same label after label-order reversal.
Surrogate Training: A differentiable neural network (surrogate model) is trained on the retained, consistent LLM judgments. This surrogate maps clinical profiles and context to probability distributions over clinical labels, effectively distilling the LLM's non-differentiable decision process into a differentiable function.

B. Synthetic-Expert Supervised Model Fitting

Core Architecture: The study employs a Conditional Variational Autoencoder (cVAE) to learn low-dimensional latent representations ( $z_{i,t}$ ) of visit-level observations, linked across time by a multivariate mixed-effects model.
Loss Function Augmentation: The standard cVAE loss is augmented with a "synthetic expert loss" term.
- Let $S_\psi$ be the frozen surrogate network.
- Let $\pi = S_\psi(y, s)$ be the label distribution for the observed profile and $\hat{\pi} = S_\psi(\hat{y}, s)$ be the distribution for the reconstructed profile.
- The loss includes the Jensen-Shannon (JS) divergence between $\pi$ and $\hat{\pi}$ : $L_{SE} = L_{recon} + \lambda_{SE} \cdot JS(\pi, \hat{\pi})$ .
Objective: This term penalizes reconstructions that are numerically close to the input but alter the clinical interpretation (e.g., crossing a disease-type boundary). The surrogate parameters $\psi$ are fixed during this phase, while gradients flow through the reconstructed input to update the cVAE parameters.

C. Downstream Evaluation

The learned latent representations are evaluated using a time-varying Cox proportional hazards model to predict future motor milestones (sitting, standing, walking).

3. Key Contributions

Novel Supervision Mechanism: The paper introduces a method to incorporate LLM-derived clinical judgments directly into the fitting objective of a generative model, moving beyond using LLMs merely for downstream label generation or weak supervision.
Differentiable Surrogate Strategy: To address the non-differentiability and stochasticity of LLM outputs, the authors propose training a surrogate model to approximate the LLM's decision boundary, enabling gradient-based optimization of the generative model.
Application to Rare Disease: The framework is applied to real-world longitudinal SMA data, a setting where clinical categories are context-dependent and difficult to formalize explicitly.

4. Results

The study was conducted on the SMArtCARE registry data involving children with SMA.

LLM Consistency: Consistency of LLM judgments was highly dependent on model size. Larger models (e.g., Gemma-4-31B) achieved high ordinal consistency (~90%) and stability under label-order reversal, whereas smaller models performed near random baselines.
Reconstruction Consistency: Increasing the supervision weight ( $\lambda_{SE}$ $λ_{S E}$ ) significantly reduced the disagreement between the observed and reconstructed profiles regarding their clinical labels.
- Disagreement in SMA type labels dropped from ~11% (unsupervised) to ~7% with optimal supervision.
- This improvement was achieved without a substantial trade-off in numerical reconstruction error (HINE-2 sum score), although profiles that previously crossed clinical boundaries saw the most significant quality improvements.
Prognostic Utility: Latent representations informed by the synthetic expert outperformed unsupervised latent representations and data-level baselines (raw scores) in predicting future motor milestones (sitting, standing, walking) using Cox regression.
- The supervised latent variables showed improved discrimination (AUC) and calibration (Brier score, ECE).
- Moderate supervision weights (e.g., $\lambda_{SE} = 25$ ) provided the best balance; excessive weights (e.g., 100) sometimes degraded performance, suggesting a need for careful hyperparameter tuning.

5. Significance and Claims

The authors claim that this work demonstrates a practical pathway to integrate clinical knowledge into statistical modeling without requiring explicit mathematical formalization of expert rules. By using LLMs as synthetic experts, the method makes domain knowledge accessible to representation learning, improving the "clinical faithfulness" of the learned latent space.

The paper emphasizes that this approach does not replace clinical expertise but creates a more practical interface between clinical reasoning and statistical modeling. It highlights that while the surrogate model introduces an approximation error, the approach successfully discourages clinically inconsistent reconstructions and improves the prognostic utility of the resulting models for longitudinal rare-disease data. The authors note limitations, including the dependence on prompt quality, the potential for systematic errors in the synthetic expert to be distilled into the model, and the need for careful auditing of LLM outputs before clinical deployment.