🤖 machine learning

DynImmune-BERT: Dynamic Immune Repertoire Modeling with Neural ODE Driven Continuous Transformers

This paper introduces DynImmune-BERT, a novel continuous-time model that leverages Neural ODEs and specialized attention mechanisms to effectively capture longitudinal T cell receptor repertoire dynamics for improved patient immune status prediction, addressing the limitations of static language models in representing temporal immune changes.

Original authors: Rong Fu, Yongtai Liu, Xiaowen Ma, Haoyu Zhao, Shuo Yin, Yiqing Lyu, Long Zhang, Wangyu Wu

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Rong Fu, Yongtai Liu, Xiaowen Ma, Haoyu Zhao, Shuo Yin, Yiqing Lyu, Long Zhang, Wangyu Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your body as a bustling, high-tech city with a massive, ever-changing security force: your immune system. This army is made up of millions of unique soldiers called T-cells, each carrying a specific ID badge (a T-cell receptor) designed to recognize different threats like viruses or cancer. When the city faces an invader, the soldiers with the right badges multiply rapidly to fight the battle, while others stand down. This entire collection of soldiers and their ID badges is called your "immune repertoire."

Scientists have long tried to read these ID badges to understand what's happening inside a person. If they see a specific badge suddenly appearing in huge numbers, it might mean an infection is brewing. If a badge disappears, it could mean the threat is gone. However, most previous computer programs treated these immune armies like a static photo album. They would take a single snapshot of a patient's blood, count the badges, and make a guess. The problem is that life isn't a photo; it's a movie. Immune systems change constantly, with soldiers arriving, leaving, and reappearing at irregular times. To truly understand the story of a disease, you need to watch the whole movie, not just one frame.

This is where a new study called DynImmune-BERT comes in. The researchers built a smart computer model that doesn't just look at a single snapshot of your immune army; it watches the entire movie over time. Instead of treating your immune system as a static bag of parts, they modeled it as a living, breathing journey that evolves between irregular check-ups.

Here is how they did it and what they found:

The Problem with "Static" Photos

Imagine trying to understand a soccer game by only looking at the final score. You'd know who won, but you'd miss the drama: the goal scored in the first minute, the player who got injured in the second, and the comeback in the last. Previous models did something similar with immune data. They summarized a patient's blood sample as a "bag of sequences," ignoring when the samples were taken or how the specific immune cells changed between visits. This missed crucial clues about how the body was reacting to diseases like cancer.

The Solution: A Time-Traveling Detective

The authors created DynImmune-BERT, a system that treats a patient's immune history as a continuous, flowing story. Here are the clever tricks they used to make it work:

The "Smoothie" of Time (Neural ODEs):
In the real world, patients don't give blood samples at perfect, regular intervals. Sometimes it's been two weeks; other times, it's been three months. Traditional computers struggle with this "jagged" timeline. DynImmune-BERT uses a mathematical tool called a Neural Ordinary Differential Equation (Neural ODE). Think of this as a time-traveling detective that can smoothly fill in the gaps between two blood draws. It doesn't just guess; it mathematically "flows" the immune story forward, predicting how the army might have changed during the weeks in between, even if no data was collected then.
The "Disappearing Act" (Event-Aware Restarts):
Sometimes, a specific immune soldier (a clone) vanishes from the blood sample. In older models, the computer might get confused, thinking the soldier just got smaller. But in reality, the soldier might have completely disappeared and then reappeared later. DynImmune-BERT is "event-aware." If a soldier vanishes and then suddenly shows up again, the model hits a "reset button." It uses a special low-rank meta-adapter (a tiny, efficient helper) to instantly re-initialize that soldier's state, acknowledging that a new chapter has begun. This prevents the model from getting lost in the math.
The "VIP and the Crowd" (Hybrid Supervision):
In any immune army, a few "VIP" soldiers (dominant clones) are very loud and numerous, while thousands of others are rare but might hold the secret to a specific disease. Old models often focused only on the VIPs or got overwhelmed by the noise. DynImmune-BERT uses a hybrid transport objective. It's like having two different types of microphones: one tuned to hear the loud VIPs clearly, and another sensitive enough to pick up the whispers of the rare soldiers. This ensures the model learns from everyone, not just the most popular cells.
The "Neighborhood Watch" (Bounded Attention):
With thousands of immune cells, it's impossible for the computer to compare every single cell to every other cell (that would be too slow). Instead, the model builds a "bounded neighborhood." For any given cell, it only looks at its closest neighbors—those that are similar in sequence or abundance. This keeps the calculation fast and efficient, like a neighborhood watch that only talks to the houses right next door, rather than calling the whole city.

What They Found

The researchers tested their new model on data from patients with lung cancer and thyroid cancer (THCA). They compared DynImmune-BERT against many existing "static" models and simpler time-based models.

The Results: In their internal tests (where they compared different versions of their own model using the same data), DynImmune-BERT achieved an AUC of 0.982 for lung cancer and 0.984 for thyroid cancer. (An AUC of 1.0 is a perfect score, so these are very high).
The Comparison: When compared to other famous models like DeepTCR or BertTCR in the literature, DynImmune-BERT showed higher scores. However, the authors are careful to note that these comparisons are tricky because the other studies used different data sets and rules. They emphasize that the real proof comes from their internal tests, which showed that adding the "time" and "event" features significantly improved accuracy over models that just looked at the last sample or averaged everything out.
The Caveats: The model works best when you have a patient's repeated blood samples over time. When they tried to test it on very small groups of patients with rare cancers (some groups had fewer than 20 people), the results had "wide uncertainty." This means that while the model looks promising, we need more data to be absolutely sure it works for every single type of cancer.

Why It Matters

DynImmune-BERT suggests that to truly understand how our immune systems fight disease, we need to stop taking snapshots and start watching movies. By respecting the irregular timing of real-world medical visits and handling the "disappearing and reappearing" nature of immune cells, this model offers a more accurate way to predict cancer status.

The authors conclude that while the model is a powerful new tool, it is not a magic wand yet. It requires careful interpretation, especially for rare diseases, and needs more testing on larger groups of people before it can be used in hospitals. But for now, it proves that when we let our computers understand the flow of time, they can see the immune system's story much more clearly than ever before.

Technical Summary: DynImmune-BERT

1. Problem Statement

Adaptive immune repertoires, specifically T cell receptor (TCR) sequences, contain temporal signals regarding clonal expansion, contraction, disappearance, and reappearance following immune perturbations. While static repertoire language models effectively summarize samples as "bags of sequences," they fail to adequately represent the irregular sampling intervals, sequencing depth variations, and dynamic clone presence patterns inherent in longitudinal data.

Existing deep learning approaches for repertoire analysis (e.g., DeepTCR, BertTCR) generally aggregate clonotypes within a single sample or compare samples independently. This creates a mismatch between the biological reality—sparse, heavy-tailed, and event-driven dynamics—and computational representations that are typically time-independent. Furthermore, TCR data presents specific challenges:

Compositional Nature: Clone frequencies are constrained by sequencing depth.
Zero Inflation: Many clones are absent at most time points due to biological disappearance, limited sampling depth, or technical censoring.
Irregularity: Longitudinal studies often span weeks or months with uneven time intervals.
Event Discontinuity: Clone birth and death violate the assumption that all latent states evolve smoothly under a single continuous vector field.

The core problem addressed is how to encode a subject's irregular TCR trajectory to predict patient-level immune status (specifically cancer detection) while preserving clone presence events, compositional stability, and supervision from both high-abundance and low-abundance clones.

2. Methodology: DynImmune-BERT

DynImmune-BERT is a continuous-time repertoire model driven by Neural Ordinary Differential Equations (Neural ODEs) and continuous Transformers. It treats the immune repertoire as an evolving patient trajectory.

2.1 Depth-Adaptive Compositional State Initialization

To handle the compositional nature of sequencing data and zero-count instability, the model employs a Depth-Adaptive Centered Log-Ratio (CLR) initialization:

Pseudocounts: A depth-adaptive pseudocount $\epsilon(t) = \kappa / (N(t) + \kappa)$ is used, where $N(t)$ is the sequencing depth. This regularizes low-depth observations more strongly.
Stabilization: The initial state $y_c(t)$ is computed as the log-ratio of the relative frequency plus the pseudocount, centered over the support set. This stabilizes abundances and enables continuous-time dynamics.
Presence Gates: A binary indicator $I_c(t)$ tracks whether a clone is observed ( $>0$ reads) or absent.

2.2 Bounded Neighborhood Construction

To manage computational complexity in repertoires with thousands of clones, the model constructs a bounded attention neighborhood $N_c(t)$ for each clone $c$ . This neighborhood is limited to size $B$ and comprises three components:

High-abundance neighbors: Clones with high read counts.
Embedding neighbors: Clones with sequence embeddings nearest to the focal clone.
Rare-clone tail: Sampled low-abundance candidates to ensure rare clonotypes are not ignored.
This reduces attention from full pairwise comparison to local aggregation while preserving context from abundance, sequence similarity, and rare exploration.

2.3 Event-Aware Continuous Dynamics

The core of the model is a Neural ODE that propagates latent states between irregular observations:
$\frac{dy_c(t)}{dt} = I_c(t) \cdot F_\theta(y_c(t), e_c, N_c(t), s(t), t)$

Gated Dynamics: The presence gate $I_c(t)$ prevents absent clones from drifting under the vector field $F_\theta$ (a continuous Transformer).
Event-Aware State Restart: When a clone transitions from absent to present (reappearance), the continuous evolution is interrupted. A Low-Rank Meta-Adapter $\Delta_\phi$ initializes the reappearing clonotype's state based on its sequence embedding and time, followed by a re-centering step to restore the CLR constraint. This handles the discontinuity of clone birth/reappearance without forcing smooth interpolation.

2.4 Hybrid Transport Supervision

To supervise both dominant and rare clones, the model uses a Hybrid Transport Objective:

Dominant Clones: Uses Entropic Optimal Transport ( $S_\lambda$ ) on the top clones to match probability distributions.
Rare Clones: Uses Sliced Wasserstein Distance ( $SW_L$ ) on the tail clones to handle diffuse distributions.
Total Loss: Combines Cross-Entropy, the hybrid transport loss, temporal regularization, and spectral regularization.

3. Key Contributions

Trajectory Formulation: The paper formulates immune repertoire classification as a patient-level trajectory problem where observation time, read counts, sequencing depth, and clone presence are explicit variables.
Event-Aware Continuous Transformer: It introduces a novel architecture combining stabilized compositional states, bounded neighborhood attention, implicit integration via Neural ODEs, and low-rank adapters for restarting reappearing clonotypes.
Comprehensive Evaluation Framework: The study separates literature-reported baselines from internally controlled temporal comparisons. It reports uncertainty for small external cohorts, adds calibration and threshold diagnostics, and visualizes latent clone trajectories and attention neighborhoods.

4. Results

The evaluation was conducted on lung cancer and thyroid cancer (THCA) datasets, alongside external validation on 17 cancer types.

Internal Temporal Comparisons: In matched patient-level splits, DynImmune-BERT achieved the highest performance.
- Lung Cancer AUC: 0.982 (vs. 0.963 for Latent ODE control and 0.949 for Time-aware Transformer).
- THCA AUC: 0.984 (vs. 0.969 for Latent ODE control).
- Ablation studies indicated that the Neural ODE core was the primary driver of accuracy gains, while event handling and adaptive pseudocounts improved robustness across random seeds.
External Validation: On external cohorts (e.g., THCA, GBM, Lung), the model showed high accuracy (e.g., 0.957 for Lung) but exhibited wide uncertainty intervals for subsets with fewer than 20 cases, highlighting the need for larger validation sets.
Comparison to Literature: While DynImmune-BERT reported higher AUCs than static baselines (e.g., BertTCR, DeepLION2) in the provided tables, the authors note these are cross-study comparisons with differing protocols and should be interpreted as context for prior performance ranges rather than direct head-to-head superiority.
Computational Cost: The full model requires approximately 122.4M parameters and 3.8 minutes per epoch on a single GPU, with inference latency of 13.5 ms per subject.

5. Significance and Claims

The paper claims that event-aware temporal modeling can complement strong static encoders when longitudinal repertoire structure is available. The significance of DynImmune-BERT lies in its ability to:

Bridge the gap between sparse, irregular biological data and continuous mathematical modeling without assuming mechanistic smoothness in clonal dynamics.
Provide a framework that explicitly handles the "disappearance and reappearance" of clones, which static models often miss.
Offer a diagnostic suite (trajectory visualization, calibration plots, threshold stability) that allows researchers to examine dynamic behavior alongside aggregate classification metrics.

The authors maintain a modest stance, acknowledging that small external cohorts and protocol differences require cautious interpretation. They emphasize that the method is most useful when repeated repertoire measurements are available and that the learned trajectories require prospective validation with clinical metadata (e.g., HLA, treatment) to confirm their correspondence to interpretable immune events.