TARDis: Time Attenuated Representation Disentanglement for Incomplete Multi-Modal Tumor Segmentation and Classification

The paper proposes TARDis, a novel physics-aware framework that addresses incomplete multi-modal tumor segmentation by disentangling time-invariant anatomical features from time-dependent hemodynamic dynamics, enabling robust diagnosis even when temporal CT phases are missing due to clinical constraints.

The Gist

The Big Problem: The "Missing Page" in a Medical Story

Imagine a doctor trying to diagnose a tumor using a CT scan. In an ideal world, the doctor gets a "movie" of the patient's body. They see the tumor at four different moments in time:

1. Before the dye is injected (Non-contrast).
2. Just after the dye hits the arteries (Arterial phase).
3. A bit later when the dye fills the veins (Venous phase).
4. Much later when the dye settles (Delayed phase).

This "movie" is crucial because tumors behave differently than healthy tissue as the dye flows through them. It's like watching a sponge soak up colored water; the way it changes color tells you if it's a rock-hard tumor or a soft cyst.

The Reality Check:
In real hospitals, getting this full "movie" is often impossible.

• Radiation Safety: Doctors can't blast patients with too much radiation, so they might skip a phase.
• Timing Issues: Different hospitals have different clocks. What one hospital calls "Arterial," another might call "Late Arterial."
• The Result: The doctor often only has a few "frames" of the movie, or the frames are out of sync.

The Old Solution (and why it fails):
Current AI models treat these missing frames like empty boxes. If the "Arterial" box is empty, the AI just sees a blank space and panics. It tries to guess the missing info by looking at the other boxes, but it doesn't understand that these boxes are actually frames of a continuous story. It treats them as unrelated photos, leading to bad guesses.

---

The New Solution: TARDis (Time Attenuated Representation Disentanglement)

The authors created a new AI framework called TARDis (a clever play on the Doctor Who time machine). Instead of treating missing data as "empty," TARDis treats it as a missing page in a continuous book.

Here is how it works, using a simple analogy:

1. The Core Idea: Separating the "Skeleton" from the "Makeup"

The paper's big breakthrough is a hypothesis: A CT scan image is made of two distinct things mixed together:

• The Skeleton (Static): The actual shape and structure of the organ. This never changes, no matter when you scan it.
• The Makeup (Dynamic): The color changes caused by the dye flowing through. This changes constantly over time.

The Analogy: Imagine a person posing for a photo.

• Their bones and face shape are the Skeleton (Static).
• The lighting and makeup are the Makeup (Dynamic).
• If you take a photo in bright sunlight vs. dim light, the person's face (Skeleton) is the same, but the shadows (Makeup) change.

Old AI models try to memorize the whole photo. TARDis learns to separate the Face from the Lighting.

2. How TARDis Works (The Two-Path System)

TARDis uses a "Dual-Path" architecture to solve the missing data problem:

Path A: The Dictionary of Shapes (The Skeleton)

• What it does: It looks at any available scan (even just one) and extracts the pure anatomy.
• The Analogy: Imagine a library of "perfect bone structures." No matter what the lighting is, TARDis looks at the scan and says, "Ah, I see a kidney. I'll pull the 'Kidney Skeleton' from the library."
• Why it helps: Even if you only have a non-contrast scan (no dye), TARDis knows exactly what the tumor looks like structurally.

Path B: The Time Machine (The Makeup)

• What it does: It guesses when the scan was taken relative to the dye injection and predicts how the dye should look.
• The Analogy: This is a "Time Machine" (hence the name TARDis). If the doctor only has a scan from 30 seconds after injection, TARDis asks, "Okay, I have the Skeleton. If I know this is 30 seconds in, what does the 'Makeup' (dye flow) look like at this exact moment?"
• The Magic: Even if the "Arterial" scan is missing, TARDis can hallucinate (generate) what that missing frame should look like based on the physics of how blood flows. It fills in the blank pages of the story.

3. Putting It Back Together

Once TARDis has the Skeleton (from Path A) and the Predicted Makeup (from Path B), it combines them to create a perfect, complete picture of the tumor. It then uses this complete picture to slice out the tumor (Segmentation) or decide if it's cancer (Classification).

---

Why This is a Game-Changer

1. It's Robust to Missing Data
In tests, when other AI models were given only one scan (e.g., just the non-contrast one), they failed miserably, often missing the tumor entirely. TARDis, however, kept performing at a high level. It was like a detective who could solve a crime even if half the evidence was missing, because they understood the logic of the crime scene.

2. It Reduces Radiation
Because TARDis can work with incomplete scans, doctors might not need to order as many phases of CT scans. This means less radiation exposure for patients and faster appointments.

3. It Understands Physics, Not Just Patterns
Most AI just memorizes patterns (e.g., "Red pixels usually mean tumor"). TARDis understands the physics of blood flow. It knows that dye moves in a specific curve over time. This makes it much smarter and more reliable in real-world hospitals where things are messy and inconsistent.

Summary

TARDis is an AI that doesn't panic when data is missing. Instead of seeing a blank space, it sees a missing moment in a timeline. By separating the permanent shape of the body from the temporary flow of dye, it can "fill in the blanks" of a medical story, allowing for accurate tumor diagnosis even with fewer, safer scans.

Technical Summary

1. Problem Statement

The accurate segmentation and classification of tumors in contrast-enhanced Computed Tomography (CT) rely heavily on hemodynamic profiles (how contrast agents flow through tissue over time). However, real-world clinical practice faces two major challenges:

• Incomplete Modalities: Strict radiation dose limits and inconsistent acquisition protocols often force clinicians to skip specific contrast phases (e.g., Arterial, Venous, Delayed), leading to missing data channels.
• Temporal Mismatch: Even when phases are acquired, timing variations across different institutions and patients cause the same "phase" to represent different points on the contrast curve, creating severe domain shifts.

Limitations of Existing Methods: Current deep learning approaches typically treat missing phases as empty channels or independent, uncorrelated inputs. They fail to capture the inherent temporal continuity of hemodynamics, leading to suboptimal feature representation and performance degradation when data is sparse or inconsistent.

2. Methodology: TARDis Framework

The authors propose TARDis (Time Attenuated Representation Disentanglement), a physics-aware framework that redefines missing modalities as missing sample points on a continuous Time-Attenuation Curve (TAC) rather than missing independent channels.

Core Hypothesis

The Hounsfield Unit (HU) intensity $H(p, \tau)$ at a spatial location $p$ and time $\tau$ can be explicitly disentangled into two components:

1. Static Component ($H_s$): Time-invariant anatomical structure (tissue density).
2. Dynamic Component ($H_d$): Time-dependent perfusion (contrast enhancement).
   $$H(p, \tau) = H_s(p) + H_d(p, \tau)$$

Architecture

TARDis employs a shared encoder to process a flexible number of input modalities ($N$) and a dual-path disentanglement mechanism:

1. Modal-Agnostic Path (Anatomy):

   • Uses a Vector Quantized (VQ) mechanism with a learnable embedding dictionary.
   • Forces features to map to a finite, discrete set of "anatomical queries."
   • Goal: Extract a consistent, noise-free anatomical representation ($X_s$) that is invariant to the specific contrast phase or missing data.

2. Modal-Specific Path (Perfusion):

   • Uses a Hemodynamic Conditional Variational Autoencoder (HCVAE).
   • Conditioning: The generation of dynamic features ($X_d$) is conditioned on the extracted anatomy ($X_s$) and an estimated relative scan time ($\tau$).
   • Time Estimation: A lightweight regressor predicts a relative time scalar $\tau \in [0, 1]$ for each input modality.
   • Generative Inference: During inference with missing data, the model samples latent variables from a prior distribution conditioned on the known anatomy and estimated time to probabilistically reconstruct the missing dynamic enhancement features.

3. Adaptive Assembly:

   • An attention-based module adaptively weights and fuses the static ($X_s$) and dynamic ($X_d$) representations before decoding, allowing the network to focus on the most relevant information for each voxel.

Training Strategy

• Sequence-Based Optimization: Handles variable input sizes ($N$) by treating batches as collections of independent sequences rather than fixed tensors.
• Modality Dropout: Aggressively drops modalities during training to simulate clinical sparsity.
• Loss Functions: Includes consistency loss (aligning anatomy across phases), ranking loss (ordering time estimates), disentanglement loss (minimizing mutual information between static and dynamic spaces), and standard segmentation loss (Dice + Cross-Entropy).

3. Key Contributions

1. Physics-Aware Disentanglement: First framework to explicitly decouple CT signals into time-invariant anatomy and time-dependent perfusion, modeling missing data as gaps in a continuous curve rather than absent channels.
2. Dual-Path Generative Architecture: Combines VQ-VAE for robust structural extraction and HCVAE for probabilistic dynamic reconstruction, enabling the inference of missing hemodynamic features.
3. Flexible Input Handling: The architecture accepts an arbitrary number of input modalities without retraining, making it robust to varying clinical protocols.
4. Superior Performance in Sparsity: Demonstrates that the model can maintain high diagnostic accuracy even with extreme data sparsity (e.g., single non-contrast scans).

4. Experimental Results

The method was evaluated on three datasets:

• Changhai (CH) Dataset: Large-scale private abdominal CT (2,282 patients, 8,065 scans).
• C4KC-KiTS: Public kidney tumor dataset.
• BraTS 2018: Brain tumor MRI dataset (to test generalization to other modalities).

Key Findings:

• Classification: On the CH dataset, TARDis achieved an average Screening AUC of 0.979 and Subtype Diagnosis AUC of 0.923, significantly outperforming state-of-the-art (SOTA) methods like M2FTrans and mmFormer (which scored ~0.76 and ~0.93 respectively).
• Segmentation: TARDis maintained a Dice score of 0.859 using only non-contrast scans, whereas baselines dropped significantly (e.g., RFNet ~0.80, mmFormer ~0.77).
• Robustness: Unlike baselines that showed jagged performance drops when specific phases (Arterial/Venous) were missing, TARDis exhibited smooth performance curves across all modality combinations.
• Generalization: The method successfully generalized to MRI (BraTS), achieving the highest Dice scores for Whole Tumor (0.860) and Enhancing Tumor (0.564), proving the disentanglement paradigm is modality-agnostic.

5. Significance and Impact

• Clinical Utility: TARDis reduces the dependency on complete multi-phase acquisition protocols. This allows for reduced radiation exposure (by skipping phases) and shorter scan times without compromising diagnostic precision.
• Data Harmonization: By modeling the underlying physics of contrast flow, the method mitigates domain shifts caused by inconsistent acquisition protocols across different hospitals.
• Future Directions: The "continuous-time disentanglement" paradigm offers a new direction for low-dose diagnostics and cross-center data harmonization, moving beyond simple data imputation to physically grounded feature reconstruction.

In summary, TARDis represents a paradigm shift from treating missing modalities as a data engineering problem to a physics-informed representation learning problem, achieving state-of-the-art robustness in tumor segmentation and classification under incomplete data conditions.