🤖 AI

TRACE: A Temporal Conditional Estimation for Multimodal Time Series Foundation Models

The paper proposes TRACE, a conditional estimation paradigm for multimodal time series foundation models that systematically infers incomplete target modalities from available auxiliary ones to address temporal misalignment and partial modality missingness, demonstrating superior robustness and performance across healthcare and affective computing benchmarks compared to existing approaches.

Original authors: Ziwen Kan, Yishuo Chen, Kecheng Li, Andrew Wen, Xiaomeng Wang, Liwei Wang, Jihao Duan, Song Wang, Hongfang Liu, Tianlong Chen

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ziwen Kan, Yishuo Chen, Kecheng Li, Andrew Wen, Xiaomeng Wang, Liwei Wang, Jihao Duan, Song Wang, Hongfang Liu, Tianlong Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex puzzle, but every time you look at the picture, some pieces are missing, and the pieces you do have are scattered across the table at different times. This is the daily reality for computers trying to analyze multimodal time series data (like a patient's heart rate, blood pressure, and doctor's notes, or a video with audio, text, and visuals).

In the real world, sensors fail, notes are written late, and data arrives at different speeds. Existing computer models usually try to fix this by guessing the missing pieces using simple math (like drawing a straight line between two known points) or just ignoring the gaps. The paper argues this is like trying to complete a puzzle by guessing the color of a missing piece based only on the piece right next to it, ignoring the rest of the picture.

Here is the simple breakdown of what the paper, TRACE, proposes:

1. The Problem: The "Broken Puzzle"

Think of a patient's health data as a story told by three different narrators:

Narrator A (Heart monitor) speaks every second.
Narrator B (Doctor's notes) speaks only when the doctor writes something.
Narrator C (X-rays) speaks only when a scan is taken.

Sometimes, Narrator B and C go silent for a long time. Old computer models (like FuseMoE) try to fill in the silence by making a "best guess" based on the last thing they heard. But this often creates a distorted story because it doesn't listen to what Narrator A is saying right now to help fill in the gaps for B and C.

2. The Solution: TRACE (The "Smart Detective")

The authors propose TRACE, which stands for Temporal Conditional Estimation.

Instead of just guessing the missing piece in isolation, TRACE acts like a detective who uses all available clues to reconstruct the missing parts of the story.

The Analogy: Imagine you are trying to remember what a friend said during a loud party, but you missed a few sentences. Instead of just guessing random words, you listen to what the friend said before and after, and you also listen to what their other friends said at the same time to fill in the blanks.
How it works: TRACE uses a technique called Diffusion. Think of this like a sculptor who starts with a block of clay (random noise) and slowly chips away the noise, guided by the "clues" from the other narrators, until a clear, realistic shape emerges. It doesn't just draw a line; it imagines the missing data based on the context of everything else that is present.

3. The Two-Step Process

TRACE works in two distinct stages, like a two-person team:

Step 1: The Reconstruction Team (Conditional Diffusion)
Before the team tries to solve the final problem, they first work together to "repair" the broken data. If the heart monitor is missing a chunk of data, the team looks at the doctor's notes and the X-rays to probabilistically (mathematically) reconstruct what that missing chunk likely looked like. They don't just fill it with a static number; they create a "cloud of possibilities" that fits the context.
Step 2: The Decision Team (Mixture-of-Experts)
Once the data is "repaired," a second team (called a Mixture-of-Experts) looks at the now-complete picture to make a prediction (like predicting if a patient will be discharged or if a video is happy or sad). Because the data is cleaner and more accurate, this team makes better decisions.

4. What They Found (The Results)

The researchers tested TRACE on two types of "puzzles":

Sentiment Analysis (CMU-MOSI/MOSEI): Determining if a video clip is positive or negative based on text, audio, and video.
Healthcare Prediction (MIMIC-IV): Predicting patient outcomes (like mortality or length of stay) based on vital signs, notes, X-rays, and ECGs.

The Verdict:

Better Accuracy: TRACE consistently outperformed previous models. In the healthcare tests, it was better at predicting patient outcomes.
Resilience: The more data that was missing, the more TRACE shined. While other models fell apart when data was sparse (missing 30% or more), TRACE kept its cool.
Closer to Reality: The paper shows that the "repaired" data TRACE creates is much closer to the "true" reality (what would have happened if all sensors worked perfectly) than the simple guesses made by other models.

Summary

In short, TRACE changes the way computers handle missing data. Instead of saying, "I'll just fill this gap with a simple average," it says, "Let me look at everything else that is happening right now to intelligently reconstruct what is missing." This leads to a much clearer picture of the past, which helps the computer make much smarter predictions about the future.

Technical Summary: TRACE – A Temporal Conditional Estimation for Multimodal Time Series Foundation Models

1. Problem Statement

Time series foundation models (TS-FMs) aim to learn generalizable temporal representations adaptable to diverse downstream tasks. However, real-world multimodal time series data, particularly in domains like healthcare and affective computing, frequently suffer from temporal misalignment and partial modality missingness. In these scenarios, different modalities are observed at heterogeneous time scales, or specific modalities are partially absent due to irregular sampling or sensor failures.

Existing approaches typically address these issues through naive imputation (e.g., zero-filling, nearest-value filling) or masking strategies prior to fusion. The authors argue that these methods fail to account for cross-modal dependencies. By deterministically completing missing values, existing models collapse conditional uncertainty, potentially distorting temporal dynamics and biasing downstream fusion. Furthermore, many current TS-FMs treat missingness as a preprocessing step rather than a latent temporal variable to be conditionally estimated, limiting their robustness under severe sparsity.

2. Methodology: TRACE

The authors propose TRACE (Temporal Conditional Estimation), a paradigm that reformulates how sparse and event-driven observations are handled. Instead of deterministic completion, TRACE treats missing modality inputs as latent variables to be conditionally estimated from available auxiliary modalities before fusion. The architecture operates as a two-stage pipeline:

Stage 1: Multimodal Conditional Diffusion

This stage performs probabilistic signal-level estimation of missing components.

Cross-Modal Context Construction: For a target modality $m$ , the model aggregates observed components from auxiliary modalities ( $M \setminus \{m\}$ ) using a Mixture-of-Experts (MoE) inspired gating mechanism. A gating function assigns importance scores to each auxiliary modality based on their observed features, producing a weighted cross-modal context $x^{(m)}_c$ .
Conditional Diffusion Process: TRACE employs a conditional denoising diffusion model (based on DDPM) to estimate the unobserved components $x^{(m)}_U$ $x_{U}^{(m)}$ .
- The process conditions on the observed part of the target modality $x^{(m)}_O$ and the aggregated cross-modal context $x^{(m)}_c$ .
- A forward noising process adds Gaussian noise only to the missing components, while observed entries remain fixed.
- A reverse denoising process, parameterized by a neural network $\epsilon_\theta$ , learns to predict the noise and reconstruct the missing values probabilistically, conditioned on the multimodal context.
Training: The diffusion network is trained using a self-supervised masking strategy (inspired by CSDI), where a subset of observed values is masked and treated as imputation targets, conditioned on the remaining observations and cross-modal context.

Stage 2: Multimodal Fusion and Prediction

Once the missing components are probabilistically estimated, the completed modality representations are passed to a downstream fusion layer.

MoE Fusion: TRACE adopts the MoE fusion layer from FuseMoE. This layer uses a sparse gating mechanism to route modality-specific representations to a shared pool of feed-forward experts, aggregating them into a unified embedding for task-specific prediction heads.
Regularization: An entropy-based regularization term is added to the loss to encourage balanced expert utilization and prevent expert collapse.

3. Key Contributions

Problem Identification: The authors identify that deterministic completion of missing observations prior to fusion is a critical bottleneck in TS-FM pipelines, often leading to misaligned or degraded representations.
TRACE Paradigm: They propose TRACE, a conditional estimation paradigm for multimodal TS-FMs. It is instantiated via a diffusion-based mechanism for probabilistic signal-level estimation, explicitly leveraging complementary multimodal context to infer missing data.
Empirical Validation: Through extensive experiments, the authors demonstrate that TRACE consistently improves downstream performance across diverse datasets and missingness patterns, establishing temporal conditional estimation as an effective system-level leverage point.

4. Experimental Results

TRACE was evaluated on three benchmarks spanning sentiment analysis and clinical prediction: CMU-MOSI, CMU-MOSEI, and MIMIC-IV.

Sentiment Analysis (CMU-MOSI/MOSEI): TRACE outperformed state-of-the-art baselines (including MulT, MAG, TFN, and FuseMoE) across all metrics (MAE, Accuracy, Correlation, F1). Notably, it achieved lower prediction error and higher classification accuracy with reduced variance, indicating more stable cross-modal estimation.
Clinical Prediction (MIMIC-IV): On tasks including 48-hour in-hospital mortality (48-IHM), length-of-stay (LOS), and phenotype classification (25-PHE), TRACE achieved the strongest average performance in AUROC and F1 scores, outperforming baselines like FuseMoE, MISTS, and HAIM (in most metrics).
Robustness to Missingness: In ablation studies and controlled missingness experiments, TRACE demonstrated superior robustness as missing rates increased. While naive imputation (FuseMoE) led to degraded representations under high missingness (e.g., 30-50%), TRACE maintained representations closer to the "oracle" (fully observed) ground truth.
Ablation Studies:
- Conditional Diffusion: Removing the cross-modal conditioning or using unconditional diffusion significantly hurt performance, confirming the necessity of leveraging auxiliary modalities.
- MoE Routing: Disabling adaptive routing (using fixed weights) reduced performance, highlighting the importance of data-dependent expert selection.
- Expert Count: Performance peaked at 5 experts in the diffusion module, with diminishing returns or degradation at higher counts due to sparse supervision.
Efficiency: TRACE requires approximately 2.7M parameters. While inference is slower than deterministic baselines due to the diffusion steps, reducing steps from 50 to 10 yields a 4x speedup with minimal performance loss, remaining tractable for clinical settings where strict real-time response is not always required.

5. Significance and Claims

The paper claims that TRACE addresses a fundamental limitation in current multimodal time series modeling: the treatment of missing data as a static value-filling problem rather than a dynamic, conditional estimation task.

Paradigm Shift: The work shifts the focus from "filling gaps" to "estimating latent variables," preserving the probabilistic structure of missing data and leveraging cross-modal dependencies.
Representation Fidelity: The authors argue that the primary benefit of TRACE is not just improved pointwise imputation accuracy, but the preservation of meaningful temporal representations in the latent space under severe missingness. This allows downstream fusion layers to operate on more faithful data, leading to better robustness.
Generalizability: While the current instantiation focuses on settings where modalities are partially observed (within-modality missingness), the framework is designed to be a general paradigm for TS-FMs. The authors note that for entirely absent modalities, the system relies on the backbone's ability to route around missing inputs, but the core contribution lies in refining representations when data is present but sparse.
Limitations: The authors modestly acknowledge that for specific tasks heavily reliant on domain-specific inductive biases (e.g., HAIM on phenotype classification), specialized architectures may still outperform general frameworks. They suggest that bridging this gap via lightweight domain priors within TRACE is a direction for future work. Additionally, they note that the current scope targets partial observation rather than the generation of entirely absent modalities.

In conclusion, TRACE demonstrates that explicit temporal conditional estimation via diffusion is a superior strategy to naive imputation for building robust, generalizable multimodal time series foundation models.