🤖 AI

PAMF: Prior-Aware Multimodal Fusion for Incomplete Time Series Data

The paper introduces PAMF, a novel multimodal time-series framework that addresses both within-modality and modality-level missingness by explicitly estimating missing data through prior-aware flow matching and coupling imputation with downstream tasks via weight sharing to achieve superior performance on healthcare benchmarks.

Original authors: Ziwen Kan, Wugeng Zheng, Tianlong Chen, Song Wang

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ziwen Kan, Wugeng Zheng, Tianlong Chen, Song Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex medical mystery, like figuring out if a patient is having a heart attack or how well they are sleeping. To do this, doctors usually look at several different "streams" of information at once: an ECG for the heart, a breathing monitor, brain waves, and muscle sensors. This is called multimodal data.

However, in the real world, these sensors are messy. Sometimes a wire pops off (a whole stream of data disappears), and sometimes a sensor just glitches for a few seconds (a few data points are missing).

The paper introduces a new system called PAMF (Prior-Aware Multimodal Fusion) designed to fix these broken data streams so doctors can still make accurate diagnoses. Here is how it works, using simple analogies:

The Two Types of "Missing" Data

The authors realized that missing data comes in two very different shapes, and you can't fix them with the same tool:

The "Glitch" (Within-Modality Missing): Imagine your heart monitor is working, but for 5 seconds, the line goes flat because of a loose wire. The rest of the data is there; just a small chunk is gone.
The "Blackout" (Modality-Level Missing): Imagine the patient refuses to wear the breathing sensor, or the machine breaks entirely. Now, you have zero breathing data for the whole night.

Most old computer programs treated these two problems the same way, or they just ignored the missing parts. PAMF treats them differently.

How PAMF Works: The "Smart Detective" Approach

1. The "Prior-Aware" Starter (Knowing Where to Begin)

Imagine you are trying to guess what a missing sentence in a story says.

If a word is missing in the middle of a sentence (The Glitch), you look at the words immediately before and after it to guess the missing word.
If the entire paragraph is missing (The Blackout), looking at the previous sentence doesn't help much. Instead, you look at the general style of the book or what other characters are doing to guess what might be in that paragraph.

PAMF does exactly this. It uses a mathematical engine called Flow Matching (think of it as a very efficient "time-travel" machine that reconstructs the past).

For Glitches, it starts its guess using the immediate neighbors (the data right before and after the gap).
For Blackouts, it starts its guess using the average behavior of similar patients or the other sensors that are working.

This "starting point" is called a Prior. By picking the right starting point for each type of missing data, the system doesn't have to wander blindly; it starts much closer to the truth.

2. The "Shared Brain" (Weight Sharing)

Usually, computer programs have two separate steps:

Step A: Fix the broken data.
Step B: Use the fixed data to make a diagnosis.

The problem is that Step A doesn't know what Step B cares about. It might fix the data perfectly for a human to read, but not in a way that helps the computer diagnose a heart attack.

PAMF connects these two steps. It uses a Shared Brain (Weight Sharing). The part of the system that learns how to diagnose the patient also helps the part that fixes the data.

Analogy: Imagine a detective (the diagnosis part) and a forensic artist (the data-fixer part). In old systems, they worked in different rooms. In PAMF, they sit at the same desk. The detective tells the artist, "I need to see the shape of the shoe print clearly to catch the suspect," so the artist focuses on sharpening the shoe print rather than just making the whole picture look pretty.

3. The "Three-Phase Training"

The system learns in three stages to get this right:

Warm-up: It learns to diagnose using the messy, broken data first. This forces it to learn how to handle missing pieces.
Reconstruction: It uses the knowledge from the first step to learn how to "fill in the blanks" of the missing data, using the smart starting points mentioned above.
Final Polish: It takes the now "repaired" data and runs the diagnosis one last time to get the best possible result.

Why It's Better (The Results)

The authors tested PAMF on four real-world medical datasets (sleep studies, heart monitors, and wearable activity trackers). They compared it against other smart systems that try to handle missing data.

The Result: PAMF consistently won. It was better at diagnosing patients whether the data had small glitches or entire sensors missing.
Speed: Because it uses "Flow Matching" instead of older, slower methods (like Diffusion models), it fixes the data much faster—like taking a direct highway instead of driving through every side street.

Summary

PAMF is a new way for computers to handle messy medical data. Instead of ignoring missing sensors or guessing blindly, it:

Knows the difference between a small glitch and a total sensor failure.
Uses the right "clues" (priors) to start guessing the missing data.
Lets the diagnosis goal guide the data repair process.

The result is a system that can look at incomplete, broken medical records and still figure out what is wrong with the patient, just as if the data were perfect.

Technical Summary: PAMF (Prior-Aware Multimodal Fusion for Incomplete Time Series Data)

Problem Definition

Multimodal time series analysis in healthcare faces a critical challenge: pervasive missing data that degrades downstream predictive performance. The paper identifies two structurally distinct patterns of missingness that existing methods fail to address simultaneously:

Within-modality missing: Individual time points are absent within an otherwise observed modality (e.g., a sensor dropout during a recording).
Modality-level missing: An entire modality is unavailable (e.g., a patient cannot undergo a specific test like ECG).

Existing approaches typically fall into two categories, both with limitations:

Missing-Aware Fusion: Methods like FuseMoE, FlexMoE, and Maestro handle missingness by using masks, gating mechanisms, or learned placeholder embeddings. However, they do not explicitly reconstruct the missing signal values, often treating missingness as a representation robustness issue rather than a signal recovery problem.
Explicit Imputation: Methods like CSDI and SSSD use generative models (e.g., diffusion) to reconstruct missing values. However, they typically treat all missingness uniformly, ignoring the structural differences between within-modality and modality-level gaps. Furthermore, imputation is often decoupled from the downstream task, meaning the imputed signals are not optimized for the specific prediction goal.

Methodology: PAMF Framework

The authors propose PAMF, a unified framework that explicitly handles both missingness patterns while coupling imputation with downstream prediction. The framework consists of four main components: an imputation encoder ( $h_{imp}$ ), a fusion encoder ( $h_{fus}$ ), a flow matching module ($ffm$), and a downstream head ( $f_{ds}$ ).

1. Prior-Aware Flow Matching

Instead of starting the generative process from a fixed, domain-agnostic Gaussian distribution (as in standard diffusion models), PAMF utilizes Flow Matching with prior-aware initialization. This allows the source state ( $x_0$ ) to be designed based on the specific type of missingness, reducing the transport distance to the target data distribution.

For Within-Modality Missing: The initialization leverages local temporal continuity. The source state is constructed using a convex combination of the nearest observed temporal neighbor and the per-channel batch mean. The weight decays exponentially with the temporal gap, ensuring the starting point remains within a plausible local observation range.
For Modality-Level Missing: Since same-modality evidence is absent, the initialization relies on cross-modal context. The source state is derived from the batch mean or context-similar samples, conditioned on the representations of the observed modalities.

The flow matching module learns a conditional velocity field $v_\theta$ that transports this structured source state $x_0$ to the target signal $x_1$ along straight ODE paths, conditioned on the imputation context $c_{imp}$ .

2. Weight Sharing and Three-Phase Training

To address the disconnect between imputation and downstream tasks, PAMF employs a weight-sharing mechanism and a three-phase training strategy:

Phase 1 (Downstream Warmup): The fusion encoder ( $h_{fus}$ ) and downstream head are trained on raw, incomplete data using a classification loss. This forces the encoder to learn representations robust to missingness.
Phase 2 (Flow Matching Training): The imputation encoder ( $h_{imp}$ ) is initialized by copying weights from $h_{fus}$ . The flow matching module is then trained to impute missing signals conditioned on $c_{imp}$ (which encodes the task-relevant context learned in Phase 1). This transfers task-relevant structure into the imputation process.
Phase 3 (Joint Fine-tuning): The imputation branch is frozen to generate completed data ( $x_{complete}$ ). The fusion encoder and downstream head are then fine-tuned on this completed data to refine the final predictive performance.

Key Contributions

Unified Framework for Dual Missingness: PAMF is the first method to simultaneously address both within-modality and modality-level missingness by explicitly reconstructing missing signals with type-specific priors.
Prior-Aware Initialization: The authors introduce a flow-matching-based generative module where the source state is initialized with structural priors (temporal neighbors for local gaps, cross-modal statistics for global gaps) rather than fixed noise. This design reduces inference overhead compared to diffusion-based alternatives.
Task-Guided Imputation: Through weight sharing and a three-phase training schedule, the framework connects imputation and classification. Task-relevant representations learned for downstream prediction guide the imputation process, ensuring the recovered signals are informative for the specific task.

Experimental Results

The authors evaluated PAMF on four physiological benchmarks: Sleep-EDF (sleep staging), PTB-XL (ECG diagnosis), PPG-DaLiA (wearable activity recognition), and Chapman-Shaoxing (ECG rhythm).

Performance: PAMF achieved the strongest overall downstream performance across all datasets and missingness settings (both 20% within-modality and modality-level missing). It consistently outperformed strong baselines like FuseMoE, FlexMoE, Maestro, and MIRA in terms of Macro-F1 and AUROC.
Robustness: Under mixed missingness (simultaneous within-modality and modality-level gaps) and high-missingness stress tests (50% missing rate), PAMF maintained superior performance, whereas baselines like Maestro showed significant degradation.
Efficiency: Compared to diffusion-based imputation baselines (CSDI, SSSD), PAMF achieved higher Macro-F1 scores with significantly lower inference latency (fewer function evaluations) due to the efficiency of flow matching and the informative nature of the prior-aware initialization.
Ablation Studies: Removing any key component (imputation, prior-aware initialization, context conditioning, or weight sharing) resulted in performance drops, confirming that the full pipeline is necessary for optimal results.

Significance and Claims

The paper claims that the primary obstacle in multimodal time series analysis is not just the presence of missing data, but the structural heterogeneity of that missingness and the decoupling of signal recovery from downstream tasks.

PAMF demonstrates that:

Explicit reconstruction of missing signals is superior to merely tolerating missing inputs via masking or placeholders.
Distinguishing missingness types is crucial; treating all missing data uniformly (e.g., with a single Gaussian prior) is suboptimal.
Coupling imputation with downstream learning via weight sharing allows the model to learn representations that are specifically adapted to the prediction task, rather than generating generic "plausible" signals.

The authors position PAMF as a research-stage method that improves robustness in healthcare settings where sensor failures or incomplete examinations are common. They acknowledge limitations, noting that the method requires sufficient training data to learn reliable temporal and cross-modal structures and that explicit imputation carries risks of generating plausible but incorrect physiological signals if not validated.