🤖 machine learning

StreamPhy: Streaming Inference of High-Dimensional Physical Dynamics via State Space Models

StreamPhy is an end-to-end framework that enables efficient, real-time, and accurate inference of high-dimensional physical dynamics from irregular sparse measurements by integrating a data-adaptive encoder, a structured state-space model, and an expressive FT-FiLM decoder, outperforming existing methods in both accuracy and inference speed.

Original authors: Panqi Chen, Yifan Sun, Shikai Fang, Xiao Fu, Lei Cheng

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Panqi Chen, Yifan Sun, Shikai Fang, Xiao Fu, Lei Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to watch a high-definition movie of a stormy ocean or a swirling tornado, but your TV screen is broken. You only get to see a few scattered pixels at random times, and sometimes the pixels you see are in completely different spots than the ones you saw a second ago. Your goal is to figure out what the entire movie looks like in real-time, filling in all the missing parts perfectly.

This is the problem StreamPhy solves. It is a new computer system designed to predict how complex physical things (like water, sound, or air) move and change, even when the data coming in is messy, sparse, and irregular.

Here is how it works, broken down into three simple parts using everyday analogies:

1. The "Smart Translator" (The Observation Encoder)

The Problem: Usually, computers expect data to come in neat, orderly rows (like a spreadsheet). But in the real world, sensors might fail, or a drone might only scan a few spots. The data arrives as a chaotic jumble of "here is a temperature reading at this spot, now here is a wind speed at a totally different spot."

The Solution: StreamPhy has a special translator. Think of it like a human interpreter at a busy airport. Even if passengers (data points) arrive in random groups, speak different languages, and are scattered around the terminal, the interpreter can instantly organize them into a clear, structured message. This part of the system takes those messy, scattered measurements and turns them into a clean "summary" that the computer can understand, no matter how irregular the input is.

2. The "Memory-Keeping Stream" (The State Space Model)

The Problem: To predict the future, you need to remember the past. But if you try to remember every single detail of a long history, your brain (or computer memory) gets overloaded. Also, if the time between observations is weird (sometimes 1 second, sometimes 10 minutes), standard memory systems get confused.

The Solution: StreamPhy uses a technique called a State Space Model (SSM). Imagine a highly efficient librarian who doesn't keep every single book in the library. Instead, they keep a single, evolving "summary note" that captures the essence of everything that has happened so far.

This librarian is trained to compress the history of the storm or the ocean into a tiny, compact note.
Crucially, this note updates instantly whenever a new piece of data arrives, even if that data comes at a weird time.
This allows the system to "remember" the long-term patterns of the physical world without needing to store terabytes of old data.

3. The "Magic Paintbrush" (The FT-FiLM Decoder)

The Problem: Once the computer has its "summary note" of the past, it needs to draw the full picture of the current moment. Standard methods are like trying to paint a masterpiece using only a few thick, blocky strokes. They often miss the fine details or the smooth curves of the physical world.

The Solution: StreamPhy uses a new tool called FT-FiLM. Think of this as a smart, shape-shifting paintbrush.

Instead of just painting a static picture, this brush can stretch, shrink, and change its color based on the "summary note" from the librarian.
It allows the system to paint a smooth, continuous, high-definition field (like a perfect ocean surface) from the tiny summary note.
The paper claims this brush is mathematically "smarter" and more flexible than older brushes (called Functional Tucker models), allowing it to capture complex, swirling movements that other methods miss.

Why is this a big deal?

The authors tested StreamPhy against other top-tier methods (like "Diffusion models," which are like trying to guess a picture by slowly erasing noise from a blank canvas).

Speed: StreamPhy is 20 to 100 times faster than these other methods. While others might take seconds or minutes to guess the next frame, StreamPhy does it almost instantly.
Accuracy: It was 48% more accurate at reconstructing the full picture, even when the data was very sparse (like only seeing 1% of the ocean).
Real-Time: Unlike other methods that need to wait for the entire movie to finish before they can start guessing, StreamPhy works live. It watches the stream of data as it comes in and updates the picture instantly.

In Summary

StreamPhy is like a super-fast, super-smart weather forecaster who can look at a few scattered, broken thermometers and instantly draw a perfect, high-definition map of the entire storm, updating it every second as new data arrives. It does this by translating messy data, compressing history into a tiny memory note, and using a flexible "magic brush" to paint the full picture.

Technical Summary: StreamPhy

Problem Statement

The paper addresses the fundamental challenge of inferring the evolution of high-dimensional, multi-modal physical fields (e.g., spatio-temporal data) from irregular, sparse measurements in real-time. Existing approaches face significant limitations:

Diffusion-based models: Typically operate in offline settings, require full temporal observations for training and inference, and incur high computational costs due to iterative sampling.
Functional Tensor methods: Often designed for batch processing rather than streaming, rely on implicit neural representations (INRs) that fit specific data but lack generalization to future data, and frequently require memory replay for online updates.
General limitations: Many methods struggle with the "curse of dimensionality," irregular time intervals, and incomplete spatial coverage inherent in real-world scientific sensing (e.g., environmental monitoring, fluid dynamics).

The objective is to perform streaming inference of a continuous field $y_t(i): \mathbb{R}^K \to \mathbb{R}$ from a time-ordered stream of irregular observations $O = \{O_{t_m}\}_{m=1}^M$ , where observations arrive sequentially and require online updates without access to future data.

Methodology: StreamPhy Framework

The authors propose StreamPhy, an end-to-end framework built upon HiPPO-based State Space Models (SSMs). The framework integrates three core components to handle irregular inputs and generate continuous fields:

1. Data-Adaptive Observation Encoder

To handle observation sets $O_t$ with varying cardinality and irregular spatial patterns, the authors design an attention-based encoder:

Functional Tensor Representation: Continuous spatial coordinates are mapped to latent embeddings using learnable functions (inspired by INRs) parameterized by frequency vectors.
Cross-Attention Aggregation: Instead of simple averaging, a learnable query embedding $q$ interacts with the latent representations of the observation set via a cross-attention mechanism. This adaptively selects informative observations, producing a structured latent representation $z_t$ independent of the number of samples.
Robustness: A stochastic masking strategy is employed during training to simulate non-uniform sampling patterns, ensuring the encoder learns robust representations.

2. HiPPO-Based State Space Model (SSM)

The core temporal evolution is modeled using a structured SSM, specifically leveraging HiPPO (High-Order Polynomial Projection Operators) theory:

Continuous-to-Discrete Mapping: The continuous-time dynamical system is discretized to handle irregular time intervals $\Delta t$ .
Augmented State Transition: The standard SSM input is replaced by the aggregated observation embedding $z_t$ . The state update is formulated as:
$X_t = \bar{A}X_{t-1} + \bar{b} \circ z_t$
where $\circ$ denotes the outer product, creating an augmented state $X_t$ that tracks the evolution of the observation features.
Long-Range Dependency: The HiPPO structure (specifically HiPPO-LegS) allows the model to compress historical information into a finite-dimensional latent state, enabling efficient online updates and long-range dependency capture.

3. Functional Tensor Feature-wise Linear Modulation (FT-FiLM) Decoder

To reconstruct the full continuous field from compact latent states, the authors introduce a novel decoder:

Mechanism: FT-FiLM extends the FiLM (Feature-wise Linear Modulation) concept to the functional tensor domain. It generates feature-wise scaling ( $\Gamma_t$ ) and shifting ( $\beta_t$ ) parameters conditioned on both the latent state $X_t$ and the observation embedding $z_t$ .
Reconstruction: The field value at any coordinate $i_n$ is computed as:
$y_t(i_n) = f_\phi(\Gamma_t u_n + \beta_t)$
where $u_n$ is the functional representation of the coordinate, and $f_\phi$ is a learnable readout function.
Expressivity: The authors prove (Theorem 1) that FT-FiLM is strictly more expressive than the standard Functional Tucker Model (FTM). While FTM is limited by its rank constraints, FT-FiLM's closure coincides with the entire space of continuous functions $C(\Omega)$ , allowing it to model complex dynamics more effectively.

Key Contributions

StreamPhy Framework: An end-to-end architecture for streaming inference of high-dimensional physical dynamics from irregular sparse data, combining an adaptive encoder, HiPPO-SSM, and FT-FiLM.
Theoretical Expressivity: A proof demonstrating that FT-FiLM offers a richer function class than functional Tucker models, removing rank-induced approximation barriers.
Efficiency and Accuracy: The framework achieves online inference with significantly lower computational complexity ( $O(T)$ ) compared to diffusion-based methods ( $O(T^4)$ ), while maintaining high accuracy under challenging sampling patterns.

Experimental Results

The method was evaluated on three physical systems: Turbulent Flow, Ocean Sound Speed, and Active Matter, under both Uniform and Slab (non-uniform) sampling patterns with observation ratios as low as 1%.

Accuracy: StreamPhy consistently outperformed state-of-the-art baselines (including diffusion-based SDIFT, online tensor OFTD, and attention-based Senseiver). It achieved at least a 48% improvement in Variance-scaled Root Mean Squared Error (VRMSE) compared to the best baselines.
Efficiency: StreamPhy demonstrated inference speeds 20–100× faster than diffusion-based methods (e.g., SDIFT). For instance, on the Turbulent Flow dataset, StreamPhy took ~0.04s per record compared to ~5.14s for SDIFT.
Robustness: The model maintained high accuracy even under "Slab sampling" (spatially concentrated measurements), a scenario where baselines like SDIFT showed significant performance degradation.
Ablation Studies: Removing the SSM component caused drastic performance drops, confirming its necessity for temporal modeling. Replacing FT-FiLM with FTM also degraded performance, validating the expressivity gains.

Significance and Claims

The paper claims that StreamPhy provides a scalable and efficient solution for real-time streaming inference in engineering and scientific applications where data is sparse and irregular. By integrating structured state space models with expressive functional tensor decoding, the framework overcomes the limitations of offline diffusion models and batch-oriented tensor methods.

The authors emphasize that their approach enables memory-efficient online updates and continuous-field generation without requiring full temporal observations. They position StreamPhy as a significant step forward for applications such as environmental sensing, fluid dynamics, and structural health monitoring, where real-time assimilation of streaming data is critical.

Note: The authors acknowledge limitations, specifically the lack of explicit incorporation of physical laws (physics-informed constraints) and the predefined nature of the SSM transition matrix $A$ , suggesting future work in integrating domain-specific priors.