Probabilistic Circuits for Irregular Multivariate Time Series Forecasting

1. Problem Statement

The paper addresses the challenge of probabilistic forecasting for Irregular Multivariate Time Series (IMTS). In real-world scenarios (e.g., healthcare, climate science), time series data is often:

• Irregular: Observations occur at non-uniform time intervals.
• Sparse: Many variables (channels) may be missing at specific timestamps.
• Multivariate: Multiple variables interact with complex dependencies.

Key Limitations of Existing Methods:

• Point Forecasting: Many models predict single values, failing to quantify uncertainty, which is critical for stochastic processes.
• Marginalization Inconsistency: State-of-the-art models like ProFITi can model complex dependencies but suffer from marginalization inconsistency. This means the predicted probability of a subset of variables changes arbitrarily depending on whether other variables are included in the query. This leads to contradictory forecasts (e.g., the probability of rain changes if you also ask about humidity), rendering them unreliable for high-stakes decision-making.
• Expressivity vs. Consistency Trade-off: Models that guarantee consistency, such as MOSES (Mixtures of Separable Flows), often struggle to capture complex joint dependencies in high-dimensional spaces due to the exponential scaling required for mixture components. They fail to learn distinct modes (e.g., bifurcations) effectively.

2. Methodology: CircuITS

The authors propose CircuITS (Circuits for Irregular Multivariate Time Series), a novel architecture based on Probabilistic Circuits (PCs). The core innovation is a structural guarantee of marginalization consistency while maintaining high expressivity.

A. Core Architecture: Recursive Sum-Product Network

CircuITS models the joint distribution $p(y | Q, X)$ (where $y$ are targets, $Q$ are query points, and $X$ are observations) using a hierarchical structure of Sum Nodes (mixtures) and Product Nodes (independencies).

• Leaf Nodes: Represent univariate or channel-specific distributions.
• Sum Nodes: Model mixtures (capturing multi-modal distributions).
• Product Nodes: Model factorization (capturing independence between variables).

Recursive Aggregation:
Instead of a full tensor product (which scales exponentially as $O(K^C)$), CircuITS uses a recursive approach:

1. Initialization: For the first channel, the state is a vector of $K$ leaf densities.
2. Recursion: To add channel $c$, the model computes the Cartesian product of the previous joint state and the new channel's leaf densities ($K^2$ terms).
3. Projection: A weighted sum (Sum Layer) projects these $K^2$ terms back to $K$ components. This prevents state explosion while allowing the model to learn correlations between the previous joint state and the new channel.
4. Final Output: The root node produces the final joint distribution as a weighted sum of the $K$ components.

B. Encoder Architecture

To handle irregular data and generate the circuit parameters dynamically:

• Observation Representation: Observations (time, channel, value) are embedded using sinusoidal time embeddings and learnable channel embeddings.
• Intra-Channel Aggregation: A Masked Cross-Attention mechanism aggregates observations for each channel into a fixed-size representation, preserving channel identity.
• Inter-Channel Interaction: A Self-Attention layer captures dependencies between channels.
• Parameter Generation: The encoder outputs:
  • Context Embeddings: For the leaf distributions (query-specific).
  • Circuit Weights: The mixing weights for Sum nodes ($W^{(c)}$) and the root ($w_{root}$). Crucially, these weights depend only on the observation history $X$, not on the query set $Q$. This structural constraint is the mathematical key to ensuring marginalization consistency.

C. Leaf Distributions

To ensure the leaves themselves are consistent and expressive:

• Univariate Marginals: Modeled using Conditional Deep Sigmoidal Flows (DSF), which approximate the Cumulative Distribution Function (CDF) via monotonic layers.
• Joint Dependence: A Gaussian Copula combines the univariate marginals. The correlation matrix is derived from the query embeddings, ensuring the dependence structure adapts to the context while remaining mathematically valid.

D. Theoretical Guarantee

The paper proves that CircuITS satisfies Marginalization Consistency (Kolmogorov extension theorem).

• Proof Logic: Because the circuit structure is fixed based on input channels and the Sum node weights depend only on $X$ (not $Q$), integrating out a variable (marginalization) simply removes the corresponding leaf term (replacing it with 1) without altering the weights of the remaining mixture. Thus, the marginal distribution derived from the joint is identical to the distribution predicted directly for the subset.

3. Key Contributions

1. CircuITS Architecture: A novel Sum-Product Network tailored for IMTS that structurally guarantees marginalization consistency, resolving the trade-off between expressivity and reliability.
2. Consistent Encoder: An encoder that generates circuit weights and leaf parameters from observation history, ensuring that predictions for subsets of variables remain consistent regardless of the query scope.
3. State-of-the-Art Performance: Demonstrated superior performance in both joint density estimation and marginal density estimation across four real-world datasets.
4. Theoretical Proof: Formal proofs establishing that the proposed architecture satisfies the requirements for a valid stochastic process.

4. Experimental Results

The model was evaluated on four datasets: USHCN (Climate), PhysioNet-2012, MIMIC-III, and MIMIC-IV (Medical).

• Metrics: Normalized Joint Negative Log Likelihood (njNLL) and Marginal NLL (mNLL).
• Baselines: Compared against ProFITi (inconsistent but expressive), MOSES (consistent but less expressive), and NeuralFlows (independent marginals).
• Performance:
  • Joint Density: CircuITS achieved the lowest (best) njNLL in the vast majority of scenarios, outperforming both ProFITi and MOSES.
  • Marginal Density: CircuITS also outperformed baselines in mNLL, whereas ProFITi often showed high mNLL errors due to its inconsistency.
  • Synthetic Task: In a synthetic "bifurcation" task (where independent channels split into two distinct modes), MOSES failed to capture the distinct branches even with 16 components, while CircuITS solved it with only 2 components per channel.
• Efficiency: CircuITS reduces computational complexity from $O((CN)^3)$ (full covariance) to $O(CN^3)$ by decoupling channels, though it introduces a dependency on the number of mixture components $K$.

5. Significance

• Reliability in Critical Domains: By guaranteeing marginalization consistency, CircuITS prevents contradictory forecasts. This is crucial for safety-critical applications like healthcare (e.g., predicting patient deterioration) and climate modeling, where inconsistent probabilities can lead to erroneous decisions.
• Handling Complexity: It successfully models complex, multi-modal joint distributions without the optimization difficulties faced by previous mixture-based consistent models.
• Generalizability: The architecture is flexible regarding the number of variables and query points, making it suitable for diverse real-world IMTS applications where data sparsity and irregularity are the norm.

In conclusion, CircuITS represents a significant advancement in probabilistic time series forecasting by unifying the theoretical rigor of probabilistic circuits with the practical flexibility required for irregular, multivariate data.