ABC: Any-Subset Autoregression via Non-Markovian Diffusion Bridges in Continuous Time and Space

1. Problem Statement

The paper addresses the fundamental challenge of generating continuous-time, continuous-space stochastic processes (e.g., videos, weather forecasts, stock prices) conditioned on partial observations. These observations may be:

• Irregularly sampled: Data points are not uniformly spaced in time (e.g., sporadic GPS readings).
• Non-causal: Conditioning on future states (e.g., infilling a video given the first and last frames).
• Arbitrary subsets: The model must handle any combination of past, present, and future observations.

Limitations of Existing Approaches:

• Noise-to-Data Diffusion: Standard diffusion models (e.g., DDPM) map Gaussian noise to data. This ignores the physical continuity of time series, where adjacent states are highly similar. Starting from uninformative noise leads to poor structural preservation and physically implausible transitions.
• Fixed Volatility Schedules: Existing diffusion bridges often use an auxiliary time variable $\tau \in [0, 1]$ independent of physical time. They inject the same amount of noise regardless of the physical time elapsed between states. This fails to capture the correct dynamics (e.g., a 1-second gap should have less volatility than a 1-hour gap).
• Markovian Assumptions: Most models assume the next state depends only on the immediate previous state, ignoring long-range temporal dependencies and path history.
• Discrete Time: Many autoregressive models operate on fixed grids, making them ill-suited for irregular sampling or continuous-time inference.

2. Methodology: The ABC Framework

The authors propose ABC (Any-Subset Autoregressive Models via Non-Markovian Diffusion Bridges), which unifies diffusion models and autoregressive generation in continuous time and space.

Core Insight

Instead of chaining multiple short bridges or mapping noise to data, ABC models the entire process as one continual Stochastic Differential Equation (SDE) where the time variable and intermediate states correspond directly to physical time and process states.

Key Technical Components

A. Data-Generating Dynamics via Change-of-Measure

• Base Process: The authors define a data-independent base process (an Ornstein-Uhlenbeck process) $dX_t = -a(t)X_t dt + \sigma(t)dB_t$.
• Doob $h$-Transform: To make the base process match the data distribution $p_{data}$ at specific time points, they apply a change of measure (Doob $h$-transform). This introduces a drift term that "pulls" the SDE toward high-density regions of the data at prescribed times.
• Non-Markovian Drift: Unlike standard diffusion, the drift $s(t, X_t; X_{0:i^*})$ is path-dependent. It conditions on the entire history of observed states $X_{0:i^*}$ (where $i^*$ is the index of the most recent observation before time $t$). This allows the model to capture complex, non-Markovian correlations.

B. Time-Adaptive Volatility

• The volatility $\sigma(t)$ in the SDE is tied to physical time.
• Theorem 3 proves that chaining separate diffusion bridges (even if they match marginal distributions) results in incorrect path measures (quadratic variations) compared to a single continual SDE.
• By using a single SDE with physical time, the noise injection scales naturally with the time elapsed, ensuring physically plausible dynamics (e.g., less change over short intervals, more over long intervals).

C. Path- and Time-Dependent Score Matching

• Training Objective: The authors derive a loss function that generalizes Denoising Score Matching (DSM) to the non-Markovian, time-dependent case.
• Target: The network learns to predict the score $\nabla_{x_t} \log p_{base}(x_{t_{i^*+1}} | x_t)$, effectively predicting how the current state $x_t$ should be adjusted to align with the next observed waypoint $x_{t_{i^*+1}}$, conditioned on the full path history.
• Any-Subset Conditioning: The neural network (a Transformer with cross-attention) takes as input:
  • The current noisy state $x_t$.
  • The current physical time $t$.
  • The time of the next target state $t_{i^*+1}$.
  • The set of all observed waypoints (past and future) $K$.
    This allows the model to handle arbitrary subsets of observations, including non-causal future frames.

D. Architecture

• Based on the Diffusion Transformer (DiT).
• Extends DiT with cross-attention layers to process the conditioning waypoints (past and future).
• Inputs include the noisy latent, current time, next waypoint time, and a set of conditioning frames with their timestamps.

3. Key Contributions

1. ABC Model: A novel SDE-based generative model that generalizes autoregression to continuous time and space, specifically addressing the Any-Subset case (irregular sampling, non-causal conditioning).
2. Theoretical Guarantees:
   • Proves that a single continual SDE with physical time tracking is necessary to capture correct path measures (Theorem 3).
   • Demonstrates that time-adaptive volatility is crucial for faithful modeling of stochastic processes.
3. Training Objective: Derives a scalable, simulation-free training objective (Path-Dependent Bridge Score Matching) that extends DSM to non-Markovian, time-dependent scenarios.
4. Empirical Validation: Shows state-of-the-art performance on video generation and weather forecasting, outperforming chained diffusion bridges and noise-to-data diffusion.

4. Experimental Results

The authors evaluated ABC on CelebV-HQ (face videos), Sky-Timelapse (weather videos), and SEVIR (radar weather data).

• Metrics: Used FVD (Fréchet Video Distance) for temporal consistency and FID for frame quality. For weather, used MAE, MSE, and CSI (Critical Success Index).
• Performance:
  • Video Generation: ABC significantly outperformed baselines (Conditional Diffusion Bridges and Noise-to-Data Diffusion) in FVD, especially in non-causal settings (infilling missing frames). It achieved Pareto optimality in both FVD and FID.
  • Weather Forecasting: ABC achieved the best overall performance on SEVIR, demonstrating superior accuracy in predicting storm events compared to baselines.
  • Ablation Studies:
    • Volatility: Models with time-adaptive volatility (ABC) consistently beat those with fixed volatility schedules.
    • Conditioning: Path-dependent conditioning (using full history) yielded better results than Markovian (only last frame) or prefix-only conditioning.
    • Data-to-Data vs. Noise-to-Data: Data-to-data bridges (ABC) were superior to noise-to-data diffusion, which suffered from numerical stiffness and incoherent dynamics.

5. Significance and Impact

• Paradigm Shift: Moves beyond the "noise-to-data" paradigm for time series, establishing a "data-to-data" bridge framework that respects physical time continuity.
• Flexibility: Solves the long-overlooked problem of Any-Subset Autoregression in continuous time, enabling applications like video inpainting with irregular frame rates, sparse sensor data imputation, and non-causal forecasting.
• Theoretical Rigor: Provides a mathematically sound derivation using measure theory and stochastic calculus, proving why previous "chained bridge" approaches fail to capture correct dynamics.
• Practical Utility: Demonstrates that modeling the physical time evolution directly leads to more realistic and coherent generations in complex domains like weather and video.

In summary, ABC represents a significant advancement in generative modeling for time series by unifying diffusion bridges, autoregression, and continuous-time dynamics into a single, theoretically grounded framework that adapts to the physical reality of the data.