Technical Summary: Generative Modeling of Approximately Periodic Time Series by a Posterior-Weighted Gaussian Process

1. Problem Statement

Discrete automated processes in industrial and cyber-physical systems (e.g., robotic assembly lines) often exhibit approximately periodic behavior. While successive repetitions follow a common trajectory, they differ in duration, amplitude, and fine-scale dynamics due to noise, wear, and environmental fluctuations.

Modeling these processes with Gaussian Processes (GPs) presents a specific challenge:

• Strictly periodic models enforce exact repetition, suppressing the necessary inter-repetition variability found in real data.
• Non-periodic models fail to capture the strong structural regularities required for long-term generation.
• Existing quasi-periodic approaches often enforce exact periodicity or introduce mean decay and variance inflation over long horizons, limiting their utility as generative models for arbitrarily many repetitions.

The goal is to develop a stochastic generative model that can produce arbitrarily long sequences of repetitions that preserve the characteristic shape of the process while allowing for smooth, controlled variation between repetitions.

2. Methodology

The authors propose a two-stage construction that decouples intra-repetition structure from inter-repetition variability. The core innovation is modulating the posterior of a GP rather than the prior, avoiding pathological behaviors associated with weighted priors.

Stage 1: Learning a Stable Repetition Template

The first stage models the intra-repetition statistics using a strictly periodic GP prior.

• Assumption: Observed repetitions $y_1, \dots, y_r$ are treated as (partially) independent realizations of the same underlying periodic process sharing common hyperparameters.
• Kernel: A periodic kernel $k_\theta$ is used, where inputs are embedded on a circle.
• Training Strategy: To address the issue that noise variance $\sigma^2_\theta$ is not identifiable from independent repetitions under a factorized likelihood, the authors employ a mini-batched training strategy. The negative log-likelihood (NLL) is evaluated over small batches of multiple repetitions. This forces the model to explain inter-repetition variations, stabilizing the noise variance estimate.
• Post-Training Adjustment: During posterior construction, repetitions are treated as observations of the same latent function to preserve global spatial structure. To prevent artificial attenuation of uncertainty (due to "accumulation of evidence" from multiple repetitions), the noise variance $\sigma^2_\theta$ is rescaled by the effective number of repetitions.

Stage 2: Posterior Weighting

The second stage introduces inter-repetition correlations by modifying the posterior covariance directly, rather than altering the prior kernel.

• Mechanism: A novel weight kernel $w_\psi$ (a Gaussian kernel in an embedding space) is applied via element-wise multiplication (Schur product) to the posterior covariance matrix $\hat{\Sigma}_\theta$ obtained in Stage 1.
  $$\Sigma_\psi := W_\psi \odot \hat{\Sigma}_\theta + \sigma^2_\psi I$$
• Weight Kernel Design: The kernel $w_\psi$ uses a specific embedding $\phi(t)$ involving the integral of a squared sine function. This ensures the kernel has vanishing derivatives at integer multiples of the period, preventing unwanted phase shifts in the weighted kernel.
• Generative Model: The final generative distribution is defined as:
  $$\gamma \sim \mathcal{N}(\hat{\mu}_\theta, \Sigma_\psi)$$
  where $\hat{\mu}_\theta$ is the mean from Stage 1 (unchanged) and $\Sigma_\psi$ is the modulated covariance.
• Rationale:
  • Leaving the mean unchanged ensures the mean structure is identical across all repetitions, satisfying the requirement for structural consistency.
  • Modulating the covariance allows local uncertainty to be preserved while ensuring correlations between distant repetitions decay monotonically to zero.
  • This approach avoids the "pathology" of using a weighted periodic prior, which would cause the predictive mean to decay to zero and variances to inflate over time.

3. Key Contributions

1. Novel Generative Model: A stochastic generative model for approximately periodic multivariate time series based on GPs that supports arbitrarily many repetitions.
2. Two-Stage Construction: A method that separates the learning of a stable periodic template (Stage 1) from the injection of controlled inter-repetition variability (Stage 2).
3. Posterior Modulation: The introduction of a kernel-modulated posterior (rather than a weighted prior) to achieve long-horizon stability, ensuring the mean remains consistent while covariance reflects the natural decay of similarity between repetitions.
4. Mini-Batch Training Strategy: A technique to stabilize the estimation of noise variance in the presence of independent repetitions by evaluating likelihoods over batches.
5. Practical Implementation: Demonstration of the model's ability to generate realistic synthetic trajectories from toy datasets, including multi-dimensional cases using Linear Models of Coregionalization (LMC).

4. Experimental Results

The model was evaluated on synthetic datasets generated from weighted-periodic GPs, including noiseless and noisy scenarios, as well as 2-dimensional data.

• Batch Size Sensitivity: Experiments showed that the estimated variance $\sigma^2_\theta$ in Stage 1 is highly dependent on the mini-batch size $|B|$. With $|B|=1$, variance collapses toward zero. As $|B|$ increases, the estimate stabilizes, correctly capturing intrinsic variability.
• Noise Handling: In noisy settings, the second-stage noise parameter $\sigma^2_\psi$ increases with observation noise levels, while the first-stage noise $\sigma^2_\theta$ remains relatively stable until noise levels become very high.
• Multi-Output Capability: The model successfully extended to 2-dimensional data using non-stationary periodic kernels, capturing varying degrees of variation in different cycles (e.g., higher variance when crossing the origin).
• Performance: The model achieved low Mean Squared Error (MSE) between the predictive mean and the test mean, and accurately modeled pointwise standard deviations.

5. Significance and Claims

The paper claims that modifying the GP posterior rather than the prior provides a principled way to model approximately periodic processes for generative purposes.

• Stability: The model achieves long-horizon generative stability, capable of producing arbitrarily long sequences without the mean decay or variance inflation seen in other quasi-periodic approaches.
• Interpretability: By decoupling the mean (strictly periodic) from the covariance (decaying correlation), the model maintains the interpretability and calibrated uncertainty of GPs.
• Applications: The authors identify specific practical applications where this stability and interpretability are crucial:
  • Generation of realistic synthetic data for unit testing.
  • Creation of digital twins.
  • Populating honeypots in industrial systems.
  • Likelihood-based anomaly detection by quantifying deviations from learned statistics.

The authors remain modest regarding future work, noting that scaling to large numbers of repetitions will require sparse or approximate inference techniques, and that systematic evaluation on real-world industrial datasets is necessary to quantify practical benefits in downstream tasks.