Efficient sequential Bayesian inference for state-space epidemic models using ensemble data assimilation

1. Problem Statement

Infectious disease modeling requires accurate, real-time estimation of latent epidemic states (e.g., number of infected individuals) and epidemiological parameters (e.g., transmission rates, incubation periods) from noisy, incomplete surveillance data.

• The Challenge: State-space models (SSMs) provide a coherent probabilistic framework for this, but fully Bayesian inference is computationally prohibitive. The observed-data likelihood is intractable because it requires integrating over the latent trajectory.
• Limitations of Current Methods:
  • Particle Markov Chain Monte Carlo (PMCMC) & SMC2: While Sequential Monte Carlo Squared (SMC2) allows for joint state and parameter inference, it relies on a nested structure where an outer SMC sampler over parameters uses an inner Particle Filter (PF) to estimate the likelihood. This "nested" approach is computationally expensive, limiting its use in near-real-time outbreak responses.
  • Ensemble Kalman Filter (EnKF): While computationally efficient and scalable, standard EnKF assumes Gaussian linearity and provides biased likelihood estimates when applied to non-linear, non-Gaussian epidemic data.
  • Approximate Bayesian Computation (ABC): Likelihood-free methods are alternatives but often require excessive simulations and careful tuning of summary statistics, making them difficult for real-time applications.

2. Methodology: Ensemble SMC2 (eSMC2)

The authors propose eSMC2, a novel algorithm that replaces the inner Particle Filter of the standard SMC2 framework with an Ensemble Kalman Filter (EnKF) to approximate the incremental likelihood.

Core Components:

1. State-Space Formulation (SEIR Model):

   • The epidemic is modeled using a stochastic Susceptible–Exposed–Infectious–Removed (SEIR) model.
   • Latent States: $x_t = (S_t, E_t, I_t, R_t, Z_t, \log(\beta_t))$, where $\beta_t$ is a time-varying transmission rate modeled as a stochastic process (Brownian motion).
   • Observation Model: Daily case counts ($y_t$) are modeled as noisy observations of cumulative incidence ($Z_t$). To handle overdispersion common in surveillance data, the authors use a Negative Binomial distribution (or Poisson as a special case), rather than a strict Gaussian assumption.

2. The EnKF Inner Loop:

   • Instead of resampling particles (as in PF), the EnKF propagates an ensemble of state particles through the system dynamics and updates them using linear transformations (Kalman gain).
   • Adaptive Variance: A key innovation is the dynamic specification of observation error variance ($V_t$). Instead of assuming constant variance, the algorithm estimates the conditional variance of the observation model (Poisson or Negative Binomial) based on the forecast ensemble. This allows the filter to adapt to the mean-variance relationship typical of count data.

3. Bias Correction (Unbiased Likelihood):

   • Standard EnKF likelihoods are biased due to finite ensemble sizes and Gaussian approximations.
   • To mitigate this, eSMC2 incorporates an unbiased Gaussian density estimator (based on Ghurye & Olkin, 1969; Drovandi et al., 2022). This correction ensures that the likelihood estimate is unbiased for the Gaussian approximation, reducing sensitivity to the number of ensemble members ($N_x$).

4. The eSMC2 Algorithm (Outer Loop):

   • Maintains a population of parameter particles ($\theta$).
   • At each time step, the weight of each parameter particle is updated using the EnKF-based likelihood estimate.
   • Resampling & Rejuvenation: When the Effective Sample Size (ESS) drops, parameters are resampled. To prevent degeneracy, a Particle Marginal Metropolis–Hastings (PMMH) step is applied to "jitter" the particles, proposing new candidates and accepting/rejecting them based on the EnKF likelihood.

3. Key Contributions

• Computational Efficiency: By replacing the inner Particle Filter with an EnKF, the method achieves a 4.8x to 6.9x speedup compared to standard SMC2 while maintaining comparable accuracy.
• Handling Overdispersion: The method explicitly adapts the EnKF to handle overdispersed count data (Negative Binomial) via state-dependent observation variance, making it suitable for real-world epidemic surveillance.
• Bias Mitigation: The integration of the unbiased Gaussian density estimator corrects the inherent bias of the EnKF likelihood, providing stable posterior inference even with moderate ensemble sizes.
• Sequential Real-Time Capability: The framework is designed for sequential updating, allowing for near-real-time inference as new data arrives, which is critical for public health decision-making.

4. Results

The authors validated eSMC2 through three synthetic simulation studies and a real-world application.

Simulation Studies:

• Scenarios: Tested on three synthetic datasets representing different epidemic dynamics: a moderate surge, a prolonged decay (mimicking 2022 Mpox), and multi-wave oscillatory behavior.
• Accuracy: eSMC2 produced posterior estimates for latent states ($S, E, I, R$) and parameters ($\alpha, \gamma, \beta_t$) that were statistically indistinguishable from the gold-standard SMC2 (Particle Filter-based).
• Performance:
  • Example 1: eSMC2 took ~467s vs. SMC2 ~2760s.
  • Example 2: eSMC2 took ~639s vs. SMC2 ~3929s.
  • Example 3: eSMC2 took ~543s vs. SMC2 ~3766s.
• Robustness: Sensitivity analyses showed the method is robust to ensemble size and reporting fractions.

Real-World Application: 2022 U.S. Monkeypox (Mpox) Outbreak

• Data: Daily confirmed cases from May to December 2022.
• Findings:
  • The model successfully tracked the epidemic trajectory and estimated the time-varying effective reproduction number ($R_{eff}$), showing a decline below 1 after the emergency declaration.
  • Parameter Estimates: Recovered a mean incubation period of ~5.3 days and infectious period of ~18.2 days, consistent with literature.
  • Efficiency: A single run took ~16 minutes with eSMC2, compared to ~2.5 hours for SMC2.
• Forecasting: 14-day-ahead forecasts demonstrated that eSMC2 provided better probabilistic calibration and lower error (MAE/WIS) than a simple autoregressive (AR) baseline, particularly once the epidemic began to stabilize.

5. Significance and Limitations

Significance:

• Bridging the Gap: eSMC2 bridges the gap between the computational efficiency of data assimilation (EnKF) and the inferential rigor of full Bayesian methods (SMC2).
• Public Health Impact: It offers a practical tool for health agencies to perform rapid, uncertainty-quantified inference on epidemic dynamics, enabling timely evaluation of interventions.
• Scalability: The method is particularly valuable for high-dimensional or complex models where standard particle filters suffer from weight degeneracy.

Limitations:

• Theoretical Bias: Unlike standard SMC2 (which is exact as particle count $\to \infty$), eSMC2 remains an approximation due to the Gaussian assumption in the EnKF. While bias is low in moderately nonlinear systems, it may increase in strongly nonlinear or multimodal scenarios.
• Not Fully Online: The PMMH step requires re-running the EnKF on all historical data for each parameter proposal, meaning it is not strictly "online" in the sense of updating only the current state without re-processing history.
• Data Streams: The current implementation focuses on a single data stream (incidence). Integrating multiple data sources (e.g., hospitalizations, serology) would improve identifiability but adds complexity.

Conclusion

The paper presents eSMC2 as a highly efficient, scalable, and accurate framework for sequential Bayesian inference in epidemic modeling. By leveraging the EnKF with unbiased likelihood corrections and adaptive variance, it overcomes the computational bottlenecks of traditional particle-based methods, making real-time, data-driven epidemic monitoring feasible for complex state-space models.