📊 statistics

High-dimensional Bayesian filtering through deep density approximation

This paper benchmarks deep density approximation methods for high-dimensional nonlinear filtering, demonstrating that the logarithmic deep backward stochastic differential equation filter significantly outperforms traditional particle-based approaches in scalability and computational efficiency, particularly in partially observed 100-dimensional systems.

Original authors: Kasper Bågmark, Filip Rydin

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kasper Bågmark, Filip Rydin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to track a lost hiker in a dense, foggy forest. You can't see them directly, but every few minutes, a drone flies overhead and sends you a blurry, noisy photo of where they might be. Your job is to guess exactly where the hiker is right now, based on all the photos you've received so far and your knowledge of how people usually walk in the woods.

This is the essence of Bayesian Filtering: guessing a hidden state (the hiker) based on noisy, partial observations (the photos).

The Problem: The "Fog" Gets Thicker

In simple scenarios (a small forest), you can use standard tools to track the hiker.

The Particle Filter: Imagine sending out 1,000,000 tiny robots (particles) to guess where the hiker might be. As the forest gets bigger (higher dimensions), you need billions of robots to cover the space. It becomes impossible to manage them all; they run out of battery (computational power) and the guess becomes useless. This is the "Curse of Dimensionality."
The Kalman Filter: This is like assuming the hiker walks in a perfectly straight line. It's fast, but if the hiker suddenly stops to tie a shoe or runs in a circle (non-linear movement), this method gets confused and fails.

The Solution: The "Deep Density" Approach

The authors of this paper propose a new way to solve this using Deep Learning (AI). Instead of sending out millions of robots, they train a super-smart AI "navigator" to learn the shape of the fog itself.

Think of the "fog" as a 3D map showing the probability of the hiker being in any spot.

The Old Way: The AI tries to learn the height of the fog map directly. But in a huge forest, the fog is so thin that the numbers representing its height become microscopic (like $10^{-40}$ ). Computers get confused by these tiny numbers, and the AI crashes or gives garbage answers.
The New Trick (Log-Density): The authors realized that instead of guessing the height of the fog, the AI should guess the logarithm of the height.
- Analogy: Imagine the fog is a mountain. Instead of measuring the mountain's height in meters (where the top is 10,000 and the bottom is 0.0000001), you measure the "steepness" or the "log-scale." This keeps the numbers manageable and prevents the AI from getting lost in the tiny details. It also guarantees the AI never predicts a "negative fog" (which is impossible).

The Two New Navigators

The paper tests two specific AI architectures for this task:

The Deep Splitting Filter: This method breaks the problem into small, manageable steps, solving a tiny puzzle at each moment in time and stitching them together.
The Deep BSDE Filter: This method looks at the whole journey backward from the end to the beginning, using a mathematical trick called a "Backward Stochastic Differential Equation" to figure out the most likely path.

The Winner: The Log-BSDE (the backward-looking AI using the log-trick) turned out to be the champion.

The Results: Why It Matters

The authors tested these methods on everything from simple 1D lines to a chaotic, 100-dimensional system called the Lorenz-96 model (which simulates weather patterns).

In Low Dimensions: The old methods (Particle Filters) worked fine, but the new AI methods were 100 to 100,000 times faster.
In High Dimensions (The 100D Forest):
- The Particle Filter (the robot army) collapsed completely. It needed so many robots that the computer ran out of memory.
- The Kalman Filter got lost because the weather system was too chaotic.
- The Log-BSDE AI kept working perfectly. It tracked the hiker accurately even when the "forest" had 100 dimensions.

The Big Takeaway

This paper shows that by combining Deep Learning with a clever mathematical trick (the logarithmic transformation), we can finally track hidden objects in massive, complex, and chaotic systems.

In everyday terms:
If you were trying to predict the stock market, track a virus spreading globally, or forecast the weather in a chaotic climate, the old methods would require a supercomputer the size of a city and still fail. The new method described in this paper is like having a single, highly trained detective who can solve the case in seconds, even when the clues are messy and the city is huge. It's a massive leap forward in making AI reliable for the real world's most complex problems.

1. Problem Statement

The paper addresses the nonlinear filtering problem in high-dimensional state spaces. The goal is to infer the conditional probability density (filtering density) of a hidden state $S_t$ evolving according to a Stochastic Differential Equation (SDE), given a sequence of noisy, partial observations.

The Challenge: Classical methods face the "curse of dimensionality."
- Particle Filters (PF): Require an exponentially growing number of particles to maintain accuracy as dimension increases, leading to particle degeneracy.
- Kalman-type Filters (EKF/EnKF): Rely on linearization or Gaussian assumptions, which fail in strongly nonlinear, non-Gaussian regimes.
Numerical Instability: Directly solving the Fokker–Planck equation for the density $p$ becomes numerically unstable in high dimensions because probability density values diminish exponentially (e.g., $10^{-40}$ in 100D), causing vanishing gradients and underflow issues during optimization.

2. Methodology

The authors propose and benchmark two deep learning-based approaches that reformulate the filtering problem using Feynman–Kac formulas, Euler–Maruyama discretization, and neural networks.

A. Core Framework

The filtering density evolves via a prediction step (governed by the Fokker–Planck PDE) and an update step (governed by Bayes' rule). The authors approximate these steps using neural networks trained via stochastic gradient descent.

B. The Two Methods

Deep Splitting Filter (DSF):
- Reformulates the filtering equations into a sequence of recursive optimization problems.
- Alternates between propagating the density via the Euler–Maruyama scheme and correcting it via Bayes' formula at observation times.
- Solves for the density at each time step by minimizing the difference between the current approximation and the operator applied to the next step.
Deep Backward Stochastic Differential Equation Filter (BSDEF):
- Formulates the prediction step as a single optimization problem per observation time using a Backward SDE (BSDE).
- Simultaneously learns the predictive density and auxiliary processes (approximating the $Z$ -component of the BSDE).
- Condenses the filtering problem into one global optimization per time step, contrasting with the stepwise recursion of DSF.

C. Key Innovation: Log-Density Formulation

To address numerical instability in high dimensions, the authors introduce logarithmic variants (LogDSF and LogBSDEF).

Transformation: Instead of approximating the density $p$ , they approximate the log-density $v = -\log(p)$ .
Theoretical Basis: Derived in Theorem 2.1, transforming the Fokker–Planck equation into a nonlinear Kolmogorov backward equation.
Benefits:
- Positivity-Preserving: Since the density is reconstructed via exponentiation ( $p = e^{-v}$ ), the resulting density is strictly positive.
- Stability: Avoids vanishing gradients and numerical underflow associated with multiplying small probability values.
- Objective Function: Provides a more robust loss landscape for training.

3. Key Contributions

Log-Density Reformulation: The derivation of log-transformed filtering equations (Theorem 2.1) and the resulting LogDSF/LogBSDEF algorithms, providing a theoretically grounded framework for stable high-dimensional filtering.
Systematic Benchmarking: A comprehensive numerical comparison of DSF, BSDEF, and their log-variants against classical baselines (EKF, EnKF, and Bootstrap PF) across linear, nonlinear, low, and high-dimensional settings.
High-Dimensional Success: Demonstration that the LogBSDEF remains accurate and stable in chaotic, partially observed regimes up to 100 dimensions (Lorenz-96), where particle filters fail completely.
Computational Efficiency: Evidence that deep density methods reduce inference time by 2 to 5 orders of magnitude compared to particle-based filters in high-dimensional settings.

4. Experimental Results

The authors tested the methods on four categories of problems:

1D Toy Examples (Ornstein–Uhlenbeck & Bistable):
- Deep methods (especially LogBSDEF) achieved lower errors (rMAE, FME, KLD) than classical methods.
- Computationally, they were 2–5 orders of magnitude faster.
High-Dimensional Linear (Ornstein–Uhlenbeck, $d=100$ ):
- Particle filters with $10^6$ particles failed to approximate the density.
- LogBSDEF performed on par with the Ensemble Kalman Filter (EnKF) but with significantly lower computational cost.
- Non-logarithmic variants failed to train due to numerical instability.
Nonlinear Bistable System (Schl¨ogl Model):
- A physically grounded model with strong nonlinearity.
- EKF and EnKF performed poorly.
- LogBSDEF matched the performance of a high-particle-count PF ( $10^5$ ) in terms of KLD, while LogDSF struggled to converge, highlighting the robustness of the BSDE approach for this specific problem.
Chaotic High-Dimensional System (Lorenz-96, $d=4$ to $d=100$ ):
- The Critical Test: In the 100-dimensional partially observed case, classical methods (PF, EnKF) diverged or became computationally intractable.
- LogBSDEF maintained stable performance across all dimensions, showing only mild growth in error.
- Scalability: While EnKF and PF errors grew exponentially with dimension, LogBSDEF errors remained manageable.

5. Significance and Conclusion

Overcoming the Curse of Dimensionality: The paper demonstrates that deep density approximation, particularly the LogBSDEF, offers a viable path for nonlinear filtering in high-dimensional spaces where traditional Monte Carlo methods fail.
Efficiency vs. Accuracy: The methods trade a one-time training cost for extremely fast inference. Once trained, they are orders of magnitude faster than particle filters, making them suitable for real-time applications in complex systems (e.g., climate modeling, finance).
Numerical Robustness: The log-transformation is identified as a crucial enabler, solving the fundamental numerical issues of density estimation in high dimensions.
Limitations: The methods require careful hyperparameter tuning and architecture selection (e.g., LSTM for long horizons). They currently rely on Gaussian observation noise and fixed time steps, though the framework is theoretically extensible.

In summary, this work establishes LogBSDEF as a state-of-the-art solution for high-dimensional nonlinear filtering, bridging the gap between theoretical probabilistic numerics and practical deep learning applications.