⚡ electrical engineering

Learning Adaptive Parameter Policies for Nonlinear Bayesian Filtering

This paper proposes using reinforcement learning to formulate adaptive parameter selection in nonlinear Bayesian filtering as a sequential decision-making problem, thereby learning policies that dynamically adjust parameters to improve estimation accuracy, consistency, and robustness over time.

Original authors: Ondrej Straka, Felipe Giraldo-Grueso, Renato Zanetti

Published 2026-03-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ondrej Straka, Felipe Giraldo-Grueso, Renato Zanetti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to track a runaway dog in a dense, foggy forest. You can't see the dog directly; you only hear occasional barks (measurements) and know roughly how fast dogs usually run (your model). Your goal is to guess where the dog is right now and where it will be next.

This is exactly what nonlinear Bayesian filtering does for robots, self-driving cars, and satellites. It's a mathematical way to guess a hidden state based on noisy, imperfect data.

However, these mathematical "guessing machines" aren't perfect. To make them work, engineers have to dial in some settings—like how many "guesses" to make, how much to trust the model versus the sensor, or how many times to repeat a calculation.

The Problem: The "Set-and-Forget" Trap

Traditionally, engineers pick these settings once at the beginning and leave them alone. They might say, "Let's use 100 guesses for the dog's location," and stick with that number forever.

The problem is that the forest changes. Sometimes the fog is thick (high uncertainty), sometimes the dog is running in a straight line (easy to predict), and sometimes it's zig-zaging wildly (highly nonlinear). A setting that works great in the fog might be terrible when the dog is sprinting.

If you use the wrong setting at the wrong time, your guess gets a little bit wrong. But here's the scary part: that small mistake gets carried over to the next guess, and the next, and the next. Eventually, your tracker loses the dog completely.

The Solution: The "Smart Coach"

This paper proposes a new way to handle these settings. Instead of a human picking a fixed number, we teach the filter to learn how to adjust its own settings in real-time.

Think of it like a smart coach for an athlete:

The Old Way: The coach tells the athlete, "Run at 80% effort for the whole race," regardless of whether it's raining, the track is slippery, or the athlete is tired.
The New Way (This Paper): The coach watches the race. If the track gets slippery, the coach whispers, "Slow down, focus on balance." If the track clears up, the coach says, "Speed up, push for the finish." The coach learns from experience what to say in every specific situation.

How Does the Coach Learn? (Reinforcement Learning)

The paper uses a technique called Reinforcement Learning (RL). Imagine the filter is a video game character, and the "settings" are the buttons it can press.

Trial and Error: The filter tries different settings (pressing different buttons) in a simulated world.
The Score: It gets points for being accurate and consistent. It loses points if it gets confused or makes a huge mistake.
Thinking Ahead: This is the magic part. Most filters only care about getting the current guess right (short-term thinking). This new method teaches the filter to think long-term.
- Analogy: A short-sighted player might take a shortcut that saves time now but leads to a dead end later. This new "Smart Coach" might choose a slightly slower path now because it knows it will avoid a traffic jam five minutes down the road. It sacrifices a tiny bit of immediate perfection to ensure the whole journey goes smoothly.

What Did They Test?

The authors tested this "Smart Coach" on two scenarios:

The "Wobbly" Dog (UNGM): A math problem where the dog's movement is very unpredictable and bounces around wildly.
The "Turning" Car (CTM): A car making sharp turns, where the math gets tricky because the sensors (bearing measurements) get confused.

They compared their new method against:

The Default: The standard, fixed settings everyone uses.
The "Myopic" Coach: A coach that only looks one step ahead.
The "Perfect" Coach: A coach that knows the future (which is impossible in real life, but good for comparison).

The Results

The "Smart Coach" (Adaptive Policy) won.

It was more accurate (found the dog/car closer to the truth).
It was more consistent (it didn't get confused or lose its mind when things got messy).
It was smarter about resources: In one test, it learned to do fewer calculations when things were easy (saving battery/computer power) and more calculations when things got hard.

The Big Takeaway

This paper changes how we think about estimation. Instead of treating the filter as a static machine with fixed dials, it treats the filter as a decision-maker.

By using AI to teach the filter how to tune itself moment-by-moment, we can build systems that are more robust, more accurate, and better at handling the messy, unpredictable real world. It's the difference between driving a car with cruise control set to a fixed speed versus driving with a co-pilot who constantly adjusts the speed based on the road, traffic, and weather.

1. Problem Statement

Nonlinear Bayesian state estimation (e.g., using Extended Kalman Filters, Unscented Kalman Filters, or Particle Filters) inevitably relies on numerical approximations. These approximations depend on tunable parameters such as:

The number of particles (in Particle Filters).
Scaling parameters (in Unscented Kalman Filters).
The number of iterations (in Stochastic Integration Filters).

The Core Issue:
Traditionally, these parameters are either fixed or adjusted heuristically based on immediate measurements. However, the accuracy of these approximations varies over time depending on the local degree of nonlinearity and uncertainty.

Myopic Limitation: Classical approaches optimize for the immediate estimation error (e.g., minimizing Mean Squared Error at time $k$ ). They fail to account for how a parameter choice at time $k$ propagates approximation errors through subsequent updates, potentially causing filter divergence, inconsistency, or information loss in the future.
Gap: There is a lack of a general framework that treats parameter selection as a sequential decision-making problem to optimize long-term filter performance rather than just instantaneous accuracy.

2. Methodology

The authors propose a framework that formulates parameter adaptation as a Markov Decision Process (MDP) and solves it using Reinforcement Learning (RL).

A. Formulation as an MDP

The paper reframes the estimation problem:

State (Information State): Instead of the true state (which is unknown), the "information state" $\bar{s}_k$ is defined by the filter's internal representation (e.g., predictive mean, covariance, sigma points, or particles).
Action: The filter parameters $\theta_k$ (e.g., UKF scaling factor, SIF iteration count) are treated as actions.
Transition: The information state evolves stochastically based on the chosen parameters and new measurements: $\bar{s}_{k+1} = \Phi_{filter}(\bar{s}_k, z_k, \theta_k)$ .
Cost Function: The immediate cost $\bar{L}$ is defined based on measurement prediction errors (e.g., Normalized Innovation Squared - NIS) or state innovation norms. This cost can also include computational penalties.
Objective: Minimize the cumulative discounted cost over a long horizon, rather than just the instantaneous cost. This introduces non-myopic reasoning, where the agent may accept a suboptimal immediate estimate to ensure better consistency and stability in future steps.

B. Reinforcement Learning Solution

Since the analytical solution to the Bellman recursion for this MDP is intractable, the authors employ an Actor-Critic RL algorithm with Temporal Difference (TD) learning:

Actor (Policy): A neural network (MLP) that maps the information state and current measurement to a probability distribution over discrete parameter choices. It learns the policy $\pi(\theta | \bar{s}, z)$ .
Critic (Value Function): A neural network that approximates the Bellman value function $V(\bar{s}, z)$ , estimating the expected future cost.
Training: The system uses an off-policy approach with a target critic network (using Polyak averaging) to stabilize training. The actor is updated via policy gradient to minimize the cost, while the critic minimizes the TD error.
Cost Functions Tested:
- NIS cost (penalizing inconsistency).
- Log-max NIS (penalizing only optimistic estimates).
- State innovation L2-norm (penalizing accuracy).
- Combined costs including computational weight.

3. Key Contributions

Decision-Theoretic Formulation: The paper establishes a novel theoretical link between Bayesian filtering and optimal control, treating parameter selection as a sequential decision problem (MDP) rather than a static tuning task.
Non-Myopic Adaptation: It demonstrates that explicitly optimizing for long-term consequences (using a discount factor $\gamma > 0$ ) yields superior filter consistency and robustness compared to greedy, myopic optimization ( $\gamma = 0$ ).
General Framework: The approach is generic and applicable to various Gaussian-assumed filters (UKF, SIF) and potentially Particle Filters, adapting parameters like scaling factors or iteration counts.
RL Integration: It provides a practical implementation using Actor-Critic RL to learn adaptive policies offline, which are then deployed online.

4. Experimental Results

The method was validated on two benchmark problems:

Univariate Nonstationary Growth Model (UNGM): A scalar nonlinear system with strong nonlinearity and multimodal posteriors.
Coordinated Turn Model (CTM): A 4D system with bearing measurements, representing a typical tracking scenario.

Algorithms Compared:

Adaptive (Proposed): RL-learned non-myopic policy.
Myopic: Same RL structure but with $\gamma=0$ (greedy).
Optimal: Parameters selected to maximize likelihood (upper bound, computationally expensive).
Default/Fixed: Standard heuristic tuning (e.g., $\kappa = 3-n_x$ for UKF).

Key Findings:

Consistency (ANEES): The adaptive policy consistently achieved the best Average Normalized Estimate Error Squared (ANEES), indicating superior statistical consistency. It prevented the filter from becoming overly optimistic or pessimistic.
Accuracy (RMSE): The adaptive policy achieved the lowest Root Mean Squared Error (RMSE) compared to myopic, optimal, and default baselines in most scenarios.
Trade-off: The RMSE-ANEES plots showed that the adaptive policy found a superior Pareto frontier, balancing accuracy and consistency better than fixed parameters.
Computational Efficiency (SIF): In the Stochastic Integration Filter (SIF) experiments, where iteration count affects computational cost, the adaptive policy successfully balanced accuracy against computational load, outperforming both fixed high-iteration (slow) and fixed low-iteration (inaccurate) baselines.
Long-term vs. Short-term: The non-myopic approach ( $\gamma=0.5$ ) significantly outperformed the myopic approach ( $\gamma=0$ ), proving that looking ahead improves long-horizon performance.

5. Significance

This work represents a paradigm shift in how nonlinear filters are tuned.

From Static to Dynamic: It moves beyond "set-and-forget" or heuristic tuning toward dynamic, context-aware parameter selection.
Robustness: By optimizing for long-term consistency, the proposed method mitigates filter divergence and information loss, which are critical issues in safety-critical applications (e.g., aerospace, robotics).
Practicality: The use of offline training allows the learned policy to be deployed in real-time with minimal computational overhead during operation (only a forward pass through a small neural network).
Future Direction: It opens the door for "learning to filter," where the entire estimation strategy can be optimized as a sequential decision process, potentially extending to learning the filter structure itself in the future.

In conclusion, the paper demonstrates that treating filter parameter adaptation as a sequential decision problem solvable via Reinforcement Learning leads to filters that are more accurate, consistent, and robust than traditional fixed or myopic approaches.