💻 computer science

Adaptive Learned State Estimation based on KalmanNet

This paper introduces Adaptive Multi-modal KalmanNet (AM-KNet), a hybrid state estimator that enhances autonomous driving perception by integrating sensor-specific learning, context-aware hypernetworks, and a specialized covariance estimation branch to achieve superior tracking accuracy and stability on real-world automotive datasets compared to existing methods.

Original authors: Arian Mehrfard, Bharanidhar Duraisamy, Stefan Haag, Florian Geiss, Mirko Mählisch

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Arian Mehrfard, Bharanidhar Duraisamy, Stefan Haag, Florian Geiss, Mirko Mählisch

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a car, and you need to know exactly where every other car, pedestrian, or bicycle is, how fast they are moving, and where they are heading. To do this, your car's computer uses "eyes" (cameras), "ears" (radar), and a "3D scanner" (Lidar).

But here's the problem: these eyes, ears, and scanners are all imperfect.

Cameras are great at seeing shapes and colors but terrible at judging exact distance.
Radar is great at measuring speed but often blurry on the sides.
Lidar is precise but can get confused by rain or fog.

The car's computer has to guess the true path of these objects despite the "noise" (mistakes) from each sensor. This is called State Estimation.

The Old Way: The Rigid Rulebook

Traditionally, cars use a math formula called a Kalman Filter. Think of this like a strict teacher who follows a rigid rulebook. The teacher knows the laws of physics (like how a car accelerates) and tries to guess where a student (the car) is.

However, this "teacher" assumes all mistakes are the same. It doesn't know that a camera's mistake is different from a radar's mistake. If the data gets messy (like in heavy rain or with a jaywalking pedestrian), the rigid teacher gets confused and starts guessing wildly, leading to unstable tracking.

The New Way: The "Adaptive Multi-modal KalmanNet" (AM-KNet)

The authors of this paper created a smarter system called AM-KNet. Instead of a rigid rulebook, they built a hybrid coach that combines the best of physics with the learning power of Artificial Intelligence (AI).

Here is how AM-KNet works, using simple analogies:

1. The Specialized Scouts (Sensor-Specific Modules)

Imagine you have a team of scouts reporting to a commander.

In the old system, the commander treated a report from a scout with bad eyesight the same as a report from a scout with perfect vision.
AM-KNet gives each sensor its own specialized scout.
- The Radar Scout knows, "I'm good at speed, but my side vision is blurry, so I'll be careful with side measurements."
- The Camera Scout knows, "I see shapes clearly, but I'm bad at depth, so I'll trust the radar for distance."
- The Lidar Scout knows, "I'm precise, but I get confused in the rain."
  By letting each sensor "speak its own language," the system learns exactly how much to trust each one in any given situation.

2. The Contextual Chameleon (Hypernetwork & Context Modulation)

Imagine a chameleon that changes its skin color to match its surroundings.

A car driving in a straight line on a highway behaves differently than a car trying to merge into traffic or a pedestrian crossing the street.
AM-KNet has a "chameleon brain" (a hypernetwork). It looks at the context:
- Who is the target? (Is it a giant truck or a small bike?)
- What is it doing? (Is it stopped, driving straight, or crossing?)
- Where is it? (Is it right in front of us or far away?)
  Based on this, the system instantly adjusts its "rules" to fit the specific scenario. It doesn't use one size-fits-all logic; it adapts its behavior on the fly.

3. The Honest Scorekeeper (Covariance Estimation)

In the old system, the computer would guess a position but often lie about how sure it was. It might say, "I'm 100% sure this car is here," when it was actually very unsure.

AM-KNet includes a special "Honesty Module." It doesn't just guess the position; it also calculates a confidence score (mathematically called covariance).
It uses a special math trick (Joseph's form) and learns from its own mistakes to say, "I think the car is here, but I'm only 60% sure because the rain is heavy." This honesty is crucial for safety. If the system knows it's unsure, the car can slow down.

The Result: A Smoother, Safer Ride

The researchers tested this new system on real-world driving data (from cities like Boston, Singapore, and Delft).

The Old Way (Rule-based filters): Often got confused, leading to "jittery" tracking where the car's position on the screen would jump around.
The New Way (AM-KNet): Stuck to the target much more smoothly. It was better at predicting where cars would go, even when they were turning or changing lanes.

In a nutshell:
The old system was like a student memorizing a textbook and failing when the test questions changed. AM-KNet is like a student who understands the principles of driving but also learns from experience, knows the strengths and weaknesses of their tools, and adapts their strategy based on whether they are driving in a school zone or on a highway. This makes self-driving cars safer and more reliable in the messy, unpredictable real world.

1. Problem Statement

While hybrid state estimators combining model-based Kalman Filtering (KF) with Deep Neural Networks (DNNs) have shown promise on simulated data, they often fail to match the performance of classical Bayesian filters on real-world automotive data.

The Gap: The base KalmanNet (a hybrid KF where an RNN learns the Kalman Gain) demonstrated insufficient accuracy and unstable tracking when applied to real-world sensor data (Radar, Lidar, Camera).
The Cause: The base model struggled to learn the distinct noise characteristics of different sensor modalities and failed to adapt to diverse traffic scenarios (e.g., varying target types, motion states, and relative poses). It lacked the physical priors and context awareness necessary for robust autonomous driving.

2. Methodology: Adaptive Multi-modal KalmanNet (AM-KNet)

The authors propose AM-KNet, an advanced architecture that retains the recursive predict-update structure of the Kalman Filter but enhances it with learned, data-driven components tailored for multi-sensor autonomous driving.

A. Architectural Innovations

Sensor-Specific Measurement Modules:
- Unlike the base KalmanNet which treats all inputs uniformly, AM-KNet employs distinct sub-networks (measurement heads) for each sensor modality (Radar, Lidar, Camera).
- This allows the network to independently learn the unique noise statistics and error distributions of each sensor (e.g., Radar's high radial accuracy vs. poor lateral accuracy; Camera's excellent lateral accuracy vs. poor depth).
Context Modulation via Hypernetwork:
- To handle diverse traffic scenarios, a hypernetwork is introduced to modulate the pre-activations of the main network.
- Input: The hypernetwork receives a 27-dimensional context vector encoding the target type (car, truck, pedestrian), motion state (stationary, same lane, crossing), and relative pose.
- Mechanism: It outputs "Gain" and "Shift" parameters that apply an affine transformation to the main network's layers. This allows a single model to dynamically adapt its internal dynamics based on the specific scenario.
Covariance Estimation Branch (Joseph Form):
- To ensure physically valid uncertainty estimates, the authors implement a covariance estimation branch based on the Joseph form of the covariance update.
- The update is split: the deterministic part is computed using state-space matrices, while the noise-dependent term is learned via a dedicated GRU.
- The network predicts the Cholesky factor of the noise term to guarantee the resulting covariance matrix is positive semi-definite.

B. Loss Function Design

The training objective is a comprehensive, component-wise loss function designed to encode physical priors:

Normalized Squared Error (MSE): The base loss, weighted by sensor characteristics, target class, motion state, and measurement flow consistency.
Flow Consistency Weighting: A dynamic weight based on the "flow" of data points. If a sensor's recent measurements are inconsistent with the ground truth motion trend, its weight is reduced.
Negative Log-Likelihood (NLL) Losses:
- State NLL: Supervises the learned state covariance ( $P$ ) using the estimation error.
- Innovation NLL: Supervises the innovation covariance ( $S$ ) using the measurement residual.
- These terms ensure the network outputs calibrated uncertainty, not just point estimates.

C. Training Strategy

Datasets: Evaluated on nuScenes (Boston/Singapore, 60/20/20 split) and View-of-Delft (VoD) (Netherlands, urban traffic).
Three-Phase Schedule:
1. Warmup: Train only on MSE to learn accurate dynamics.
2. Stationary Loss: Enable loss for static targets to encourage inertia.
3. NLL Activation: Enable NLL losses for $P$ and $S$ to refine uncertainty estimation.
Hypernetwork Training: The main AM-KNet is trained first; weights are then frozen, and the hypernetwork is trained separately on the full dataset to learn context adaptation.

3. Key Contributions

Multi-Modal Adaptation: Introduction of sensor-specific heads that allow the network to learn distinct noise profiles for Radar, Lidar, and Camera, addressing a major limitation of the original KalmanNet.
Context-Aware Filtering: The use of a hypernetwork with context modulation (target type, motion, pose) enables a single model to adapt to complex, heterogeneous traffic scenarios without needing multiple parallel models.
Calibrated Uncertainty: The integration of the Joseph form and dual NLL losses provides explicit, physically consistent covariance estimates, bridging the gap between data-driven flexibility and probabilistic rigor.
Comprehensive Loss Function: A novel loss design that incorporates sensor reliability, object class, and motion flow consistency to guide the learning process.

4. Experimental Results

The model was compared against OAFuser (Online Adaptive Fuser) and UKF-EOT (Unscented Kalman Filter Extended Object Tracker).

Accuracy (MAE):
- VoD Dataset: AM-KNet achieved the lowest Mean Absolute Error (MAE) for position ( $x$ : 0.27m, $y$ : 0.21m) and velocity ( $v_x$ : 0.48 m/s). It significantly outperformed baselines in estimating object extent (length/width).
- nuScenes Dataset: AM-KNet reduced position error by roughly 50% compared to UKF-EOT and 70% compared to OAFuser. It also achieved the best accuracy for object dimensions.
Consistency (NEES/NIS):
- NEES (Normalized Estimation Error Squared): AM-KNet showed significantly better consistency (closer to the ideal 90% acceptance region) than baselines.
  - VoD: 77% consistency for position (vs. 33-45% for baselines).
  - nuScenes: 60% consistency (vs. 33-37% for baselines).
- Interpretation: The high NEES values of baselines (e.g., OAFuser mean of 106.95) indicate severe overconfidence (underestimating uncertainty), whereas AM-KNet provides well-calibrated uncertainty estimates.

5. Significance

This work addresses a critical bottleneck in autonomous driving: the lack of robust, interpretable, and accurate state estimation in real-world, multi-sensor environments.

Bridging the Gap: It successfully narrows the performance gap between learned models and classical Bayesian filters on real-world data, a feat previous attempts (like base KalmanNet) failed to achieve.
Interpretability & Safety: By preserving the KF structure and providing calibrated uncertainty (covariance), the system offers the safety guarantees required for motion planning, unlike "black-box" end-to-end approaches.
Scalability: The context modulation approach allows a single model to handle diverse scenarios (different vehicle types, maneuvers, and sensor setups) without the computational cost of maintaining multiple parallel filters.

In conclusion, AM-KNet represents a significant step forward in hybrid state estimation, demonstrating that combining physical priors (KF structure, Joseph form) with deep learning (sensor-specific heads, context modulation) yields superior accuracy and reliability for autonomous driving applications.