💻 computer science

MUSE: Multimodal Uncertainty Quantification of State Estimation

This paper introduces MUSE, a novel real-time learning-based framework that leverages the Mamba architecture to effectively quantify multimodal uncertainty in visual-inertial state estimation, demonstrating superior reliability and robustness compared to existing methods.

Original authors: Minkyung Kim, Henry Che, Bhargav Chandaka, Bhumsitt Pramuanpornsatid, Chengyu Yang, Sheng Cheng, Xiaofeng Wang, Naira Hovakimyan, Shenlong Wang

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Minkyung Kim, Henry Che, Bhargav Chandaka, Bhumsitt Pramuanpornsatid, Chengyu Yang, Sheng Cheng, Xiaofeng Wang, Naira Hovakimyan, Shenlong Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a car in a thick fog. You can see the road right in front of you, but as you look further ahead, everything gets blurry. You have a GPS, but it's a bit glitchy. You know you're moving, but you aren't 100% sure exactly where you are or how fast you're going.

In the world of robots and self-driving cars, this is called State Estimation. It's the robot's way of figuring out its position and orientation. For years, robots have gotten very good at guessing their location using cameras and motion sensors. But there's a big problem: they don't know when they are wrong.

They might be 99% sure they are in the right spot, but if they are actually 100% wrong, they could crash. They need a way to say, "I'm pretty sure," or "I have no idea, please be careful."

The Problem: The "Overconfident" Robot

Current robots are like a student who takes a test and gets a question wrong but still marks it with a confident "A."

Visual Odometry (VO) is like the robot looking at pictures to guess where it is.
Inertial Odometry is like the robot feeling its own movement (like a human feeling dizzy after spinning).

When these two disagree, or when the camera sees a blurry image (like a smudge on a lens) or the motion sensor drops out, the robot usually just keeps guessing. It doesn't realize the conditions have changed. It lacks Uncertainty Quantification—the ability to measure how much it should trust its own guess.

The Solution: MUSE (The Robot's "Second Opinion")

The authors of this paper created a new system called MUSE (Multimodal Uncertainty Quantification of State Estimation).

Think of MUSE as a super-smart co-pilot that sits next to the robot's main navigation system.

It watches everything: While the robot's main system might only look at the camera, MUSE looks at everything at once: the camera images, the motion sensors (IMU), and the robot's own speedometer data.
It spots the trouble: If the camera sees a blurry image (like a smudge) but the motion sensor says, "Hey, we just stopped moving," MUSE notices this conflict. It's like a detective noticing that a witness's story doesn't match the physical evidence.
It gives a confidence score: Instead of just giving a location, MUSE says, "Here is where you are, AND here is a 'confidence meter' showing how much you should trust that number."
It fixes the mistake: If the robot is drifting off course, MUSE doesn't just warn it; it actually nudges the robot's position back to where it should be.

How It Works: The "Mamba" Brain

To do this in real-time (without slowing the robot down), MUSE uses a special type of AI brain called Mamba.

The Old Way (Transformers): Imagine trying to remember a long story by reading the whole book every time you want to recall one sentence. It's accurate but very slow and heavy.
The Mamba Way: Mamba is like a librarian who can instantly scan a long shelf of books, remembering the important parts and forgetting the irrelevant ones. It's incredibly fast and efficient at processing long streams of data (like a video feed from a drone).

This allows MUSE to look at a sequence of events over time. It can say, "The camera was fine 5 seconds ago, but 2 seconds ago the IMU sensor dropped out, and now the image is blurry. Therefore, my confidence in the current location should drop."

The Results: A Better Navigator

The researchers tested MUSE on two types of data:

Standard Tests: Using public datasets where robots flew in indoor arenas.
Hard Mode: Using their own new dataset called UnCal-Flight, which included tricky situations like sudden movements, changing lights, and people walking in front of the sensors.

The findings were clear:

Better Accuracy: MUSE corrected the robot's position better than previous methods, especially when the robot was confused.
Honest Confidence: When the robot was in a tricky situation (like a blurry image), MUSE correctly lowered its confidence score. Other methods kept being overconfident even when they were wrong.
Speed: It runs fast enough to be used as a "plugin" for existing robot systems without needing a supercomputer.

The Bottom Line

MUSE is like giving a robot a "gut feeling" about its own navigation. It combines all the robot's senses to detect when things are going wrong, corrects the robot's path, and honestly tells the robot when it should be worried. This makes robots safer and more reliable, especially in the messy, unpredictable real world.

Technical Summary: MUSE (Multimodal Uncertainty Quantification of State Estimation)

1. Problem Statement

Accurate visual state estimation is critical for robotics applications such as autonomous driving, drone navigation, and operation in GPS-denied environments. While recent advances in Visual Odometry (VO) and Visual-Inertial Odometry (VIO) have improved pose estimation accuracy, a fundamental challenge remains: quantifying and calibrating the precision of these estimates.

Existing methods struggle with uncertainty quantification due to the heteroscedastic (variance depends on input) and multimodal nature of odometry errors.

Filter-based methods (e.g., Kalman filters) often suffer from inconsistency due to linearization errors and model mismatch, leading to overconfident or conservative covariances.
Optimization-based methods estimate uncertainty post-hoc, yielding local and poorly calibrated measures.
Learning-based methods achieve high accuracy but frequently remain overconfident, providing unreliable uncertainty estimates.
Prior learning-based uncertainty approaches (e.g., DICE, D-DICE) typically rely on instantaneous visual inputs, failing to exploit temporal information or multimodal cues (such as IMU anomalies or sensor degradation) that are crucial for predicting drift and failure.

2. Methodology: MUSE Framework

The paper introduces MUSE, a real-time, learning-based framework designed to jointly correct poses and predict heteroscedastic uncertainty from asynchronous multi-sensor streams.

Core Architecture

Backbone: MUSE leverages Mamba, a structured state-space model (SSM), to efficiently capture sequential relationships and long-horizon dependencies in sensor data. This addresses the computational limitations of Transformers while maintaining the ability to process long sequences.
Input Modalities: The system ingests three asynchronous streams:
1. Visual: Stereo images encoded via a pretrained SuperPoint encoder.
2. Inertial: IMU data (linear acceleration and angular velocity) encoded via a pretrained RONIN encoder.
3. Odometry: Raw pose estimates and associated covariances from the upstream VO/VIO module, encoded via a fully connected network.
Output: The model predicts a non-zero-mean Gaussian distribution over the pose error in $SE(3)$, represented as $\xi_i \sim \mathcal{N}(\mu_i, \Sigma_i)$ $ξ_{i} \sim N (μ_{i}, Σ_{i})$ .
- Mean ( $\mu_i$ ): Used to correct the estimated pose ( $\hat{T}'_i = \exp(\mu_i^\wedge)\hat{T}_i$ ).
- Covariance ( $\Sigma_i$ ): Provides a calibrated measure of uncertainty.

Technical Implementation

Error Modeling: The pose error is mapped to the Lie algebra $\mathfrak{se}(3)$ via a logarithm map. The covariance matrix $\Sigma_i$ is reconstructed using LDL Decomposition ( $\Sigma_i = L_i D_i L_i^T$ ) to ensure positive semi-definiteness and training stability.
Training Objective: The model is trained end-to-end using two decoupled losses:
1. Geodesic Loss ( $L_{geo}$ ): Minimizes the distance between the bias-corrected estimate and ground truth, including a smoothness penalty for trajectory consistency.
2. Negative Log-Likelihood ( $L_{nll}$ ): Calibrates the uncertainty by minimizing the discrepancy between observed errors and the predicted Gaussian distribution.

3. Key Contributions

The authors summarize their contributions as follows:

Novel Architecture: A framework that learns multimodal, heteroscedastic uncertainty in odometry estimation, explicitly addressing the limitations of single-modality, frame-based approaches.
UnCal-Flight Dataset: The release of a challenging, open-source drone navigation dataset featuring complex motion patterns, varying lighting, and dynamic obstacles to evaluate VO/VIO robustness.
Real-Time Plugin: Demonstrated effectiveness as a deployable plugin for existing VO/VIO systems (e.g., VINS-Fusion, MSCKF, ORB-SLAM 3, DPVO) across diverse datasets, improving both accuracy and reliability.

4. Experimental Results

Experiments were conducted on the EuRoC dataset and the authors' UnCal-Flight dataset.

Pose Correction: MUSE significantly reduced pose errors compared to baselines. On the EuRoC dataset, it reduced the RMSE of VINS-Fusion by 45.7% (from 0.164m to 0.089m), outperforming D-DICE and DPC-Net, which showed minimal improvement or regression.
Uncertainty Calibration:
- Zero-Mean Setting: MUSE achieved superior Expected Normalized Calibration Error (ENCE) across all tested odometry methods (e.g., ENCE of 0.300 for VINS-Fusion vs. 0.765 for DICE), indicating better-calibrated confidence.
- Non-Zero-Mean Setting: MUSE outperformed D-DICE in both Log-Likelihood (LL) and ENCE metrics, demonstrating its ability to correct systematic biases while quantifying uncertainty.
Robustness: In the challenging UnCal-Flight dataset, MUSE improved RMSE by 22.2% and 38.4% over raw odometry in different trajectory subsets.
Ablation Studies: Results confirmed that fusing all modalities (Camera + IMU + Odometry) yields the best performance, validating the necessity of multimodal integration for capturing drift and uncertainty.
Efficiency: The system achieves an inference time of under 3 ms per window on a single NVIDIA RTX 6000 GPU.

5. Significance and Claims

The paper claims that MUSE represents the first framework to jointly address the multimodal, temporal, and heteroscedastic nature of odometry uncertainty. By utilizing Mamba for efficient sequential modeling, MUSE overcomes the limitations of prior visual-only or instantaneous uncertainty predictors.

The authors position MUSE not merely as a standalone estimator but as a versatile, real-time plugin capable of enhancing safety-critical decision-making in dynamic and resource-constrained environments. The work bridges the gap between high-accuracy state estimation and reliable uncertainty quantification, enabling downstream planners to reason about the confidence of their state estimates. The paper concludes that multimodal integration is essential for robust uncertainty prediction, a finding supported by their extensive ablation studies.