💻 computer science

A Proprioceptive-Only Benchmark for Quadruped State Estimation: ATE, RPE, and Runtime Trade-offs Between Filters and Smoothers

This paper presents a comprehensive benchmark comparing three proprioceptive state estimators (MUSE, IEKF, and IS) for quadruped robots on the GrandTour Dataset, evaluating their trade-offs in long-term accuracy, short-term precision, and computational runtime to guide practitioners in selecting the most suitable approach for their specific application constraints.

Original authors: Ylenia Nisticò, João Carlos Virgolino Soares, Joan Solà, Claudio Semini

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ylenia Nisticò, João Carlos Virgolino Soares, Joan Solà, Claudio Semini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a thick, foggy forest with no GPS, no landmarks, and no map. To know where you are, you have to rely entirely on your own body: how your legs move, how your inner ear feels the tilt, and whether your feet are touching the ground. This is exactly the challenge faced by four-legged robots (quadrupeds) when they lose their "eyes" (cameras) or "GPS."

This paper is like a taste test for three different "brains" (algorithms) that help these robots figure out where they are using only their body sensors. The authors put these three brains through a rigorous obstacle course (a real-world dataset called the Grand Tour) to see which one is the most accurate and which one is the fastest.

Here is the breakdown of the three "brains" they tested:

1. The Three Competitors

MUSE (The Modular Specialist): Think of MUSE as a team of specialists working in a relay race. One person checks your balance, another counts your steps, and a third combines that info. It's designed to be very fast and modular.
IEKF (The Steady Navigator): This brain uses a specific mathematical trick (Lie groups) that keeps its calculations stable even when the robot slips or stumbles. It's like a navigator who never loses their sense of direction, no matter how bumpy the ride gets.
IS - Invariant Smoother (The Historian): This brain doesn't just look at the now; it looks at a short "movie clip" of the last few seconds. By reviewing recent history, it can correct small mistakes it made a moment ago. It's like a detective who re-examines the crime scene to get the perfect answer, but it takes a little longer to think.

2. The Race Results

The authors measured two main things: Long-term accuracy (Did the robot end up in the right place after a long walk?) and Short-term accuracy (Did the robot know exactly where it was right now?). They also measured speed (how long it took the brain to think).

The Long Walk (ATE): Over the entire 300-meter course, the "Historian" (IS) and the "Steady Navigator" (IEKF) were the most accurate. They ended up closest to the true destination. The "Specialist" (MUSE) drifted a bit more, ending up about a meter further off than the best performers.
The Instant Check (RPE):
- Over 1 meter: When looking at short distances (like walking one step), the IEKF and IS were much better at staying on track than MUSE.
- Frame-by-Frame: When looking at the split-second between updates, MUSE and IEKF were actually sharper and less jittery than the IS. The IS, because it's busy reviewing its "movie clip," was slightly slower to react to tiny, rapid movements.
Speed (Runtime):
- MUSE was the lightning-fast sprinter, taking only about 0.012 milliseconds to think.
- IEKF was also very fast, taking about 0.02 milliseconds.
- IS was the thoughtful marathon runner. It took longer (0.1 to 0.6 milliseconds) because it was doing more math to smooth out the path. The more history it looked back at, the slower it got.

3. The "Waking Up" Test (Convergence)

The researchers also tested what happens if the robot starts the race with its head spinning (a huge error in its initial orientation).

MUSE was the quickest to "wake up" and realize, "Oh, I'm upside down!" and correct itself.
IEKF and IS were slower to correct this big initial mistake because they rely on small, local adjustments that can get confused if the starting error is too massive.

The Bottom Line

The paper doesn't declare one single "winner." Instead, it gives a clear menu of trade-offs:

Choose MUSE if you need the robot to react instantly to control its legs (low latency) and don't mind a tiny bit of long-term drift.
Choose IEKF if you want a great balance of high accuracy and speed, with excellent stability.
Choose IS if you need the highest possible accuracy for mapping or navigation and can afford a tiny delay (latency) to let the algorithm "think" over a short window of time.

The authors released all their code and data so that anyone else can run this same race and verify the results. It's a guide to help engineers pick the right "brain" for their specific robot job.

Technical Summary: A Proprioceptive-Only Benchmark for Quadruped State Estimation

Problem Statement

Reliable proprioceptive state estimation is critical for agile quadruped locomotion, particularly in environments where exteroceptive sensing (e.g., visual, LiDAR, GPS) is unreliable or unavailable due to dust, fog, featureless corridors, or signal outages. In such scenarios, robots must rely solely on Inertial Measurement Units (IMUs), joint encoders, and contact information to estimate body motion. The core challenge involves balancing two competing requirements:

Long-term consistency: Maintaining accuracy over extended trajectories despite drift and modeling errors.
Short-horizon fidelity: Preserving local accuracy essential for feedback control and foothold planning, all while adhering to strict latency and computational budgets on embedded platforms.

While various approaches exist, there is a lack of direct, "apples-to-apples" comparisons between different state-of-the-art paradigms (filtering vs. smoothing, invariant vs. non-invariant) on a common real-world dataset to guide practitioners in selecting the appropriate estimator for their specific application constraints.

Methodology

The authors present a comparative benchmark of three representative, open-source state estimators for quadruped robots, all operating in a proprioceptive-only mode on the CYN-1 sequence of the GrandTour Dataset. All methods estimate the floating-base pose $(R, p) \in SO(3) \times \mathbb{R}^3$ and body velocity $v \in \mathbb{R}^3$ .

1. MUSE (Multi-sensor Unified State Estimator)

Paradigm: Modular, real-time fusion stack (Filtering).
Mechanism: In this study, MUSE operates purely proprioceptively, disabling exteroceptive blocks. It utilizes a cascaded Nonlinear Observer (NLO) and an eXogeneous Kalman Filter (XKF) for attitude estimation, ensuring global stability. Leg odometry is derived from forward kinematics of legs in stable contact. These measurements are fused via a Kalman Filter to output pose and twist.
Key Feature: Designed for robustness and real-time performance, capable of handling large initial orientation errors due to its globally stable attitude observer.

2. Invariant Extended Kalman Filter (IEKF)

Paradigm: Invariant Filtering.
Mechanism: Formulated on a group-affine system where the right-invariant error dynamics are state-independent. This structure improves convergence and consistency, particularly during contact events.
Mechanism Details: Propagation integrates IMU data, while updates assume stable contact (zero foot velocity). It leverages Lie group structures to linearize error dynamics, incorporating robust measurement costs and slip awareness.

3. Invariant Smoother (IS)

Paradigm: Fixed-lag Smoothing (Maximum A Posteriori).
Mechanism: Unlike the IEKF, which updates only the most recent state, the IS optimizes over a fixed window of states ( $n$ ) and measurements. It formulates the problem as a nonlinear least-squares problem using structure-aware residuals adapted from the IEKF's group-affine properties.
Key Feature: By optimizing over a window, it achieves improved tolerance to dynamic contacts (e.g., slips) and higher accuracy, but at the cost of increased computational load and algorithmic latency proportional to the window size.

Evaluation Metrics

The study employs standard metrics computed via the evo Python package:

Absolute Trajectory Error (ATE): Measures global pose accuracy and accumulated drift by aligning the estimated trajectory to the ground truth via SE(3) Umeyama alignment.
Relative Pose Error (RPE): Measures short-horizon accuracy. Evaluated under two schemes:
- Spatial window ( $\Delta = 1$ m): Probes meter-scale drift.
- Temporal window ( $\Delta = 1$ frame): Probes frame-to-frame consistency and sensitivity to measurement jitter.
Velocity RMSE: Assesses the accuracy of estimated velocity profiles.
Runtime: Per-update computation time measured on a fixed hardware/software stack.

Key Results

Accuracy

Long-term (ATE): Both the IEKF and the Invariant Smoother (IS) achieved lower ATE than MUSE. The difference between the best (IS) and worst (MUSE) was approximately 0.91 m over a 300 m trajectory (approx. 0.3%). This suggests that group-affine modeling and smoothing regularization help reduce accumulated drift.
Short-term (RPE):
- At $\Delta = 1$ m: IEKF and IS significantly outperformed MUSE (approx. 40% reduction in translational RPE), indicating better handling of systematic errors accumulating over multiple updates.
- At $\Delta = 1$ frame: IEKF and MUSE exhibited lower RPE than IS (approx. 72% and 66% reduction, respectively). This suggests that the higher-rate fusion in filters better suppresses high-frequency noise compared to the smoothing approach in this specific metric.
- Rotational RPE: Results were broadly similar across all methods.
Velocity: All three methods provided accurate velocity profiles with small RMSE differences.

Runtime and Latency

MUSE: Fastest, with an average runtime of $\approx 0.012$ ms per iteration.
IEKF: Slightly slower, $\approx 0.02$ ms per iteration.
Invariant Smoother: Demonstrates a clear accuracy-latency trade-off dependent on the window size (WS).
- WS = 1: $\approx 0.110$ ms.
- WS = 5: $\approx 0.583$ ms.
- Note: Fixed-lag smoothing introduces algorithmic latency proportional to the lag, which may delay state estimates depending on pipeline implementation.

Convergence Behavior

Under large initial orientation errors (180-degree roll misalignment), all three estimators converged to the correct orientation. However, MUSE converged faster than IEKF and IS. This is attributed to MUSE's globally stable attitude observer design, whereas IEKF and IS rely on local linearization which can be violated by large initial errors, leading to slower convergence.

Significance and Contributions

The paper provides a concise, reproducible snapshot of the trade-offs between accuracy, latency, and computational cost for deployable quadruped systems. Its primary contributions are:

Direct Comparison: An "apples-to-apples" evaluation of three distinct, high-performing estimation paradigms (MUSE, IEKF, IS) on a shared real-world dataset.
Practitioner Guidance: Clear data on how different metrics (ATE vs. RPE at different scales) and computational constraints (latency vs. accuracy) influence estimator selection.
Open-Source Reproducibility: The release of a complete benchmarking pipeline, including code, documentation, and data processing tools, to facilitate future standardized benchmarks.

The authors conclude that while smoothing (IS) offers superior long-term accuracy, filters (MUSE, IEKF) offer lower latency and potentially better short-term noise suppression. The choice of estimator should be dictated by the specific application's requirements regarding global consistency, control loop frequency, and available compute resources. The study does not claim to solve all state estimation challenges but aims to clarify the current landscape for practitioners.