NanoBench: A Multi-Task Benchmark Dataset for Nano-Quadrotor System Identification, Control, and State Estimation

This paper introduces NanoBench, an open-source multi-task benchmark dataset collected on a 27g Crazyflie 2.1 nano-quadrotor that provides synchronized actuator-level signals, controller internals, and millimeter-accurate ground truth to address the unique modeling and control challenges of nano-scale aerial robotics.

The Gist

Imagine you are trying to teach a tiny, 27-gram drone (about the weight of a large chocolate bar) how to fly. Now, imagine that most of the "flight school" data available to researchers is like a training manual for a heavy, 2-kilogram helicopter.

The physics are totally different. The heavy helicopter flies through thick air like a swimmer in a pool, while the tiny drone flies through air that feels more like thick syrup. The tiny drone's motors are also different, and its computer brain is so small it can barely do math, let alone complex calculations.

The Problem:
Until now, researchers had no good "practice exam" for these tiny drones. They either had to:

1. Build their own private test tracks (which makes it impossible to compare who is doing better).
2. Use data from big drones (which doesn't work because the physics don't match).
3. Guess what happens inside the drone's brain (because they couldn't see the raw motor commands or the internal calculations).

The Solution: NanoBench
The authors of this paper, Syed Izzat Ullah and José Baca, created NanoBench. Think of this as the "Olympic Standard" for tiny drone research.

Here is what makes it special, using some everyday analogies:

1. The "Black Box" Recorder

Imagine a race car driver. Usually, we only see where the car went (the GPS track). But to fix the car, we need to know exactly how hard the driver pressed the gas pedal, what the engine temperature was, and what the driver thought the car was doing.

NanoBench is the first dataset to record everything for a tiny drone:

• The Truth: Where the drone actually was (measured by super-accurate cameras).
• The Commands: The exact electrical signals sent to the motors (the "gas pedal").
• The Brain: What the drone's internal computer thought was happening (its estimates).
• The Battery: How much power was left, because as the battery dies, the drone gets weaker, just like a runner getting tired.

2. The "Drill Sergeant" Training Course

To make sure the data is useful, they didn't just let the drone fly in circles. They designed 12 specific "drills" to stress-test the drone:

• The Shaker: Flying in a complex, wiggly pattern to shake the drone and see how it reacts to sudden changes (like a dentist shaking a tooth to check for looseness).
• The Marathon: Hovering in place for a long time while the battery drains from full to empty, to see how the drone handles getting tired.
• The Obstacle Course: Flying through figure-eights, stars, and spirals at different speeds.

3. The "Three-Part Exam"

The paper uses this data to test three different skills, like a driver's license test:

• Task 1: The Crystal Ball (System Identification)

  • The Challenge: "If I push the throttle like this, where will the drone be in 1 second?"
  • The Result: They found that simple physics formulas work great for the first split-second, but after that, they get confused. However, a "hybrid" model (physics + AI) worked best, acting like a student who knows the rules of the road but also learns from experience.

• Task 2: The Steering Wheel (Control)

  • The Challenge: "Can the drone follow a path perfectly?"
  • The Result: They tested different "pilots" (algorithms). One standard pilot (PID) kept the drone on track but sometimes got shaky. A more advanced geometric pilot (Mellinger) was much steadier and didn't crash, but it moved a bit faster. They also found that a fancy AI pilot (MPPI) completely failed, crashing 75% of the time because it didn't understand the tiny drone's limits.

• Task 3: The Glasses (State Estimation)

  • The Challenge: "How well does the drone's internal computer know where it is?"
  • The Result: The drone's tiny computer does a great job when flying slowly (within 2 centimeters of the truth). But when the drone flies fast, the computer gets dizzy and loses track, much like a human trying to read a sign while running at full speed.

Why This Matters

Before NanoBench, researchers were all speaking different languages and using different rulers. Now, everyone has the same dataset and the same rules.

• For Engineers: It's a standardized test to prove their new drone software actually works.
• For AI: It helps train robots to fly safely in tight spaces (like inside a house or a forest) without crashing.
• For Everyone: It's a step toward having tiny drones that can deliver medicine, inspect bridges, or fly in swarms without needing a human to hold a remote control.

In short: NanoBench is the first time we've given the tiny drone world a fair, transparent, and complete "report card" so we can finally teach these little machines to fly like pros.

Technical Summary

Here is a detailed technical summary of the paper "NanoBench: A Multi-Task Benchmark Dataset for Nano-Quadrotor System Identification, Control, and State Estimation."

1. Problem Statement

Existing aerial robotics benchmarks (e.g., EuRoC, UZH-FPV) primarily target vehicles weighing hundreds of grams to several kilograms. These datasets typically expose only high-level state data, omitting the low-level actuator signals and controller internals necessary to study nano-scale quadrotors (mass < 50 g, e.g., Crazyflie 2.1 at 27 g).

Nano-scale platforms operate under unique constraints that invalidate models and controllers developed for larger drones:

• Aerodynamics: Low Reynolds numbers ($Re \approx 10^4$) lead to laminar separation bubbles and viscous losses, causing thrust/torque characteristics to differ significantly from larger props.
• Actuation: Coreless DC motors introduce deadbands and response lags absent in brushless motors.
• Computation: Limited processing power (e.g., 168 MHz Cortex-M4) restricts algorithms to lightweight Extended Kalman Filters (EKF) and optimized architectures.
• Data Gap: No public dataset currently provides synchronized actuator commands (PWM), controller internals, estimator outputs, and millimeter-accurate ground truth on a commercial nano-platform. This forces researchers to use private or synthetic data, preventing systematic comparison of system identification, control, and state estimation algorithms.

2. Methodology

A. Hardware and Data Acquisition

• Platform: Crazyflie 2.1 (27 g takeoff mass) equipped with a Bosch BMI088 IMU and STM32F405 microcontroller.
• Environment: A 6×4×2 m volume captured by a 12-camera Vicon motion capture system (100 Hz, sub-millimeter accuracy).
• Acquisition Pipeline: A custom ROS 1 framework using the cflib Python library communicates via radio. Three data streams are recorded simultaneously:
  1. Vicon Path: Ground truth pose and derived linear velocity.
  2. Firmware Telemetry: Raw IMU data, onboard EKF state estimates, PID controller internals, and motor PWM commands (100 Hz).
  3. Battery Telemetry: Voltage readings (10 Hz) to capture thrust degradation over discharge.
• Synchronization: A cross-correlation procedure aligns the onboard gyroscope data with Vicon-derived angular velocity to correct clock offsets between the firmware and the host. Residual misalignment is kept below 0.5 ms.

B. Dataset Composition

The dataset contains over 170 flight trajectories (approx. 97.5 minutes of flight time, 603k+ synchronized samples) categorized into three types:

1. Category A (Excitation): Multi-sine trajectories (0.1–5 Hz) designed to excite all dynamic modes for system identification.
2. Category B (Tracking): 10 geometric shapes (Circle, Figure-8, Helix, etc.) executed at varying speeds (0.5–1.0 m/s) to test closed-loop control.
3. Category C (Battery-Drain): Long-duration hovers capturing the full discharge curve (4.2 V to 3.1 V) to model voltage-dependent thrust degradation.

C. Benchmark Tasks

NanoBench defines standardized protocols for three distinct tasks:

1. System Identification (Open-Loop): Predicting 6-DoF state trajectories given motor commands. Metrics include Mean Absolute Error (MAE) over horizons of 0.1s, 0.5s, and 1.0s.
2. Controller Benchmarking (Closed-Loop): Evaluating geometric tracking performance against reference trajectories. Metrics include Tracking RMSE, Average Displacement Error (ADE), and trajectory divergence rates.
3. State Estimation: Assessing the accuracy of lightweight onboard estimators (EKF) against ground truth. Metrics include Absolute Trajectory Error (ATE) and Relative Trajectory Error (RTE).

3. Key Contributions

1. First Multi-Task Nano Dataset: The first public dataset to jointly provide actuator-level commands (PWM), controller internals, estimator outputs, and ground truth on a commercial nano-scale platform.
2. Comprehensive Synchronization: A validated method to align firmware and motion capture clocks with sub-millisecond precision, enabling rigorous validation of lightweight estimators.
3. Standardized Evaluation Suite: Defined train/test splits, metrics, and open-source baseline implementations for system identification, control, and state estimation.
4. Voltage-Aware Data: Inclusion of battery telemetry allows for modeling non-stationary thrust effects caused by LiPo discharge, a factor often ignored in existing benchmarks.

4. Experimental Results and Analysis

Task 1: System Identification

• Physics vs. Learning: Pure rigid-body physics models achieve sub-millimeter accuracy at short horizons (10 ms) but diverge rapidly (636 mm error at 50 steps) due to compounding thrust modeling errors.
• Hybrid Approach: A Physics + Residual MLP hybrid model achieved the best cumulative accuracy, outperforming pure data-driven models (LSTM, Residual MLP) at intermediate horizons. However, pure residual models eventually outperformed physics-based hybrids at very long horizons, suggesting a trade-off between physical priors and data-driven correction.

Task 2: Controller Benchmarking

• Real-Flight Performance: The Mellinger geometric controller (SE(3)) significantly outperformed the default cascaded PID in robustness, reducing trajectory divergence from 4.20% to 0.01%. However, this came at the cost of higher velocity transients (0.89 m/s vs. 0.72 m/s RMSE) due to sharper attitude corrections.
• Offline Learning: Behavior cloning (BC-MLP/LSTM) achieved near-perfect tracking in simulation (0 divergence) but failed to translate to physical readiness without domain adaptation.
• MPPI Failure: Model Predictive Path Integral (MPPI) control failed catastrophically on the nano-platform, with a 75.22% divergence rate, highlighting that standard sampling-based optimization is ill-suited for the extreme actuation limits (<0.6 N thrust) of sub-50g vehicles without specific tuning.

Task 3: State Estimation

• EKF Limits: The onboard EKF maintained high accuracy at slow/medium speeds (ATE RMSE < 22 mm, Attitude RMSE < 3°).
• Dynamic Boundary: At high speeds (1.0 m/s), the EKF diverged (ATE > 3 m), marking the computational and dynamic limits of the 168 MHz Cortex-M4 processor for this specific estimation task.

5. Significance

NanoBench addresses a critical gap in aerial robotics research by providing a standardized, open-source benchmark specifically for the nano-scale regime.

• Reproducibility: It eliminates the variability of private datasets, allowing fair comparison of algorithms across the entire autonomy stack (identification, control, estimation).
• Realism: By capturing coreless motor nonlinearities, low-Re aerodynamics, and battery-induced thrust degradation, it forces algorithms to confront the realities of embedded autonomy.
• Community Impact: The release of over 170 trajectories and baseline code accelerates the development of safe, agile, and swarm-capable nano-drones, which are essential for indoor navigation, search-and-rescue, and swarm robotics.

The dataset and code are publicly available at: https://github.com/syediu/nanobench-iros2026.git.