💻 computer science

Reinforcement Learning with Inner-loop Dynamics Estimator for Aerial Manipulation under Uncertainty

This paper presents a hierarchical control framework that combines Reinforcement Learning for high-level whole-body coordination with an inner-loop dynamics estimator for low-level uncertainty compensation, demonstrating improved tracking accuracy and task success rates for aerial manipulators under varying payload conditions and dynamic uncertainties.

Original authors: Shivansh Pratap Singh, Samaksh Ujjwal, Ishita Chaudhary, V R Vasudevan, Rishabh Dev Yadav, Spandan Roy

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shivansh Pratap Singh, Samaksh Ujjwal, Ishita Chaudhary, V R Vasudevan, Rishabh Dev Yadav, Spandan Roy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a drone that doesn't just fly around looking at things, but actually has a robotic arm to pick up, move, and drop objects. This is called an aerial manipulator.

The paper tackles a very tricky problem: What happens when this drone-arm combo tries to grab something heavy, move it quickly, or let it go?

Think of it like a human trying to juggle while riding a unicycle. If you suddenly grab a heavy bowling ball with one hand, your balance shifts instantly. If you swing your arm fast, the unicycle wobbles. For a drone, these shifts in weight and movement are chaotic. The drone's computer often doesn't know exactly how heavy the object is or how the arm's movement will shake the whole machine, leading to crashes or dropped items.

Here is how the authors solved this, broken down into simple concepts:

The Two-Brain Solution

The researchers built a "two-brain" system to handle this chaos.

1. The "Big Picture" Brain (The RL Policy)
Imagine a skilled dance instructor. This brain doesn't worry about the tiny details of how to move a specific muscle. Instead, it looks at the goal (e.g., "Move the gripper to that red box") and tells the whole body: "Okay, lean left, spin slightly, and reach forward."

What it does: It uses Reinforcement Learning (a type of AI that learns by trial and error) to figure out the best overall movement plan. It learns how to coordinate the drone's flight and the arm's reach together, rather than treating them as two separate things.
The limitation: Even the best dance instructor can't predict exactly how the wind will hit them or how the unicycle will wobble if the floor is slippery.

2. The "Reflex" Brain (The Inner-Loop Estimator)
This is the second brain, and it acts like a super-fast reflex. While the "Big Picture" brain is planning the dance, the "Reflex" brain is constantly watching the actual movement.

How it works: It uses a clever trick called a Dynamics Estimator. It doesn't need a perfect manual of how the drone works. Instead, it looks at what happened a split second ago. If the drone started to wobble unexpectedly (maybe because the arm swung too fast or the payload was heavier than expected), this brain instantly calculates, "Oh, we are shaking! Let's push back harder to fix it right now."
The Analogy: It's like riding a bike. You don't consciously calculate the physics of every turn. You just feel the bike leaning and your body automatically shifts to keep you upright. This system does that automatically for the drone.

The Experiment: The "Figure-Eight" Test

To prove this works, they built a real drone with a 3-jointed arm and a gripper. They made it fly in a figure-eight pattern (which requires constant turning and speed changes) while carrying different weights (200g and 400g).

They tested three different control methods:

The Old Way: The AI plans the move, but a standard, rigid controller tries to execute it (like a robot following a script without feeling).
The "Better" Way: The AI plans the move, and a slightly smarter controller tries to adjust (but still relies on a simplified model).
The New Way (Their Method): The AI plans the move, and the "Reflex" brain instantly corrects any wobbles or surprises.

The Results

The new method was the clear winner:

Accuracy: The drone's hand stayed much closer to the target path. It was about 41% more accurate than the standard method and 26% more accurate than the "better" method.
Reliability: It successfully completed the task more often, even when carrying heavy loads or moving fast.
Consistency: Every time they tried the test, the results were very similar (low variation), meaning the system is stable and predictable.

The Bottom Line

The paper claims that by combining a smart, learning-based "planner" with a fast, model-free "reflex" system, you can make aerial robots much better at handling heavy or unpredictable loads. It proves that you don't need a perfect mathematical model of the drone to make it work; you just need a system that can learn the big moves and instantly fix the small mistakes as they happen.

Technical Summary: Reinforcement Learning with Inner-loop Dynamics Estimator for Aerial Manipulation under Uncertainty

Problem Statement

Aerial manipulators, which integrate multi-rotor UAVs with robotic arms, are essential for physical interaction in complex 3D environments. However, achieving reliable direct whole-body aerial manipulation remains a significant challenge. The core difficulty lies in the strong coupling between the underactuated aerial base and the manipulator arm. Rapid arm motion, payload changes (grasping/releasing), and external disturbances cause transient shifts in the center of gravity, inertia, and dynamic coupling. These uncertainties are difficult to model analytically, making stable and precise control under varying operating conditions a largely unsolved problem. Existing methods often rely on simplified linear models or decoupled control strategies that fail to account for these nonlinear, coupled dynamics, leading to degraded performance during aggressive maneuvers or payload variations.

Methodology

The authors propose a hierarchical control framework that combines Reinforcement Learning (RL) for high-level coordination with an inner-loop dynamics estimator for low-level uncertainty compensation. The system operates without requiring a fully accurate coupled dynamic model in the policy layer.

1. Hierarchical Architecture

Outer Loop (RL Policy): A learned policy maps a desired 6-DOF end-effector target pose to coordinated whole-body commands. The observation space (33 dimensions) includes task-space geometric errors, base states (position, velocity, attitude), and joint states. The action space (10 dimensions) outputs desired base linear acceleration, desired yaw, desired body rates, and desired joint positions.
- Training: The policy is trained using Proximal Policy Optimization (PPO) in a GPU-accelerated vectorized simulation (Aerial Gym) with domain randomization (varying payload mass, link lengths, motor time constants, etc.).
- Reward Shaping: The reward function utilizes a two-scale exponential shaping to balance global convergence (reaching the target) with local precision (maintaining stability and smoothness). A temporal progress reward encourages movement toward the goal while penalizing backward motion to prevent oscillations.
Inner Loop (Estimator-Based Controller): This layer tracks the high-level commands generated by the RL policy. It employs a delay-based dynamics estimator to compensate for transient inertial shifts and unmodeled coupling without prior system knowledge.
- The controller treats the lumped unknown dynamics (including coupling terms, Coriolis forces, and disturbances) as an uncertainty function $\bar{H}_\eta$ .
- Using a diagonal nominal inertia matrix $\bar{M}_\eta$ and a delay $L$ , the controller estimates the uncertainty $\hat{\bar{H}}_\eta(t)$ from past state and control inputs: $\hat{\bar{H}}_\eta(t) = \tau_\eta(t-L) - \bar{M}_\eta \ddot{\eta}(t-L)$ .
- The control law combines a standard tracking term with this estimated uncertainty to generate low-level torque commands for both the quadrotor body and the manipulator joints.

2. System Implementation

The framework was validated on a custom hardware platform consisting of a Tarot-450 quadrotor, a 3-DoF Dynamixel manipulator, and a Jetson Orin Nano Super for high-level computation. The RL policy runs at 100 Hz, while the inner-loop controller operates at 300 Hz.

Key Contributions

Hierarchical Architecture: A novel integration of RL-based whole-body command generation with an estimator-based inner-loop for tracking and uncertainty compensation.
Model-Free Execution Layer: A dynamics-estimation execution layer capable of handling transient UAV-arm coupling changes, including inertial variations from grasp/release events and rapid manipulator motion, without requiring an a priori system model.
Robust Evaluation: Comprehensive evaluation under off-nominal conditions, specifically varying payload conditions and dynamic coupling during arm motion.
Demonstrated Improvement: Empirical evidence showing that combining learned coordination with estimator-based feedback significantly improves end-effector regulation accuracy and smoothness under dynamic uncertainty.

Experimental Results

The proposed method was tested on a planar figure-eight trajectory with varying payloads (200g and 400g) and speeds (0.5 m/s and 1.0 m/s), including in-flight pick-up and drop-off actions. It was compared against two baselines: RL+PID and RL+INDI+PID (Indirect Adaptive Control).

Tracking Accuracy: The proposed method achieved the lowest Root Mean Square Error (RMSE) across all tested conditions.
- Compared to RL+PID, it reduced the mean RMSE by 40.9%.
- Compared to RL+INDI+PID, it reduced the mean RMSE by 26.4%.
Success Rate: The proposed method demonstrated higher task success rates, particularly in aggressive scenarios (e.g., 400g payload at 1.0 m/s), where baselines showed significant performance degradation.
Repeatability: The standard deviation of the RMSE was lowest for the proposed method (0.024 m) compared to the baselines (0.038 m and 0.062 m), indicating superior repeatability under hardware variability.

Significance and Claims

The paper claims that the proposed approach effectively addresses the open challenge of direct whole-body aerial manipulation under abrupt payload changes and dynamic uncertainty. The authors assert that learned whole-body coordination is most effective when paired with estimator-based low-level compensation. This combination allows the system to:

Generate task-level commands without relying on an accurate analytical model of the coupled dynamics.
Compensate for transient inertial shifts and unmodeled disturbances in real-time.
Improve precision, robustness, and repeatability in high-speed and high-payload regimes where traditional model-based or simplified learning approaches struggle.

The results are presented as strong empirical evidence supporting the efficacy of this hierarchical design, though the authors note the study is limited to the specific figure-eight hardware experiment and does not include formal significance testing.