Learn Structure, Adapt on the Fly: Multi-Scale Residual Learning and Online Adaptation for Aerial Manipulators

1. Problem Statement

Autonomous Aerial Manipulators (AAMs), which integrate drones with robotic arms, face significant control challenges due to their inherently coupled, nonlinear, and non-stationary dynamics.

• Residual Dynamics: The system exhibits complex residual dynamics caused by configuration-dependent inertia changes (as the arm moves), nonlinear coupling forces between the drone and arm, and aerodynamic phenomena (e.g., rotor downwash, flow interference).
• Limitations of Existing Methods:
  • Analytical Models: Struggle to accurately model configuration-dependent inertial coupling and complex aerodynamics.
  • Static Data-Driven Models (e.g., DNNs, GPs): Assume stationary dynamics. They degrade rapidly when payloads change or the manipulator reconfigures, leading to accumulating prediction errors.
  • Existing Learning Architectures: Often fail to explicitly separate cross-variable coupling (spatial interactions between degrees of freedom) from multi-scale temporal effects (fast inertial interactions vs. slow aerodynamic transients). Furthermore, many require full network retraining or heavy computational overhead for online adaptation, making them unsuitable for real-time control.

2. Methodology

The authors propose a Predictive-Adaptive Framework consisting of two main components: a structured offline learning model and a lightweight online adaptation mechanism.

A. Factorized Dynamics Transformer (FDT) - Offline Learning

The core innovation is the FDT, a dual-stream Transformer architecture designed to align with the physical structure of AAM dynamics.

• Variable-Wise Tokenization: Unlike standard Transformers that tokenize time steps, FDT treats physical variables (e.g., position, velocity, torque of each joint) as independent tokens. This induces an inductive bias for learning cross-variable coupling.
• Multi-Scale Factorization:
  • Short-Horizon Stream: Uses Self-Attention on a short history window ($T_s$) to capture instantaneous, fast cross-variable inertial couplings and Coriolis effects.
  • Long-Horizon Stream: Uses Cross-Attention to retrieve compressed memory from a longer history window ($T_\ell$). This captures slow-evolving regime effects like aerodynamic wakes and payload-induced inertia shifts.
• Global Token Aggregation: A learnable global token aggregates information from both streams, creating a unified latent representation of the current dynamic regime.
• Output: The model predicts multi-step residual dynamics ($\hat{r}_{t+j|t}$) to minimize error accumulation during control rollout.

B. Latent Residual Adapter (LRA) - Online Adaptation

To handle distribution shifts (e.g., unseen payloads) without retraining the entire network, the LRA performs rapid linear adaptation in the latent space.

• Linear Correction: The frozen FDT produces a base prediction and a latent feature vector ($g'_t$). The LRA assumes deviations are linear with respect to this latent feature: $r_t = \hat{r}^{base}_t + W^T g'_t$.
• Recursive Least Squares (RLS): The adaptation matrix $W$ is updated online using RLS, which provides closed-form, computationally efficient updates.
• Adaptive Covariance Reset: To prevent "covariance collapse" (where the estimator becomes overconfident and unresponsive to sudden changes), the system monitors prediction errors. If errors exceed a threshold, the covariance matrix is reset, restoring estimator plasticity for rapid adaptation to new dynamics.

C. Residual-Compensated Adaptive Control

The adapted residual prediction is integrated into a control law.

• The controller uses the corrected residual estimate to compensate for modeling errors in the closed loop.
• It employs a robust adaptive term to handle bounded residual estimation uncertainties, ensuring Lyapunov stability without requiring full system identification.

3. Key Contributions

1. Structure-Aligned Residual Modeling (FDT): Introduced a dual-stream Transformer that explicitly separates short-horizon inertial interactions from long-horizon aerodynamic memory by tokenizing physical variables rather than time steps.
2. Latent Linear Online Identification (LRA): Proposed a mechanism that restricts online adaptation to a linear correction in the latent space via RLS. This preserves the offline nonlinear representation while enabling rapid, low-overhead adaptation to unseen payloads.
3. Integrated Control Framework: Demonstrated a complete pipeline where the adapted residual forecast directly compensates for modeling errors in a real-time adaptive controller, avoiding the need for full system identification.

4. Experimental Results

The framework was validated on a real-world aerial manipulator (Tarot-650 quadrotor + 2-DOF arm) under varying payloads (0g, 200g, 400g, and unseen 300g/500g) and complex trajectories (S-shaped pick-and-place, Figure-8).

• Prediction Accuracy:
  • FDT + LRA achieved the lowest Root Mean Squared Error (RMSE) of 0.21 and the highest $R^2$ of 0.972, significantly outperforming baselines like Neural-Fly (RMSE 0.48), standard Transformers (RMSE 0.67), and System ID (RMSE 1.36).
  • Inference time remained low (3.02 ms), ensuring real-time feasibility.
• Trajectory Tracking:
  • The proposed method demonstrated superior closed-loop tracking precision and faster disturbance attenuation compared to state-of-the-art learning baselines and L1-adaptive control.
• Ablation Studies:
  • Removing the long-horizon memory or short-horizon encoding caused the largest performance degradation, confirming the necessity of multi-scale factorization.
  • Disabling the online LRA resulted in consistent performance drops under payload shifts, validating the importance of latent adaptation.

5. Significance

This work addresses a critical gap in aerial robotics by providing a solution that is both structurally aware and adaptively robust.

• Physical Interpretability: By separating spatial coupling from temporal scales, the model offers better generalization than "black-box" sequence models.
• Real-Time Viability: The decoupling of offline representation learning from lightweight online parameter identification allows the system to adapt to sudden changes (like dropping or picking up a payload) without the computational burden of retraining deep networks.
• Robustness: The framework enables AAMs to operate reliably in unstructured environments with unknown or varying payloads, a prerequisite for advanced tasks like assembly and contact inspection.

In summary, the paper presents a novel architecture that aligns machine learning structures with the physical realities of aerial manipulation, achieving high-fidelity prediction and robust control in dynamic, non-stationary environments.