💻 computer science

AI-Enabled Image-Based Hybrid Vision/Force Control of Tendon-Driven Aerial Continuum Manipulators

This paper proposes an AI-enabled cascaded hybrid vision/force control framework for tendon-driven aerial continuum manipulators that integrates constant-strain modeling, fast fixed-time sliding mode control, and graph neural network-based feature extraction to achieve robust, online learning of uncertainties for autonomous physical interaction with static environments.

Original authors: Shayan Sepahvand, Farrokh Janabi-Sharifi, Farhad Aghili

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shayan Sepahvand, Farrokh Janabi-Sharifi, Farhad Aghili

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a flying robot that doesn't just have rigid metal arms like a crane, but a long, flexible, snake-like arm made of soft materials. This is a Tendon-Driven Aerial Continuum Manipulator (TD-ACM). Think of it as a drone with a "octopus arm" attached to it.

The problem? Controlling a snake-like arm while flying is incredibly hard. If the arm bumps into a wall, it bends. If the wind blows, the drone shakes. If the camera sees a blurry line, the robot gets confused. Traditional robots use rigid arms and simple math, but those methods fail here because the arm is constantly changing shape.

This paper presents a new "brain" for this robot that allows it to gently touch and push objects while flying, even when things get messy. Here is how they did it, explained simply:

1. The "Smart Eye" (Vision)

Most robots use cameras to find specific dots or corners (like a chessboard pattern) to know where they are. But in the real world, those dots might get hidden, or the lighting might change.

The Innovation: Instead of looking for dots, this robot looks for lines (like the edges of a door or a window).
The Analogy: Imagine trying to navigate a room. It's hard if you only look for a specific red ball. It's much easier if you look at the long lines of the walls and the floor. The robot uses a special "AI eye" (a Graph Neural Network) that is really good at spotting these lines, even if they are partially blocked or the lighting is weird. It's like having a super-artist's eye that can see the structure of a room even in the dark.

2. The "Feel" (Force Control)

The robot needs to push against a wall with just the right amount of pressure—not too hard (or it breaks the wall), not too soft (or it doesn't do the job).

The Challenge: Because the arm is soft, it bends when it pushes. The robot doesn't know exactly how much it's bending just by looking at its motors.
The Solution: The robot uses a "force sensor" at its tip (like a sensitive fingertip) to feel the pressure. It combines this feeling with what the camera sees.

3. The "AI Coach" (The Control System)

This is the most important part. The robot has to do two things at once: keep the image steady (so the line stays in the center of the camera) and keep the pushing force steady.

The Problem: The math for a bending, flying snake is a nightmare. Plus, the wind and the camera noise make the data messy.
The Solution: The researchers gave the robot an AI Coach (a Radial Basis Function Neural Network).
- How it works: Imagine a student learning to ride a bike on a windy day. A normal teacher might say, "Pedal harder." But this AI Coach watches the student, feels the wind, sees the wobble, and instantly says, "Lean left a bit more, pedal faster, and don't worry about that gust of wind."
- No Homework: Usually, AI needs to be trained for months in a lab before it works. This AI learns on the fly. It figures out the mistakes and corrects them in real-time, without needing a pre-written manual.

4. The "Fixed-Time" Promise

Most controllers say, "I will get the error to zero eventually." This one says, "I will get the error to zero within a specific, guaranteed amount of time, no matter how bad the start was."

The Analogy: If you are driving a car and you miss a turn, a normal GPS might take a long time to reroute. This new system is like a teleportation GPS that says, "I know you missed the turn, but I guarantee we will be back on the right path in exactly 5 seconds, no matter how far off you are."

Why Does This Matter?

Safety: Because the arm is soft and the controller is smart, the robot can gently touch fragile things (like a flower or a human) without breaking them.
Robustness: It works even if the camera is dirty, the wind is blowing, or the robot starts in a weird position.
Real World: It moves beyond "perfect lab conditions" to messy, real-world environments where things aren't perfectly predictable.

In Summary

The authors built a flying robot with a soft, snake-like arm. They taught it to "see" using lines instead of dots, "feel" with a sensitive touch, and "think" using an AI coach that learns instantly while flying. The result is a robot that can fly up to a wall, gently push it, and hold its position perfectly, even if the wind is blowing or the camera is shaky. It's like teaching a jellyfish to fly and perform delicate surgery at the same time.

1. Problem Statement

The paper addresses the challenges of Aerial Physical Interaction (APhI) using Tendon-Driven Aerial Continuum Manipulators (TD-ACMs). Unlike traditional rigid-arm aerial manipulators (RA-AMs), TD-ACMs possess compliant, deformable structures that offer superior adaptability in cluttered environments and safer contact forces. However, controlling them is significantly more difficult due to:

Complex Dynamics: The continuous deformation of the arm creates strong coupling between shape, force, and motion, making accurate modeling computationally intensive (often requiring Partial Differential Equations).
State Estimation Uncertainty: Unlike rigid arms, TD-ACMs lack direct joint encoders; their state depends heavily on contact forces and aerodynamic effects (e.g., downwash), leading to uncertain localization.
Control Limitations: Existing hierarchical control schemes used for rigid arms fail because arm deformations occur on similar time scales as UAV attitude dynamics. Furthermore, traditional visual servoing often relies on point features or heuristic line extraction, which are sensitive to noise, occlusion, and lighting changes.

The core problem is designing a robust hybrid vision/force controller that can autonomously interact with a static environment, stabilize image feature errors, and regulate contact forces despite modeling inaccuracies, sensor noise, and aerodynamic disturbances, without requiring offline training.

2. Methodology

The authors propose a cascaded hybrid vision/force control framework integrated with AI-based uncertainty estimation. The methodology consists of four main pillars:

A. Modeling in SE(3)

Constant-Strain Model: The authors adopt a constant-strain modeling approach within the Special Euclidean group SE(3). This provides a compact representation that captures vehicle-arm coupling and deformation effects while remaining computationally tractable for real-time control, balancing accuracy better than Piecewise Constant Curvature (PCC) or Cosserat rod models.
Kinematics: The system models the kinematics of the UAV and the continuum arm as a coupled system, discretizing the arm into significant points using Gaussian quadrature.

B. Visual Servoing with Deep Learning

Line-Based IBVS: Instead of point features, the system uses line features for Image-Based Visual Servoing (IBVS). Line features are more abundant in man-made environments and robust to viewpoint variations and partial occlusions.
Deep Feature Extraction: To overcome the computational burden and uncertainty of traditional line extractors (like Hough transforms), the system employs a Graph Neural Network (GNN) (specifically the "Gluestick" framework). This deep learning approach extracts line parameters (angle $\theta$ and distance $\rho$ ) in real-time, leveraging connectivity information to disambiguate line segments.

C. Control Architecture

The controller is a Cascaded Fast Fixed-Time Sliding Mode Controller (CFTSMC) augmented with Radial Basis Function Neural Networks (RBFNN).

Hybrid Loop: The control law simultaneously minimizes the image feature error ( $e_s$ ) and the normal force error ( $e_f$ ).
Uncertainty Estimation: The RBFNNs are used to estimate and compensate for uncertainties in the image interaction matrix and force measurements ( $\Delta_s, \Delta_f$ ) online, eliminating the need for offline training.
Fixed-Time Stability: The controller guarantees that errors converge to zero within a bounded time ( $T_{max}$ ) regardless of initial conditions, ensuring rapid response.
Virtual Camera: To handle UAV underactuation (inability to directly control roll/pitch), a virtual camera frame is constructed where roll and pitch are constrained to zero, effectively decoupling the visual servoing from the UAV's attitude dynamics.

D. Stability Proof

The paper provides a rigorous Lyapunov-based stability proof demonstrating fixed-time stability for the closed-loop system, ensuring that both visual and force tracking errors converge to the origin uniformly.

3. Key Contributions

First Hybrid Control for TD-ACMs: This is the first work to propose a hybrid position/force control strategy specifically for tendon-driven aerial continuum manipulators, addressing the unique localization and modeling challenges of soft aerial robots.
AI-Enhanced Robustness: Integration of a GNN-based line extractor and RBFNN-based uncertainty estimators. This allows the system to handle unstructured environments without markers and learn uncertainties (aerodynamic, modeling, sensor noise) in real-time.
SE(3) Constant-Strain Modeling: A novel formulation of coupled kinematic and dynamic models in SE(3) that balances accuracy and computational efficiency, suitable for real-time implementation.
Fixed-Time Convergence: The design ensures rapid convergence of errors within a fixed time bound, superior to asymptotic or finite-time controllers in terms of predictability and speed.

4. Results

The methodology was validated through extensive software-in-the-loop simulations and hardware-in-the-loop experiments.

Simulation Performance:
- The proposed controller was compared against standard PI/PD and Cascaded Integral Sliding Mode Controllers (CISMC).
- Vision Tracking: The proposed method achieved an RMSE of 0.06, nearly half that of the PD controller (0.12), and improved the Standard Deviation (STD) by 28%.
- Force Tracking: Outperformed CISMC by >50% in RMSE and STD.
- The system successfully tracked time-varying trajectories (e.g., infinity-shaped paths) and maintained stable contact forces (e.g., 2.5 N to 0.5 N) despite initial condition variations.
Experimental Validation:
- Contactless Tests: Demonstrated convergence of image errors to negligible values within ~20–25 seconds using both AprilTag markers and deep feature extraction.
- Push Tests (Ground & Aerial): The system successfully performed contact tasks on vertical surfaces.
  - It maintained low interaction forces ( $< 2$ N), ensuring safety.
  - It handled scene variations (different numbers of line features) and aerodynamic disturbances (UAV downwash) effectively.
  - In aerial tests, despite increased oscillations due to downwash, the controller stabilized errors within 5–7 seconds.

5. Significance

This work represents a significant leap forward in aerial robotics, specifically for soft continuum manipulators.

Safety and Adaptability: By leveraging the compliance of continuum arms and robust control, it enables safe interaction with fragile or deformable objects in hazardous or remote environments.
Autonomy: The removal of reliance on external localization (GPS/SLAM) and offline training makes the system more autonomous and deployable in GPS-denied or unstructured environments.
Robustness: The combination of deep learning for perception and adaptive neural networks for control uncertainty handling sets a new standard for robustness in aerial physical interaction, overcoming the limitations of rigid-arm control strategies.

In summary, the paper presents a comprehensive, theoretically grounded, and experimentally verified framework that enables aerial robots to perform delicate, force-regulated manipulation tasks in complex, real-world scenarios.