⚡ electrical engineering

Path Planning and Reinforcement Learning-Driven Control of On-Orbit Free-Flying Multi-Arm Robots

This paper proposes a hybrid control framework that combines trajectory optimization for generating feasible paths and reinforcement learning for adaptive tracking to enable robust, efficient motion planning and stabilization for free-flying multi-arm robots in uncertain on-orbit servicing scenarios.

Original authors: Álvaro Belmonte-Baeza, José Luis Ramón, Leonard Felicetti, Miguel Cazorla, Jorge Pomares

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Álvaro Belmonte-Baeza, José Luis Ramón, Leonard Felicetti, Miguel Cazorla, Jorge Pomares

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to move a giant, floating piece of furniture (like a sofa) inside a spaceship that has zero gravity. You can't just push it; if you push one side, the whole thing spins out of control. Now, imagine that furniture has four robotic arms attached to it, and your goal is to walk this furniture across the floor of the ship, pick up a tool, and move to a new spot, all without bumping into anything or drifting away.

This is the challenge faced by on-orbit robots (robots working in space) that need to service satellites or build space stations. A new paper proposes a clever "two-brain" system to solve this problem, combining perfect planning with instinctive reaction.

Here is how it works, broken down into simple concepts:

1. The Two Brains: The Architect and the Athlete

The researchers created a hybrid system that splits the job into two parts:

Brain A: The Architect (Trajectory Optimization)
Think of this as a super-smart architect who draws a perfect blueprint before you even start moving. It calculates the exact path the robot should take. It figures out:
- How to move the arms.
- How to use the robot's body thrusters (tiny rocket engines) to stay steady.
- Exactly when to grab the wall and when to let go.
- The Catch: The Architect is very smart, but it assumes the world is perfect. It doesn't know what to do if a sudden gust of solar wind pushes the robot or if the robot's sensors are slightly off.
Brain B: The Athlete (Reinforcement Learning)
Think of this as a highly trained athlete who has practiced in a gym with random obstacles, slippery floors, and changing weights. This "athlete" doesn't have a blueprint; it has muscle memory. It watches the Architect's blueprint and tries to follow it. If the robot gets bumped or the math is slightly wrong, the Athlete instantly adjusts its muscles (joints and thrusters) to stay on track. It learns through trial and error, just like a dog learning to catch a frisbee.

2. The Secret Weapon: The "Thruster Backpack"

In the past, space robots tried to move by just wiggling their arms. Imagine trying to walk across a frozen lake by only pushing with your hands; you'd likely spin in circles or slip.

This new robot has a backpack of tiny thrusters (mini-rockets).

The Analogy: Imagine you are walking on a tightrope. If you start to wobble, you don't just try to balance with your arms; you have a friend (the thruster) who gives you a tiny, precise push to keep you upright.
The Result: The robot uses these thrusters to stabilize its body while the arms do the walking. This makes the movement much smoother, safer, and requires less force from the arms, meaning less wear and tear on the robot.

3. The Training Camp: The "Sim-to-Real" Gap

How do you teach the "Athlete" (Brain B) to handle space? You can't just send a robot to space and let it crash a thousand times to learn.

The Virtual Gym: The researchers built a super-realistic video game (a simulation) that looks exactly like space.
The Chaos Factor: To make the robot tough, they made the game "broken." They changed the robot's weight, added fake sensor noise, and made the physics slightly different from the Architect's perfect blueprint.
The Result: When the robot finally goes to the real space station, it's like a boxer who trained in a gym with heavy sandbags and slippery floors. The real world feels easy by comparison. It can handle the "mismatches" between the perfect plan and the messy reality.

4. The Two Big Tests

The team tested this system in two scenarios:

The "Crawl" Test: The robot was already holding onto the spaceship and needed to walk 1.2 meters to a new spot.
- Result: The robot moved smoothly. The thrusters acted like a stabilizer, keeping the robot from drifting while the arms "walked" it forward.
The "Approach" Test: The robot was floating in space, not touching the ship yet. It had to fly close, grab on, and then walk.
- Result: The Architect planned a path to fly close using the thrusters. The Athlete executed the plan, grabbed the ship, and seamlessly switched to "walking mode" without losing its grip or crashing.

Why This Matters

Space is a dangerous, unpredictable place. If a robot relies only on a perfect plan, a tiny error can cause a mission failure. If it relies only on instinct, it might be inefficient or unsafe.

This paper shows that combining a perfect plan with a tough, adaptable instinct is the winning strategy. It's like having a GPS that knows the exact route, but also having a driver who can swerve to avoid a pothole instantly.

In short: This new system allows space robots to move with the precision of a surgeon and the adaptability of a gymnast, making them ready for the complex job of fixing and building things in orbit.

1. Problem Statement

The paper addresses the challenges of motion planning and control for free-flying multi-arm robots in on-orbit servicing scenarios (e.g., satellite servicing, debris removal, in-space assembly). Key difficulties include:

Dynamic Coupling: The movement of robotic arms induces reaction forces and torques on the robot's base, causing unintended drift in microgravity.
Environmental Uncertainty: Space environments are non-deterministic, featuring orbital perturbations, communication delays, and imperfect state estimation.
Model Inaccuracy: Traditional trajectory optimization (TO) relies on precise mathematical models. Simplifications made to these models (e.g., ignoring complex dynamics or contact states) often lead to suboptimal performance or failure when executed in the real world.
High-Dimensional Control: Controlling multiple arms (high degrees of freedom) simultaneously with a mobile base requires managing a complex, high-dimensional action space.

Existing methods often struggle to balance the computational efficiency of optimization-based planning with the robustness required to handle real-world disturbances.

2. Methodology

The authors propose a hybrid framework that integrates Trajectory Optimization (TO) for high-level planning and Reinforcement Learning (RL) for low-level, adaptive control.

A. System Architecture

Robot: A generic multi-arm robot ( $\zeta$ arms, $n$ DoF each) equipped with end-effector docking mechanisms and a set of thrusters on its base for precise body control.
Dynamics: The system models full orbital dynamics, including base-arm coupling, orbital perturbations, and multi-phase contact interactions.

B. Trajectory Optimization (TO) - The Planner

The TO module formulates an Optimal Control Problem (OCP) to generate feasible, efficient reference trajectories.

Parametrization: Uses polynomial splines (third-order) to represent base trajectories, thruster forces, and end-effector paths.
Phases: The motion is divided into contact phases (arms fixed to the target surface) and non-contact phases (arms swinging). The OCP optimizes the duration of these phases (gait) and the timing of docking/undocking.
Constraints:
- Kinematic limits (joint ranges, self-collision avoidance).
- Dynamic constraints (Newton-Euler equations for the base).
- Contact constraints (end-effectors must match the target surface map $m(x,y)$ and have zero velocity during contact).
- Collision avoidance (maintaining a minimum distance $\delta$ from the surface during swing phases).
Objective: Minimize contact forces, base velocities, and fuel consumption while ensuring smooth transitions.
Solver: The continuous OCP is transcribed into a Nonlinear Programming (NLP) problem solved via IPOPT.

C. Reinforcement Learning (RL) - The Controller

The RL module acts as a model-free controller to track the TO-generated reference trajectory, compensating for model mismatches and disturbances.

MDP Formulation:
- Observations: Current base pose/velocity, desired base pose, joint states, previous actions, and current/desired end-effector poses.
- Actions: Target joint positions (converted to torques via a high-frequency PD controller) and thruster force vectors. The action space is 27-dimensional (4 arms $\times$ 6 DoF + 3 thruster axes).
- Reward Function: A composite reward including:
  - Task Rewards: Logarithmic error minimization for base and end-effector tracking.
  - Normalization Rewards: Penalizing high power, acceleration, and thruster usage to ensure smoothness and fuel efficiency.
  - Penalty Rewards: Heavy penalties for collisions (body or non-docking arm contacts).
Training Strategy:
- Curriculum Learning: Starts with small target displacements and gradually increases difficulty as the policy improves.
- Domain Randomization: Randomizes system mass ( $\pm 10\%$ ), initial joint positions, and sensor noise to ensure robustness and sim-to-real transfer.
- Algorithm: Proximal Policy Optimization (PPO) trained in NVIDIA Isaac Sim.

3. Key Contributions

Contact-Aware Trajectory Optimization with Full Dynamics: The authors formulate an OCP that jointly optimizes end-effector trajectories, thruster forces, and contact timing. It captures complete system dynamics, enabling smooth, fuel-efficient behavior suitable for microgravity.
Adaptive Timing and Cost Design: The optimization dynamically adjusts phase durations and penalizes tracking errors and contact forces, allowing for autonomous scheduling of docking actions.
High-Dimensional Policy Learning: Development of an RL policy for a 27-DoF action space (multi-arm + thrusters), managing coupled dynamics and model mismatches in one of the most complex settings in robotics literature.
Robust RL Framework: Implementation of orbital-aware reward shaping, curriculum strategies, and noise injection to ensure generalization under realistic spaceflight uncertainties.

4. Experimental Results

The framework was validated through two case studies in simulation:

Case 1: Surface Locomotion (Initial Contact):
- Task: The robot moves 1.2m (x-axis) and later 1.5m (x) + 0.5m (y) while maintaining contact.
- Findings: The hybrid approach with thrusters significantly outperformed a version without thrusters. Thrusters reduced base oscillations, smoothed acceleration profiles, and drastically lowered the interaction forces required at docking points (improving safety).
- Tracking: The RL policy tracked the TO reference with high precision (mean errors in the range of millimeters to centimeters), successfully handling the full dynamics of the system despite the planner using a simplified model.
Case 2: Free-Flying Approximation (No Initial Contact):
- Task: The robot starts 0.5m away from the target, uses thrusters to approach, docks, and then moves to a target pose.
- Findings: The TO successfully generated a "diagonal approach" maneuver using thrusters to close the gap before switching to arm-based locomotion. The RL policy seamlessly tracked this complex transition from free-flight to contact-based motion.
Comparative Performance:
- The RL-driven controller achieved a mean tracking error of $6.7 \times 10^{-3}$ m, significantly outperforming classical methods like Differential Inverse Kinematics ( $9.2 \times 10^{-2}$ m) and Velocity-based controllers.

5. Significance and Impact

Enhanced Autonomy: The framework enables robots to operate autonomously in uncertain environments by combining the precision of optimization with the adaptability of learning.
Safety and Efficiency: The integration of thrusters reduces reliance on arm-generated reaction forces, minimizing stress on docking points and preventing drift, which is critical for delicate on-orbit servicing.
Scalability: The modular design allows for the addition of more arms or joints with manageable increases in computational load.
Sim-to-Real Potential: By training on a full-dynamics model while planning with a simplified one, and using domain randomization, the system is robust to the "reality gap," making it a strong candidate for future space missions.

In conclusion, this paper establishes a strong foundation for advancing the autonomy of space robotic systems by effectively bridging the gap between precise, task-specific planning and robust, adaptive control in dynamic orbital environments.