💻 computer science

Rainbow Deep Q-Learning with Kinematics-Aware Design for Cooperative Delta and 3-RRS Parallel Robot Insertion

This paper proposes a kinematics-aware framework that optimizes the geometry of a cooperative Delta and 3-RRS parallel robot system to maximize its safe workspace, then employs a two-stage curriculum-trained Rainbow Deep Q-Network to achieve robust and reliable peg-in-hole insertion with fewer constraint violations than baseline methods.

Original authors: Hassen Nigatu, Gaokun Shi, Jituo Li, Wang Jin, Lu Guodong

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hassen Nigatu, Gaokun Shi, Jituo Li, Wang Jin, Lu Guodong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a very tricky puzzle: you have a peg (like a large nail) that needs to be pushed into a hole on a dome-shaped object. But there's a catch: you can't just use one hand. You need two robotic arms working together perfectly.

One arm is a Delta robot (think of it as a high-speed spider with three legs) that holds the peg and moves it up, down, left, and right. The other arm is a 3-RRS robot (a different kind of mechanical arm) that holds the dome with the holes and tilts it to line up the hole with the peg.

The problem is that these two robots have to move in perfect sync. If the dome tilts too much, the robot holding the peg might get stuck or break. If the peg moves too fast, it might crash into the side of the hole. This is a "cooperative insertion" task, and it's incredibly hard for standard computer programs to figure out on the fly.

Here is how the authors of this paper solved it, broken down into simple steps:

1. Designing the "Gym" Before the "Athlete" Trains

Before they even started teaching the robots how to move, the authors realized they needed to build a better "gym" for them to practice in.

The Analogy: Imagine training a gymnast. If the balance beam is wobbly or has a narrow path where they can easily fall, they will fail a lot. But if you design a wider, more stable beam, they can practice more safely and learn faster.
What they did: They mathematically redesigned the shape of the second robot (the 3-RRS) before training began. They tweaked its geometry to make its "safe zone" (where it can move without getting stuck or breaking) much larger. This gave the learning robot a bigger playground to explore without crashing.

2. The "Super-Student" Robot Brain

Once the "gym" was ready, they taught the robots using a special type of Artificial Intelligence called Rainbow Deep Q-Learning.

The Analogy: Think of a regular robot learning as a student who only reads one textbook and makes random guesses. The "Rainbow" robot is like a super-student who has six superpowers:
1. Double Check: It doesn't trust its first guess; it double-checks its own predictions to avoid overconfidence.
2. Focus: It pays extra attention to the lessons where it made big mistakes (so it learns faster).
3. Memory: It remembers not just the last step, but a whole sequence of steps to understand cause and effect better.
4. Curiosity: Instead of just guessing randomly, it has a built-in "curiosity" that helps it try new things in a smart way.
5. Value vs. Advantage: It separates "how good is this situation?" from "how good is this specific move?" to make smarter decisions.
6. Distributional Thinking: Instead of guessing a single number for how good a move is, it guesses a range of possibilities, making it more robust.

3. The Training Process

The robots learned through a "curriculum" (a step-by-step school system):

Stage 1: They started with an easy version of the puzzle (only 4 holes to fill).
Stage 2: Once the robots got good at the easy version (75% success rate), they moved to the hard version (all 6 holes).
The Reward System: The robots were given points for getting closer to the hole and huge bonuses for successfully popping the peg in. They were heavily penalized (lost points) if they hit the walls or got stuck in a "singularity" (a mechanical dead-end where the robot loses control).

4. The Results

The paper tested this system in a high-fidelity computer simulation.

The Winner: The "Rainbow" robot with the pre-optimized design was the clear champion.
The Stats: It succeeded 95% of the time.
The Losers:
- A "Vanilla" robot (using standard AI without the superpowers) only succeeded about 68% of the time.
- A "Classical" robot (using old-school math planning without learning) only succeeded 55% of the time.
Why it mattered: The optimized design meant the robots spent less time crashing and more time learning. The "Rainbow" brain learned faster and made fewer mistakes than the others.

Summary

In short, the authors didn't just throw a complex problem at a smart AI and hope for the best. They first engineered a better physical robot to make the job easier, and then they used a super-charged AI brain to learn how to do the job quickly and safely. The result was a system that could cooperatively insert a peg into a hole with near-perfect accuracy in a simulation.

Note: This paper is strictly about a computer simulation. The authors have not yet tested this on real physical robots or in real-world applications like surgery or factory assembly, though that is a logical next step they mention for the future.

Technical Summary: Rainbow Deep Q-Learning with Kinematics-Aware Design for Cooperative Delta and 3-RRS Parallel Robot Insertion

Problem Statement
The paper addresses the challenge of cooperative peg-in-hole manipulation using a heterogeneous pair of parallel robots: a Delta manipulator (providing three translational degrees of freedom) and a 3-RRS (Revolute–Revolute–Spherical) manipulator (providing two rotational and one vertical translational degree of freedom). The task requires precise synchronization to insert a peg into a target hole on a dome-shaped platform carried by the 3-RRS. The authors identify three primary challenges: (1) the need for precise temporal alignment of translation and rotation under dynamic conditions; (2) the requirement for adaptation to changing relative poses without explicit reprogramming; and (3) the necessity of maintaining safe motion within a confined workspace while respecting kinematic constraints, specifically avoiding singularities and workspace boundaries which complicate control for parallel mechanisms.

Methodology
The proposed solution is a "kinematics-informed" Deep Reinforcement Learning (DRL) framework that integrates a geometric design-optimization stage with a Rainbow Deep Q-Network (DQN).

Geometric Optimization Stage: Before learning begins, the geometric parameters of the 3-RRS manipulator are optimized. Using a gradient-free simplex method (Nelder–Mead), the authors maximize the area of the singularity-free orientation workspace and improve the condition number of the Jacobian. This stage enlarges the safe region available for the reinforcement learning agent to explore, reducing the likelihood of episodes terminating due to kinematic violations.
Markov Decision Process (MDP) Formulation: The cooperative task is modeled as a finite-horizon MDP.
- State Space: A 12-dimensional vector comprising the Delta platform's Cartesian pose, the 3-RRS platform's orientation (roll/pitch) and vertical position, the relative displacement between the peg tip and target hole, and the target hole's normal vector.
- Action Space: A discrete set of 12 actions, consisting of positive and negative incremental commands for each of the 6 controlled degrees of freedom (3 translations for Delta, 2 rotations and 1 vertical translation for 3-RRS).
- Reward Function: A shaped reward combines dense proximity guidance (distance to target, incremental progress), penalties for workspace or inverse kinematic (IK) violations, and sparse bonuses for successful insertions.
Learning Algorithm: The authors employ Rainbow DQN, which integrates six extensions to standard DQN: Double Q-learning (to reduce overestimation bias), a Dueling architecture (separate state-value and advantage estimation), Prioritized Experience Replay (for sample efficiency), Multi-step returns (faster reward propagation), Noisy Linear Layers (for state-dependent exploration without $\epsilon$ -greedy schedules), and a Distributional Value Head.
Training Protocol: A two-stage curriculum is used. The agent first trains on a subset of four peripheral holes (Stage C0) and advances to all six holes (Stage C1) once the moving-average success rate exceeds 75%.

Key Contributions
The paper outlines three primary contributions:

Sequential Co-Design Workflow: A methodology where the 3-RRS geometry is optimized to maximize the singularity-free workspace and improve conditioning prior to DRL training. This enlarges the safe exploration region for the heterogeneous Delta + 3-RRS pair.
Task-Specific Rainbow DQN Formulation: A unified framework featuring a 12-dimensional state, a discrete action set of 12 incremental commands, and a reward function that explicitly incorporates kinematic knowledge (workspace constraints and IK validity) for dense shaping.
Comprehensive Simulation Study: An ablation study evaluating the impact of individual Rainbow DQN components and the geometric optimization stage, benchmarked against a Vanilla DQN and a classical sampling-based planner (RRT-Connect).

Results
Validation was performed in a high-fidelity kinematic simulator using a custom Python environment. The results, averaged over 100 test episodes with 5 seeds, demonstrate:

Performance Superiority: The Rainbow DQN approach achieved a 95% ± 2% success rate, significantly outperforming the Vanilla DQN (68% ± 5%) and the classical RRT-Connect planner (55% ± 4%).
Efficiency: The proposed method reduced the average completion time to 7.1 seconds (vs. 9.8s for Vanilla DQN and 12.4s for the planner) and achieved a lower RMS trajectory error (1.5 mm).
Ablation Findings:
- Removing Prioritized Replay or the Dueling Head resulted in the most significant performance drops (success rates falling to 82% and 85%, respectively).
- The Geometric Optimization stage was shown to reduce the number of environment steps required to reach 75% success by approximately one-third (from $11.2 \times 10^5$ to $7.6 \times 10^5$ steps) and completely eliminated singularity-related terminations during training.
Robustness: The policy maintained an 88% success rate when subjected to Gaussian noise (1% of normalized state range) during testing.

Significance and Claims
The paper claims that the integration of geometric optimization with kinematics-informed DRL provides a robust solution for cooperative manipulation in constrained, singular-prone environments. The authors assert that the pre-training geometric optimization is not merely a design step but a critical enabler that reduces sample complexity and prevents training failures caused by kinematic singularities.

The work positions itself as a step toward adaptive, data-driven policy synthesis for multi-robot systems where explicit planning pipelines struggle with dynamic coordination and tight tolerances. The authors explicitly note limitations, including the reliance on simulation (lack of hardware validation), the use of a discrete action space which may limit trajectory smoothness, and the centralized nature of the controller. They identify hardware validation, continuous-action extensions (e.g., SAC, PPO), and decentralized multi-agent formulations as necessary future directions.

1. Designing the "Gym" Before the "Athlete" Trains

2. The "Super-Student" Robot Brain

3. The Training Process

4. The Results

Summary

More like this