💻 computer science

Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation

This paper benchmarks four action spaces for vision-based robotic manipulation, demonstrating through sim-to-real experiments that joint velocity yields the smoothest motion and best task performance for picking and pushing, while offering practical guidance for RL practitioners.

Original authors: Seyed Alireza Azimi, Homayoon Farrahi, Abhishek Naik, Colin Bellinger, A. Rupam Mahmood

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Seyed Alireza Azimi, Homayoon Farrahi, Abhishek Naik, Colin Bellinger, A. Rupam Mahmood

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot arm to perform tasks like picking up a block or pushing it across a table. To do this, you use a method called Reinforcement Learning, which is like a video game where the robot learns by trial and error: it tries something, gets a "score" (reward) if it does well, and learns from its mistakes.

However, there's a tricky part: How do you tell the robot what to do? This is the "action space." It's the language you use to give the robot instructions.

This paper is like a cooking competition where four different chefs (the four action languages) try to teach the same robot how to cook two specific dishes: Picking up a block and Pushing a block. The researchers trained the robot in a virtual kitchen (simulation) and then sent it to the real kitchen to see if the lessons stuck.

Here is how the four "chefs" (action spaces) tried to give instructions:

The "Step-by-Step" Chef (Pose Increment): "Move your hand 1 inch forward, then 1 inch up." It tells the robot exactly where to go next in small steps.
The "Speedometer" Chef (Pose Velocity): "Move your hand forward at 1 inch per second." It tells the robot how fast to move in a specific direction.
The "Joint-by-Joint" Chef (Joint Position Increment): "Bend your elbow a tiny bit, then rotate your wrist a tiny bit." It focuses on moving the individual joints of the robot arm.
The "Flow" Chef (Joint Velocity): "Move your elbow at this speed, and your wrist at that speed." It tells the robot's joints how fast to spin.

The Big Test: Virtual vs. Real Life

The researchers trained all four chefs in a perfect, frictionless video game world (Simulation). In this perfect world, all four chefs did a great job. They all learned the tasks quickly and got high scores. It was hard to tell who was the best.

But then, they sent the robots to the Real World. This is where things got messy. The real world has friction, gravity, and imperfect sensors. The "perfect" instructions from the video game didn't always translate well.

The Results

The "Step-by-Step" and "Speedometer" Chefs (Cartesian/World Space): These chefs struggled the most. When they tried to move the robot's hand in the real world, the robot often got confused. It would drag the block instead of lifting it, or drift into unstable positions and fail completely.
- Why? These chefs rely on a complex translator (called Inverse Kinematics) to turn "move hand here" into "move joints like this." In the real world, tiny errors in the robot's measurements get magnified by this translator, causing the robot to make jerky, inaccurate moves.
The "Joint-by-Joint" Chef: This chef did very well at picking up the block (100% success rate) and was the fastest. However, it required a lot of fine-tuning of the robot's "muscle stiffness" to keep it from shaking or vibrating.
The "Flow" Chef (Joint Velocity): This was the winner.
- Success: It achieved a 100% success rate in picking up the block.
- Smoothness: It moved the most smoothly. The robot's movements were fluid, not jerky.
- Safety: It caused the least amount of "jerk" (sudden, harsh changes in force) and had the fewest collisions.
- Pushing: It pushed the block the furthest distance.

The Secret Sauce: Why Joint Velocity Won

The paper explains this with a simple physics analogy.

Position Control (The other chefs) is like telling a car, "Drive exactly to the 100-meter mark." If you make a tiny mistake in your calculation, the car has to slam on the brakes or floor the gas to correct it immediately. This causes jerky, harsh movements.
Velocity Control (The winner) is like telling a car, "Drive at 50 km/h." The car naturally glides. If you want to change direction, you gently steer. The physics of the robot's joints naturally smooth out the movement, acting like a shock absorber.

The Takeaway

If you are building a robot that needs to see the world with a camera and interact with objects (like picking up a cup or pushing a box), telling the robot's joints how fast to move (Joint Velocity) is the most reliable way to get it to work smoothly in the real world.

While the other methods worked perfectly in the video game, they were too sensitive to the messy reality of the real world. The "Joint Velocity" method was robust enough to handle the imperfections of real life, resulting in a robot that moved gracefully and got the job done.

Technical Summary: Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation

Problem Statement

Reinforcement learning (RL) has emerged as a powerful framework for robotic control, yet its practical application in real-world manipulation faces significant hurdles regarding safety, motion smoothness, and sim-to-real transfer performance. A critical, yet under-explored, design choice in RL formulations is the action space. While prior studies have investigated action-space representations (e.g., joint position, torque, velocity), they have largely relied on state-based observations within simulation-only environments. This leaves a gap in understanding how action-space design impacts learning dynamics and transfer performance when vision-based observations (which introduce partial observability and perceptual noise) are utilized, particularly in contact-rich tasks like picking and pushing.

Methodology

The authors conducted a comprehensive benchmarking study comparing four common action-space representations across two vision-based manipulation tasks on a Franka Emika Panda robot arm:

PandaPickCuboid: A prehensile task requiring the robot to reach, grasp, and lift a red cuboid to a target height.
PandaPushCuboid: A non-prehensile task requiring the robot to push a cuboid across a surface.

Action Spaces Evaluated

The study compared the following representations:

Pose Increment ( $\delta x$ ): Cartesian position and orientation increments.
Pose Velocity ( $\dot{x}$ ): Cartesian velocity commands.
Joint Position Increment ( $\delta q$ ): Increments to joint angles.
Joint Velocity ( $v$ ): Target joint velocities.

System Architecture

Algorithm: Policies were trained using Proximal Policy Optimization (PPO), chosen for its compatibility with high-dimensional visual inputs and demonstrated effectiveness in sim-to-real transfer.
Simulation: Training occurred in MuJoCo, utilizing domain randomization (visual and dynamic perturbations) to bridge the sim-to-real gap.
Real-World Deployment: Policies were deployed via the Franka ROS Interface. The system utilized specific actuation interfaces:
- Position-based actions ( $\delta x, \delta q$ ) were mapped to joint targets via Inverse Kinematics (IK) and executed using a joint impedance controller.
- Velocity-based actions ( $\dot{x}, v$ ) were mapped directly to target joint velocities.
Observations: The agent received an RGB image (64x64), proprioceptive data (joint positions/velocities, grasp status), end-effector height, and the previous action.

Key Results

1. Simulation vs. Real-World Discrepancy

In simulation, all action spaces achieved comparable high success rates (0.98–1.00) for the picking task. However, this performance did not translate linearly to the real world. The choice of action space significantly altered the robustness of the sim-to-real transfer.

2. Performance on PandaPickCuboid (Picking)

Joint Velocity ( $v$ ): Achieved a 100% success rate with a median completion time of 3.58s. It exhibited the lowest measured jerk and consistently avoided collisions.
Joint Position Increment ( $\delta q$ ): Also achieved a 100% success rate with the fastest median time (2.12s). However, stable deployment required careful tuning of impedance gains to prevent vibrations.
Pose Increment ( $\delta x$ ): Performed poorly (41.67% success). The policy frequently dragged the object toward the robot base before lifting, a behavior not observed in simulation.
Pose Velocity ( $\dot{x}$ ): Failed completely (0% success), often drifting into unstable configurations or exiting the workspace.

3. Performance on PandaPushCuboid (Pushing)

Joint Velocity ( $v$ ): Achieved the largest median displacement (0.405m) but exhibited more forceful collisions with the object edges.
Joint Position Increment ( $\delta q$ ): Achieved the second-highest displacement (0.104m) but demonstrated more compliant behavior with fewer forceful contact events due to the tuned impedance controller.
Cartesian Actions ( $\delta x, \dot{x}$ ): Performed poorly in terms of displacement.

4. Analysis of Jerk and Safety

The study analyzed jerk (rate of change of acceleration) as a proxy for motion smoothness and torque variation.

Joint Velocity consistently produced the lowest jerk in both simulation and real-world trials.
Explanation: The authors posit that position-based control (using high proportional gains $K_p$ ) amplifies high-frequency fluctuations in the policy output, leading to large torque variations and high jerk. In contrast, velocity control acts as a low-pass filter; the integration of velocity commands to generate position naturally smooths out high-frequency noise, resulting in smoother trajectories.
IK Sensitivity: Cartesian action spaces rely on Inverse Kinematics (IK). Discrepancies in link parameters between simulation and reality propagate through the IK mapping, generating inaccurate joint targets that real-world actuators struggle to track, especially near singularities.

Key Contributions and Significance

The paper provides practical guidance for RL practitioners by demonstrating that action-space representation is a critical factor in sim-to-real transfer, particularly for vision-based tasks.

Superiority of Joint Velocity: The authors conclude that joint velocity offers the best balance of robustness, smoothness, and task performance for vision-based picking and pushing. It achieved the highest success rates and smoothest motions without requiring the complex impedance tuning necessary for position-based methods.
Limitations of Cartesian Spaces: The study highlights that Cartesian action spaces (pose increment/velocity) are highly sensitive to sim-to-real discrepancies, particularly regarding IK mapping errors and actuator tracking capabilities.
Safety and Compliance: While joint velocity provided the smoothest motion, the authors note that position-based actions (via tuned impedance) offered more compliant behavior during contact, reducing the severity of collisions.

The work underscores that while simulation performance is a necessary baseline, it is insufficient for predicting real-world success. The choice of action space fundamentally shapes the learning dynamics and the physical safety of the deployed policy.