Learning to Play Piano in the Real World

1. Problem Statement

The paper addresses the grand challenge of achieving human-level manipulation in robotics. Specifically, it targets the task of playing the piano, which requires:

• Contact-rich interactions: Precise control of forces and timing.
• Dexterity: Coordinated movement of multiple fingers.
• Flow: Smooth, continuous motion rather than discrete steps.
• Sim2Real Gap: A significant discrepancy exists between simulation (where most learning happens) and the physical world. Previous works either relied on hand-designed controllers for real robots or trained Reinforcement Learning (RL) agents only in simulation. There was no prior system that successfully learned piano playing via RL and deployed it on a physical dexterous robot.

2. Methodology

Hardware Setup

• Robot: An Ufactory xArm7 robot arm equipped with an Allegro Hand v4.0 (4 fingers, 4 joints each).
• Modifications: The thumb is disabled (cannot reach keys reliably). The fingertips are replaced with smaller, 3D-printed tips to allow pressing single keys without triggering neighbors.
• Instrument: An M-Audio Keystation 49e MIDI keyboard (4 octaves).
• Sensing: The system relies on proprioception (joint positions) and MIDI feedback (which keys are pressed). It notably lacks tactile sensors or vision during the primary experiment, relying on the robot "blindly" trusting its training.

Learning Framework: Sim2Real2Sim

The core innovation is an iterative Sim2Real2Sim pipeline:

1. Initial Simulation: Train a policy in a physics simulator (Mujoco-based, using the RoboPianist-Suite).
2. Real World Deployment: Deploy the policy on the physical robot to collect real-world data (observations of key presses and joint behaviors).
3. Simulation Refinement: Use Bayesian Optimization to tune the simulator parameters (e.g., friction, damping, key stiffness) based on the real-world data to minimize the Sim2Real gap.
4. Retraining: Retrain the policy in the refined simulator and repeat the cycle.

Reward Function Design

The total reward ($r_{total}$) is a weighted sum of five components:

• $r_{energy}$: Penalizes fast, forceful movements to prevent oscillation and hardware damage.
• $r_{\Delta-action}$: Penalizes large differences between consecutive actions to ensure smooth control signals.
• $r_{keypress}$: Rewards pressing the correct keys.
• $r_{optimal-transport}$: Encourages efficient fingering by minimizing the horizontal distance the fingertip must travel to reach the target key (avoiding unnecessary finger switching).
• $r_{collision}$: A binary reward (0.5 or 0) to penalize collisions between the hand and the piano body (excluding fingertips), forcing the agent to lift fingers correctly during transitions.

Observation Space

The agent observes a 369-dimensional state vector, including:

• Current joint positions and velocities of the hand and forearm.
• One-hot encoding of currently pressed keys (49 keys).
• One-hot encoding of the target music sheet (6 octaves $\times$ 49 keys).
• Note: The system does not use manual fingering annotations; the agent must learn optimal fingering strategies autonomously.

Training Details

• Algorithm: DroQ (Dropout Q-functions), an off-policy Soft Actor-Critic (SAC) variant.
• Domain Randomization (DR): Applied during training to randomize parameters like key height, starting position, joint damping, and friction to improve robustness.
• Constraints: Fixed "fences" (boundaries) were added between white keys in the simulation to mimic the physical separation of keys, preventing the soft-body physics from accidentally triggering adjacent keys.

3. Key Contributions

1. First Learned Real-World Piano System: Demonstrated the first RL-based system capable of playing piano on a physical dexterous robot without pre-programmed movements.
2. Sim2Real2Sim Pipeline: Validated an iterative approach where real-world data is used to update the simulator, significantly reducing the Sim2Real gap compared to standard Sim2Real transfer.
3. Physical Constraints in Simulation: Proved that adding "fences" between keys in the simulator improves training convergence and real-world performance by mimicking physical key separation.
4. Open Source: Released the codebase and video demonstrations to lower the barrier for future research in robotic manipulation.
5. Benchmarking: Established a baseline (average F1-score of 0.881) for evaluating future piano-playing and dexterous manipulation approaches.

4. Experimental Results

Performance Metrics

The success of the agent is measured using the F1-score (harmonic mean of Precision and Recall):

• Precision: Ability to avoid pressing wrong keys.
• Recall: Ability to press all required correct keys.

Key Findings

• Songs Played: Successfully played Are You Sleeping, Happy Birthday, Ode To Joy, Twinkle Twinkle Little Star, and the C-Major Scale.
• Scores:
  • Simulation: Average F1-score of 0.946.
  • Real World: Average F1-score of 0.881.
  • Sim2Real Gap: Reduced to an average of 6.8% (compared to 11.2% in a fence-free simulation).
• Impact of "Fences": Adding boundaries between keys in simulation reduced the Sim2Real gap by nearly half, as it forced the agent to learn to lift fingers completely between notes, mirroring real-world physics.
• Impact of Collision Penalty: The collision reward term accelerated learning by discouraging the agent from exploiting soft-body physics to press keys sideways, though it did not change the final converged strategy.

Ablation Studies (Appendix)

• Execution Modes: The authors compared Joint Mirroring (simulation only), Real World Execution (full real-world obs), and Hybrid Execution (real-world key states + simulation joint states). Hybrid execution performed best, allowing the robot to correct key errors while maintaining the stability of simulated joint trajectories.
• Domain Randomization (DR): Moderate DR intensity ($C_{dr} \approx 0.6$) yielded the best real-world performance. Too little DR led to poor robustness; too much DR degraded the model's ability to learn the task.

5. Significance and Future Work

• Benchmark for Manipulation: The paper argues that piano playing is a superior benchmark for human-level manipulation compared to tasks like in-hand rotation, as it requires natural, anthropomorphic movements rather than optimized, non-human strategies.
• Limitations:
  • Hardware: The Allegro hand lacks finger abduction (spreading fingers), limiting reach. The wrist orientation is fixed.
  • Sensing: Lack of tactile sensors and vision makes the system vulnerable to misalignment; it relies on "blind" execution.
  • Complexity: Currently limited to simple, single-hand songs.
• Future Directions:
  • Integrating smaller tactile sensors (e.g., next-gen DIGIT) and vision for closed-loop correction.
  • Developing bimanual (two-handed) playing capabilities.
  • Tackling advanced songs with faster tempos and complex fingering.

In conclusion, this work provides a robust proof-of-concept that Sim2Real2Sim, combined with careful reward design and physical constraints in simulation, can bridge the gap between simulated learning and real-world dexterous manipulation, paving the way for more complex robotic tasks.