💻 computer science

CANTABILE: Learning Expressive Dynamics for Robotic Piano Performance

The paper introduces CANTABILE, a dynamics-aware framework for robotic piano performance that significantly improves expressive accuracy by closing the score-to-contact loop, coupling velocity-fidelity with onset-coverage rewards, and refining policies with finger-only residuals, thereby achieving substantial gains in pitch, onset, and intensity metrics on the EXPRESSIVE-51 benchmark.

Original authors: Woosik Kim, Wonhyeok Choi, Sunghoon Im

Published 2026-09-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Woosik Kim, Wonhyeok Choi, Sunghoon Im

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

For decades, the dream of giving a robot a human touch has centered on the piano. It is a deceptively simple stage for a machine: eighty-eight keys, two hands, and a need for precision that rivals the most delicate surgery. In the world of robotics, playing the piano has become a standard test for dexterity, a way to see if a machine can coordinate two complex hands to strike specific keys at the exact right moment. For a long time, success in this arena was measured by a single, rigid standard: did the robot hit the right notes at the right time? If the robot played the correct melody without hitting the wrong keys, it was considered a success. But this metric missed the soul of the music. A piano is not just a machine that strikes keys; it is an instrument where the volume and emotional weight of a note depend entirely on how fast the hammer hits the string. A gentle touch produces a whisper; a swift strike creates a roar. Until now, robots could play the notes, but they could not play the feeling, because the systems used to train them did not care about the speed of the strike, only the accuracy of the finger placement.

A team of researchers at KAIST and DGIST in South Korea has built a new system called CANTABILE to change that. Their work addresses a fundamental gap in how robots learn to play music. Previous attempts to teach robots piano dynamics often failed because the robots learned a trick: to get a high score for playing the right volume, they simply stopped playing the difficult notes that were hard to control. By omitting the tricky parts, they avoided the risk of making a mistake, resulting in a performance that sounded perfect in the data but was incomplete in reality. The researchers realized that to teach a robot true expression, they had to force it to care about two things at once: hitting every single note on the sheet music and striking each one with the correct intensity. They designed a learning framework that treats the speed of the key press as a primary goal, just as important as hitting the note itself.

The core of their innovation is a method that connects the written music directly to the physical movement of the robot's fingers. In a traditional setup, a computer might tell a robot to press a key, and the robot would do so at a fixed speed. CANTABILE, however, looks ahead at the music score to see what kind of sound is required next. If the score calls for a loud note, the system anticipates this and guides the robot's fingers to move faster. Crucially, the system measures the actual speed of the key the moment it is pressed and translates that physical speed into the digital language of volume. This creates a closed loop where the robot can see the result of its own action. If it strikes too softly, it knows immediately. The researchers also introduced a rule that prevents the robot from skipping notes. The system rewards the robot only if it successfully plays the note and gets the volume right. If the robot skips a difficult note to avoid a volume error, it is penalized. This forces the robot to learn how to control its speed without sacrificing the melody.

To test this approach, the researchers created a new set of fifty-one songs that were specifically chosen for their wide variety of loud and soft passages, a collection they named EXPRESSIVE-51. They compared their new system against the previous best methods, which were excellent at hitting the right notes but terrible at controlling volume. The results were a dramatic shift in capability. The old systems managed to play the correct notes about 78% of the time, but their ability to match the intended volume was almost non-existent, scoring near zero on a scale that measured both timing and loudness together. The new system, CANTABILE, raised this combined score significantly, more than doubling the performance of the previous best method and cutting the error in volume by more than half. The robot was no longer just playing the notes; it was playing the music with the intended dynamic shape, moving from soft whispers to loud declarations as the score demanded.

The researchers also discovered that the way the robot learned was just as important as what it learned. They found that trying to teach the robot everything from scratch—how to move its hands, which keys to press, and how fast to hit them all at once—often led to confusion. Instead, they used a two-step process. First, they trained a "base" robot to play the notes accurately, just like the older systems. Then, they froze that base and added a small, specialized layer of learning that focused only on the fingers. This tiny adjustment layer learned to tweak the speed of the finger strikes to get the volume right, without disturbing the careful hand coordination the base robot had already mastered. This approach allowed the robot to refine its expression without losing its accuracy. Furthermore, they showed that this system could be controlled in real time. By giving the robot a simple command to play "louder" or "softer," the system could adjust the entire performance on the fly, shifting the volume of the music up or down without needing to be retrained.

While the results are impressive, the researchers are careful to note the boundaries of their work. The entire study was conducted in a highly detailed computer simulation, not on a physical piano in the real world. The relationship between the speed of a key and the sound it produces is complex and nonlinear, and the system uses a mathematical approximation to bridge that gap. The team has not yet tested this on a real robot playing a real piano, a step that remains a significant challenge in the field. Additionally, the system currently focuses only on volume; it does not yet control the speed of the music or the way notes are separated, which are other vital parts of musical expression. Despite these limitations, the work demonstrates a clear path forward. By making the robot aware of the physical consequences of its actions and forcing it to balance accuracy with expression, the researchers have moved robotic piano playing from a mechanical exercise into the realm of genuine musical performance. The machine is no longer just hitting the keys; it is learning to listen to the music it is making.

Technical Summary: CANTABILE – Learning Expressive Dynamics for Robotic Piano Performance

Problem Statement
Robotic piano playing has become a standard benchmark for dexterous bimanual manipulation, yet current progress is measured almost exclusively by note accuracy (pitch and onset timing). Standard benchmarks, such as RoboPianist, evaluate performance by determining which keys are pressed and when, rendering every detected keystroke at a fixed MIDI velocity. This approach fails to capture musical dynamics—the pattern of intensity across individual notes—which is essential for expressive performance. In piano acoustics, loudness is determined by hammer impact speed, which is directly induced by the key's angular velocity at the moment of contact. Consequently, existing systems cannot learn or evaluate the ability to vary strike intensity, and optimizing for velocity alone risks "selective omission," where a policy achieves high velocity fidelity by simply avoiding difficult notes rather than playing them correctly.

Methodology
The authors propose CANTABILE, a dynamics-aware framework that closes the loop between score-level dynamics and embodied execution. The system comprises three core components:

Score-to-Contact Dynamics Interface:
- Velocity-Goal Look-ahead: The policy observation is augmented with upcoming velocity goals ( $g_{vel}$ ) derived from the score, allowing the robot to anticipate and adjust motion before key contact.
- Onset and Velocity Sensing: Upon detecting a key press, the system reads the key's angular speed ( $\omega$ ) from the simulator. This is mapped to a realized MIDI velocity ( $\hat{v}_{vel}$ ) using a linear interpolation based on physical parameters (key length, action gain, and maximum hammer velocity), effectively translating physical contact speed into the symbolic MIDI velocity domain.
Coverage-Aware Event Learning:
- To prevent the policy from omitting difficult notes to maximize velocity scores, the framework couples velocity-fidelity with onset-coverage.
- Velocity-Fidelity Reward: Measures the agreement between realized and target velocity for matched onsets using a Gaussian tolerance.
- Onset-Coverage Reward: Rewards the fraction of target onsets realized at the correct step.
- The total reward combines the standard note-accuracy reward with these two new terms, ensuring that dynamics improvements do not come at the cost of note coverage.
Structure-Preserving Residual Control:
- To avoid disrupting the complex coordination required for note selection and hand transport, CANTABILE employs a two-stage, coarse-to-fine control scheme.
- Frozen Base Policy: A dynamics-aware base policy is trained first to handle nominal note execution.
- Finger-Only Residual: A secondary policy refines the base action by applying a bounded, $\alpha$ -scaled correction only to the finger dimensions. This allows the system to adapt strike intensity locally without overwriting the base policy's established coordination.

Key Contributions

Score-to-Contact Interface: A mechanism that conditions control on upcoming velocity targets and maps realized key-onset motion back to MIDI velocity and audio.
Coverage-Aware Learning: A novel objective function that couples velocity supervision with onset coverage to prevent selective note omission.
Coarse-to-Fine Controller: A residual control architecture that refines a frozen base policy with finger-localized corrections, preserving nominal coordination while enabling expressive dynamics.
New Benchmark and Metrics: The introduction of EXPRESSIVE-51, a 51-song subset of the RoboPianist repertoire with significant velocity variation, and Velocity F1, a metric that jointly assesses pitch, onset, and intensity within a $\pm8$ MIDI-velocity tolerance.
Runtime Control: Demonstration that intensity-randomized training enables runtime control of performance intensity without retraining.

Experimental Results
Evaluated on EXPRESSIVE-51, CANTABILE demonstrates significant improvements over the RoboPianist baseline:

Velocity F1: Increased from 0.06 (RoboPianist) to 0.34, a more than five-fold improvement.
Velocity Error: Reduced the matched-note velocity error (MAE) by more than half (from 37.1 to 17.2).
Audio Fidelity: Reduced the log-mel distance to reference audio by 8%.
Generalization: The system successfully followed commanded intensity variations (ranging from 0.5x to 1.5x of written loudness) at runtime.
Ablation Studies: Confirmed that the combination of velocity and coverage rewards is necessary for high performance, and that restricting the residual to finger dimensions yields faster convergence and lower error than expanding the action space to wrists or arms.

Significance and Claims
The paper claims that CANTABILE makes dynamics measurable, learnable, and controllable within a robotic piano setting. By treating velocity as a first-class control objective and decoupling intensity adaptation from nominal note execution, the framework achieves expressive performance without sacrificing the fundamental accuracy of note playing. The authors note that while the current results are in simulation and limited to the velocity dimension (excluding tempo and articulation), the approach of exposing a sensing channel, defining event-level rewards, and attaching a residual may extend to other pretrained dexterous controllers for expressive objectives. The work highlights that dynamics and event coverage are inseparable in both learning and evaluation for embodied musical performance.

Technical Summary: CANTABILE – Learning Expressive Dynamics for Robotic Piano Performance

More like this