💻 computer science

Leakage-Controlled Multimodal Prediction of Piano Performance Expressivity Using Sequential and Engineered MIDI Representations

This paper proposes a Residual Cross-Gated Dual-Branch MIDI Attention Fusion Network (RCG-DMAF-Net) to predict piano performance expressivity by integrating sequential note events with engineered descriptors, demonstrating through a leakage-controlled evaluation that multimodal fusion significantly enhances prediction accuracy over sequence-only models while identifying pedalling, velocity variation, and tempo as key drivers of expressivity.

Original authors: Geping Jin, Dian Jin

Published 2026-10-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Geping Jin, Dian Jin

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Music is more than just the notes written on a page; it is the living breath of a performance, shaped by the subtle variations a musician introduces to convey emotion, tension, and meaning. When a pianist plays a piece, they do not merely hit the keys at the exact moments and speeds indicated by the score. Instead, they stretch time, soften or sharpen the volume of individual notes, and use the sustain pedal to blur or clarify the sound, creating a unique interpretation that feels alive to the listener. For decades, scientists and musicians have tried to teach computers to understand these invisible qualities. The challenge lies in the fact that what we call "expressivity" is not a single physical measurement like temperature or speed. It is a complex blend of timing, dynamics, and texture that emerges from how notes relate to one another over time. While computers are excellent at reading the raw data of a performance—the exact pitch, duration, and force of every key press—translating that data into a judgment of how moving or convincing a performance feels has remained a difficult puzzle.

In a recent study, researchers set out to solve this puzzle by teaching a computer to predict how humans rate the expressiveness of piano performances. They worked with a large collection of recorded performances, each broken down into short segments, and paired them with expert ratings that described how "purely expressive" or "convincing" each performance felt. The team faced a specific hurdle: they needed to ensure the computer was learning to recognize the art of performance, not just memorizing the specific musical pieces it had seen before. To do this, they treated every unique musical excerpt as a distinct family. If a piece of music appeared in the training data, no version of that same piece was allowed in the testing data. This strict separation ensured that the computer had to generalize its understanding to entirely new musical situations, much like a student who must apply a lesson to a new problem rather than reciting a memorized answer.

The researchers built a system that looked at the performances in two different ways simultaneously. First, they fed the computer a detailed, moment-by-moment record of the notes as they happened in time, capturing the flow and sequence of the music. Second, they provided a summary sheet of fifty-one specific characteristics derived from the performance, such as the average speed of the playing, the range of volume changes, and how often the sustain pedal was used. The idea was that the computer could use the summary sheet to get a solid baseline understanding of the performance's overall style, while the moment-by-moment record could offer fine-tuned corrections for specific emotional nuances. They designed a neural network, a type of computer program inspired by the brain, to combine these two streams of information. This network was taught to let the summary sheet make the main prediction and then use the detailed sequence to make small, targeted adjustments, controlled by a mechanism that decided how much influence the sequence should have for each specific type of emotional rating.

When the team tested their system, they found that the most reliable predictor of human judgment was actually the summary sheet of characteristics. Models that relied solely on these fifty-one engineered descriptors, particularly a type of tree-based algorithm known as ExtraTrees, achieved the highest accuracy. These models correctly predicted the human ratings with a level of precision that explained nearly half of the variation in how people perceived the performances. The researchers' complex system, which combined the summary data with the detailed sequence, performed very well and was statistically comparable to the best tree-based models. However, it did not significantly outperform them. The study explicitly ruled out the idea that simply adding more complex sequence data or sophisticated fusion methods would automatically lead to better results. In fact, the most complex models did not show a clear advantage over the simpler, more direct approaches when tested on unseen musical excerpts.

The investigation did reveal, however, that combining the two types of information was far superior to using either one alone. A system that looked only at the moment-by-moment sequence of notes performed much worse than the summary-based models, suggesting that without the broader context of the performance's overall statistics, the computer struggled to grasp the bigger picture of expressivity. The study confirmed that the summary data was the primary driver of success, while the sequence data provided a valuable, though secondary, layer of refinement. By analyzing which features mattered most, the researchers discovered that the computer placed the highest importance on how much the sustain pedal was used and how much the volume of the notes varied. These two factors, along with the timing of the notes and the density of the chords, were the strongest indicators of whether a performance felt expressive or convincing to a human listener.

Ultimately, the work demonstrates that while deep learning systems are powerful, they do not always need to be the most complicated solution to solve a difficult problem. In the realm of musical expressivity, a clear, structured summary of performance traits proved to be the most effective tool for prediction. The study suggests that the key to understanding how a performance feels lies not just in the intricate dance of individual notes, but in the broader statistical patterns of how a musician shapes the sound. The computer learned that the most expressive performances were those where the pianist made deliberate, varied use of the pedal and dynamic shifts, creating a rich texture that human listeners could immediately recognize. This finding offers a grounded path forward for future research, showing that careful engineering of data and rigorous testing protocols are just as important as the complexity of the algorithms used to analyze them.

Technical Summary: Leakage-Controlled Multimodal Prediction of Piano Performance Expressivity

Problem Statement

Predicting the expressivity of piano performance is a complex challenge because perceptual judgment arises from a combination of temporal, dynamic, articulatory, pedaling, and tonal features. Unlike low-level physical parameters, expressivity is not directly measurable and depends on the temporal relations between notes and the global statistical properties of an interpretation.

Existing approaches face specific limitations:

Sequential Models (e.g., LSTMs, Transformers): While effective at capturing local note-event evolution, they often struggle with moderate-sized datasets and may underutilize stable global features.
Engineered Descriptors (Tabular Data): These provide compact summaries of global properties (e.g., tempo distribution, velocity variation) but lose fine-grained temporal ordering.
Data Leakage: A critical issue in music performance modeling is the random splitting of observations. Since multiple performers may play the same musical excerpt, random splitting can lead to "leakage" where the model recognizes specific excerpts rather than learning generalizable performance-to-perception relationships.
Evaluation Rigor: Many studies rely on point estimates without accounting for statistical uncertainty or fail to compare sophisticated neural fusion models against strong tree-ensemble baselines under strict, composition-aware validation protocols.

Methodology

Data Source and Representation

The study utilizes the PercePiano dataset, comprising 1,202 annotated performance segments from 22 musical works, grouped into 99 composition-segment groups. The data is represented using two complementary modalities:

Sequential Representation: Variable-length note-event sequences containing eight attributes (pitch, velocity, onset time, etc.) derived from MIDI files.
Engineered Descriptors: 51 fixed-length, interpretable global features grouped into six categories: general temporal structure, pitch/polyphony, pedaling/tempo, velocity, note duration, and inter-onset intervals.

The primary prediction targets are two high-level perceptual dimensions:

Music Making Pure Expressive
Interpretation Doubtful Convincing

Proposed Architecture: RCG-DMAF-Net

The authors propose the Residual Cross-Gated Dual-Branch MIDI Attention Fusion Network (RCG-DMAF-Net). The architecture is designed to leverage the strengths of both modalities while mitigating the risk of overfitting or noise introduction:

Dual Branches:
- A Tabular Branch processes the 51 engineered descriptors to generate a primary prediction.
- A Sequential Branch uses a Bidirectional LSTM with temporal attention to encode the note-event sequences.
Residual Cross-Gated Fusion: Instead of simple concatenation, the network employs a residual formulation where the sequential branch learns a correction to the tabular prediction.
- A Target-Specific Gate (learned via a sigmoid function) controls the magnitude of the sequence-informed residual correction for each perceptual dimension. This allows the model to adaptively decide how much sequential information is relevant for specific targets.
- The final prediction is the sum of the tabular estimate and the gated residual.

Evaluation Protocol

To ensure rigorous generalization, the study employs a Leakage-Controlled evaluation strategy:

Group-Disjoint Validation: The 99 composition-segment groups are partitioned such that all performances of a specific segment remain in a single partition (training, validation, or test). This prevents the model from memorizing specific musical excerpts.
Locked Evaluation Set: 20 groups (244 observations) were held out as a "locked" blind set, never used for tuning or model selection.
Statistical Uncertainty: Performance differences are assessed using paired group-bootstrap procedures (2,000–5,000 replications) to generate 95% confidence intervals, ensuring that small numerical differences are not overinterpreted.
Comparative Baselines: The proposed model is compared against linear models (Ridge), tree ensembles (Random Forest, ExtraTrees, XGBoost), pure sequence models (BiLSTM, Transformer), and alternative fusion strategies (Simple Fusion, Multi-Task Fusion).

Key Results

Performance on the Locked Set

On the 244 locked observations, Tree Ensembles (specifically ExtraTrees) outperformed the proposed neural model, though the difference was not statistically significant at the 95% confidence level:

Interpretation Doubtful Convincing: ExtraTrees achieved $R^2 = 0.4930$ (RMSE = 0.0894), while RCG-DMAF-Net achieved $R^2 = 0.4765$ (RMSE = 0.0908).
Music Making Pure Expressive: ExtraTrees achieved $R^2 = 0.4979$ (RMSE = 0.0837), while RCG-DMAF-Net achieved $R^2 = 0.4581$ (RMSE = 0.0870).

The 95% bootstrap confidence intervals for the RMSE differences between RCG-DMAF-Net and ExtraTrees included zero, indicating no statistically significant superiority of the neural model over the strong tree-ensemble baseline.

Ablation and Fusion Analysis

Controlled ablation studies on the development set revealed:

Multimodal Advantage: Multimodal fusion significantly improved performance over sequence-only learning (e.g., reducing RMSE from ~0.0975 to ~0.0900).
Fusion Mechanism: The differences between fusion variants (Simple Fusion, Ungated Residual, Gated Residual) were relatively negligible. The full RCG-DMAF-Net did not significantly outperform simpler fusion strategies in terms of accuracy.
Residual Contribution: The target-specific gates had a mean activation of ~0.39, suggesting that the sequential branch provided a partial correction rather than a dominant prediction.

Interpretability (SHAP Analysis)

Using SHAP values on the ExtraTrees baseline, the study identified the most influential features for perceptual prediction:

Top Features: Pedal-down fraction, velocity variation (standard deviation and coefficient of variation), pedal-event count, and onset/tempo structure.
Insight: High-level perceptual judgments are strongly associated with sustained resonance (pedaling) and dynamic shaping (velocity), as well as temporal organization.

Significance and Claims

The paper positions its contribution not as a demonstration that deep learning is superior to traditional machine learning for this specific task, but as a methodological framework for rigorous evaluation in the domain of symbolic music performance.

Leakage-Aware Evaluation: The study emphasizes that without composition-segment-disjoint validation, performance estimates are likely optimistic due to data leakage.
Complementary Representations: It validates the complementary value of combining sequential note-event data with engineered global descriptors, showing that multimodal fusion boosts sequence-only learning even if it does not strictly surpass strong tree ensembles in this dataset size.
Statistical Rigor: The paper advocates for reporting statistical uncertainty (via group bootstrapping) and controlled ablation to distinguish between genuine architectural improvements and random variance, particularly when margins between models are small.
Interpretability: It demonstrates that while neural models are complex, the underlying drivers of expressivity (pedaling, velocity, tempo) can be effectively captured and interpreted using engineered descriptors and SHAP analysis.

In conclusion, the study suggests that for moderate-sized music datasets where tabular representations summarize informative domain knowledge, tree ensembles remain strong contenders. The proposed RCG-DMAF-Net offers a structured, interpretable residual-fusion approach that verifies and interprets perceptual predictions within a strict generalization protocol, rather than claiming a universal performance leap over established baselines.