⚡ electrical engineering

Velocity Prediction in Automatic Guitar Transcription

This paper presents a methodology for automatic guitar transcription that successfully predicts note velocity by pretraining a model on synthetic data generated with virtual instruments and then transferring those weights to a model trained on real guitar audio, achieving state-of-the-art transcription performance alongside velocity prediction capabilities.

Original authors: Jackson Loth, Xavier Riley, Simon Dixon, Emmanouil Benetos

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jackson Loth, Xavier Riley, Simon Dixon, Emmanouil Benetos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a robot that can listen to a guitar song and write down the sheet music for you. This is called "Automatic Guitar Transcription." For a long time, these robots were really good at figuring out which notes were played (like "C" or "E"), but they were terrible at figuring out how hard the guitarist hit the strings.

In the world of digital music, "how hard you hit" is called velocity. On a piano, this is easy to measure because the computer knows exactly how fast the key was pressed. But on a guitar, it's like trying to guess how hard someone kicked a soccer ball just by listening to the thump—it's vague, and there's no standard way to measure it. Because of this, most guitar-transcribing robots just ignore it.

This paper introduces a clever trick to teach a robot to understand guitar "velocity" (or intensity) without needing a special, expensive guitar that measures force.

The Problem: No Training Manual

To teach a robot anything, you usually need a huge pile of examples where the answer is already known (like a teacher's answer key). For piano, we have these answer keys. For guitar, we don't. We have recordings of real guitarists, but we don't know the exact "velocity" numbers for every note they played.

The Solution: The "Virtual Guitar" Simulator

The researchers decided to build a training simulator instead of using real recordings at first.

The Simulator: They used computer software (virtual instruments) that acts like a guitar. They told the computer, "Play this note at velocity 50," and the computer generated a sound file. Because the computer generated the sound, it knew the exact "velocity" number for every single note.
The First Lesson: They taught their robot model using this fake, synthetic data. The robot learned to look at the sound waves and guess, "Ah, this sounds like a velocity of 50."
The Transfer: Once the robot learned the concept of intensity from the fake data, they took that specific part of its brain (the "velocity weights") and moved it to a new model.
The Real Lesson: They then taught this new model using real recordings of actual guitarists (from datasets like GAPS and GOAT). They told the robot, "Forget the fake numbers now; just learn to recognize the real notes and strings." But, they kept the "velocity brain" they learned in step 2 frozen in place.

Think of it like teaching a student to drive. First, you let them practice on a driving simulator where you know exactly how fast they are going. Once they understand the concept of speed and braking, you put them in a real car. You don't need to tell them how fast they are going anymore; they already have the "feel" for it, and now they just need to learn how to handle the real road.

What Did They Find?

The researchers tested their robot in two ways:

1. Can it guess the intensity on the fake data?
Yes, and it was very good at it. When tested on the synthetic data (where the "correct" answer was known), the robot made very small mistakes compared to a robot that didn't get the special training. It successfully learned the "personality" of the virtual guitar.

2. Does this help it transcribe real guitar songs?
This is the tricky part. Since we don't have a "correct answer key" for real guitar intensity, they couldn't prove the robot was guessing the exact right number for real songs. However, they did find a small side benefit:

The robot that had the "velocity training" was slightly better at figuring out when the notes started and stopped (the timing) compared to the robot that didn't have that training.
The improvement was tiny (about 0.1%), but it was there. It's like saying, "Knowing how hard the ball was kicked helped the robot guess the ball's path just a tiny bit better."

The Bottom Line

The paper claims they successfully built the first guitar transcription model that can predict "velocity" (intensity). They did this by:

Creating a fake dataset where the intensity numbers were known.
Teaching the model on that fake data.
Transferring that knowledge to a model trained on real guitar audio.

The Limitations:
The authors are honest about the flaws. Because "velocity" on a guitar isn't a hard number like it is on a piano, they can't prove their robot is 100% right about the intensity of real songs. They are essentially guessing based on how loud and bright the note sounds, which is a good approximation, but not a perfect measurement.

In short, they taught a robot to "feel" the guitar's intensity using a video game simulator, and that "feeling" helped it play the real game just a little bit better.

Technical Summary: Velocity Prediction in Automatic Guitar Transcription

Problem Statement
Automatic Music Transcription (AMT) has achieved significant success in polyphonic pitch transcription for various instruments, including the guitar. However, the prediction of note velocity (intensity/dynamics) remains underexplored in Automatic Guitar Transcription (AGT). Unlike piano, where velocity is a precisely measured physical value (e.g., via Disklavier), guitar velocity is an abstract concept related to both loudness and timbre, lacking a standardized definition or ground-truth labels in existing datasets. Consequently, most AGT models ignore velocity to focus solely on pitch, onset, and offset. The primary challenge is the absence of real-world guitar datasets containing accurate velocity annotations, which hinders the training and evaluation of deep learning models for this task.

Methodology
The authors propose a two-stage training methodology that leverages synthetic data to overcome the lack of real velocity labels:

Synthetic Data Generation:
- The authors utilized the FrançoisLeduc dataset, which contains aligned real guitar audio and MIDI annotations.
- They estimated velocity values for each note by isolating audio segments, applying bandpass filters centered on fundamental frequencies and harmonics, and computing total energy with exponentially decaying weights.
- These estimated values were normalized to the MIDI range (0–127) and applied to the MIDI annotations.
- Synthetic audio was rendered using five different timbres from three guitar virtual instruments (Native Instruments Session Guitarist and Straight Ahead Samples), creating approximately 20 hours of training data where the "true" velocity is defined by the virtual instrument's internal velocity-to-timbre mapping.
Model Architecture and Training Strategy:
- The base model is a Convolutional Recurrent Neural Network (CRNN) adapted from the High Resolution Piano Transcription model (Kong et al.), designed to predict onset, offset, frame activation, and velocity across 10-second log mel-spectrogram inputs.
- Stage 1 (Pretraining): The model is trained on the synthetic dataset using a loss function ( $L_A$ ) that includes terms for onset, offset, frame, and velocity. The goal is to learn the specific velocity curve defined by the virtual instruments.
- Stage 2 (Fine-tuning): The weights from the velocity submodule of the Stage 1 model are transferred, loaded, and frozen. The model is then fine-tuned on real guitar datasets (GAPS and GOAT) using a modified loss function ( $L_B$ ) that excludes the velocity term. This prevents the model from learning incorrect velocity patterns from real data (which typically have constant velocity labels) while retaining the learned velocity prediction capability.
- Data augmentation (peak filtering, reverb, impulse responses) was applied during training.

Key Contributions

First AGT Velocity Model: To the authors' knowledge, this is the first work to extend guitar transcription to include velocity prediction.
Synthetic-to-Real Transfer: A novel methodology for training velocity prediction on guitar by pretraining on synthetic data with defined velocity curves and transferring those weights to a model trained on real audio.
Evaluation Framework: The establishment of evaluation protocols for guitar velocity using synthetic test splits (song-based and timbre-based) to assess generalization, given the lack of ground truth in real data.

Results

Velocity Prediction: On the synthetic FrançoisLeduc dataset, the proposed model significantly outperformed a baseline (without pretrained velocity weights).
- Song Split: Mean Absolute Error (MAE) reduced from 32.39 (baseline) to 7.04.
- Timbre Split: MAE reduced from 33.59 to 11.53.
- The model demonstrated the ability to generalize to unseen songs and, to a lesser extent, unseen timbres, though performance degraded slightly when the velocity definition differed between training and testing virtual instruments.
Note Transcription (Onset/Offset): The integration of pretrained velocity weights yielded a marginal improvement in note transcription metrics (Precision, Recall, F1) on real datasets (GuitarSet and EGDB).
- Improvements were approximately 0.1% across metrics.
- Statistical significance (paired t-test) was found for GuitarSet metrics but was inconsistent for EGDB (significant for precision, but not recall or F1).
- The authors note that the magnitude of improvement is small and not always significant depending on the test data.

Significance and Claims
The paper claims that the proposed model achieves results comparable to the state-of-the-art in guitar transcription while successfully adding velocity prediction capabilities. The authors emphasize that the primary value of this work is the successful demonstration of a methodology to predict velocity in the absence of ground-truth labels.

The authors remain modest regarding the impact on note transcription, acknowledging that the performance gains are minimal and that future improvements might be better achieved through architectural changes or data augmentation rather than velocity prediction alone. They conclude that while the current approach relies on synthetic definitions of velocity, it opens new possibilities for expressive analysis and educational tools in Music Information Retrieval (MIR), provided future work addresses the fundamental limitation of defining and measuring velocity in real guitar performance.

The Problem: No Training Manual

The Solution: The "Virtual Guitar" Simulator

What Did They Find?

The Bottom Line

More like this