A Dataset and Benchmark for Optical Music Recognition of String Quartet Scores

Imagine you are a librarian trying to save the world's most beautiful, handwritten recipes. But instead of ink and paper, these recipes are written in musical notes on sheet music. For decades, computers have gotten pretty good at reading simple, single-line melodies—like a solo flute player's part. They can look at a picture of the music and turn it into a digital file that a computer can play. This field is called Optical Music Recognition (OMR). It's like teaching a robot to read a songbook so it can sing the tune back to you or help a musician practice.

However, things get messy when you try to read a whole band's music at once. Imagine a string quartet: four different instruments (two violins, a viola, and a cello) playing four different lines of music simultaneously, all packed onto the same page. It's a visual puzzle with notes crossing over each other, different symbols for different instruments, and messy handwriting from centuries ago. Until now, computers have struggled with this "multi-part" music because they didn't have a good practice set to learn from. It's like trying to teach a student to read a complex novel when they've only ever practiced on single-sentence flashcards.

This is where a new team of researchers steps in with a massive new tool called OSSQ-OMR. They built the first-ever dedicated training dataset specifically for these complex string quartet scores. Think of it as a giant, perfectly organized library where every single scanned page of an old score is paired with its perfect digital twin. They didn't just grab random pictures; they spent hundreds of hours manually fixing the digital versions so they matched the old scans exactly, down to every tiny dot and line.

With this new library in hand, the researchers tested two different "robot readers" (computer models) to see how well they could learn. They found that the robots learned best when they were taught to read one instrument's line at a time, rather than trying to swallow the whole page at once. They also discovered that the way the music is written down digitally matters more than the robot's brain design. The best results came from a specific format called LMXE, which achieved a very low error rate of just 3.6% on clean, computer-generated images and 5.9% on real, scanned pages. While this proves that teaching computers to read complex quartet music is definitely possible, the results also show that there is still a lot of room for improvement, especially when dealing with the messy, imperfect reality of old paper scores.

The Big Picture: What They Did and Found

The researchers created OSSQ-OMR, the first dataset designed specifically to help computers recognize music for string quartets. Before this, most computer vision tools for music were trained on simple, single-line songs or piano music. String quartets are much harder because they involve four independent voices playing at the same time, often with complex layouts, different clefs, and handwriting that varies wildly from publisher to publisher.

To build this dataset, the team started with 116 existing digital scores of string quartets. They then hunted down the original scanned versions of these scores from the IMSLP library. Here is the tricky part: the original digital files didn't always match the scans perfectly. Sometimes a measure was missing, or a symbol was in the wrong place. The authors spent over 100 hours manually editing the digital files to make them visually identical to the scanned images. They also organized the data into two levels: system-level (the whole line of music with all four instruments) and staff-level (just one instrument's line). Finally, they converted the music into three different digital text formats: LMXE, kern, and ABC.

The dataset is huge. It contains 24,544 system images and 98,172 staff images derived from those 116 scores. About half of these images are synthetic (clean, computer-generated versions), and the other half are scanned from real, old books.

To test if this dataset actually helps, the researchers set up a "benchmark," which is like a standardized exam for computer models. They tested two popular types of AI models:

1. Zeus: A model based on older, sequential processing (LSTM).
2. SMT: A newer model using a Transformer architecture (similar to the tech behind modern chatbots).

They ran these models on the dataset using different settings to see which combination worked best. Here is what they found:

• Reading one line at a time is better: The models performed significantly better when they were asked to read just one instrument's staff (staff-level) rather than the whole page at once (system-level). Even though reading the whole page gives the computer more context (like seeing what the other instruments are doing), the models didn't seem to use that extra information to make fewer mistakes. In fact, the error rates were lower when the models focused on a single line.
• The format matters more than the brain: The researchers tested nine different ways of turning the music into text tokens (the "words" the computer reads). They found that the LMXE format consistently produced the best results, regardless of which model they used. Surprisingly, the newer Transformer model (SMT) actually did worse on the kern format than on LMXE, even though SMT was originally designed for kern.
• Compression didn't help: They tried using "Byte-Pair Encoding" (BPE) on the ABC format, which is a method to shorten the text by grouping common characters together. They expected this might help, but it actually made the results worse. As the vocabulary size for the compression got larger, the error rate went up.
• Real-world vs. Clean data: The models did much better on the clean, synthetic images (error rates as low as 3.6%) than on the messy, scanned images (error rates around 5.9%). This is expected, as scanned images have noise, shadows, and uneven ink. However, the older-style LSTM model (Zeus) was more robust to this mess than the newer Transformer model (SMT). When switching from clean to scanned data, Zeus's performance dropped by about 39%, while SMT's performance dropped by a massive 100%.

The paper also tested an external model called Legato, which was trained on a different dataset. When they tried to use Legato on this new string quartet data without retraining it, the results were terrible (error rates over 36% on synthetic data and 66% on scanned data). This suggests that just pre-training a model on general music data isn't enough; you need to train it specifically on the type of music you want it to read.

In conclusion, the authors show that teaching computers to read complex string quartet scores is feasible, but it requires the right data and the right way of formatting that data. The best setup they found achieved an error rate of 3.6% on synthetic data and 5.9% on scanned data. While this is a great start, the paper suggests that there is still a lot of work to do to make these systems as good as humans at reading the messy, beautiful chaos of real-world sheet music. They also hint that future research should look at even larger ensembles, like full orchestras, and try to build models that can read entire pages at once without needing to chop them up first.