Technical Summary: Sheet Music Benchmark (SMB) and OMR-NED

1. Problem Statement

Optical Music Recognition (OMR) remains a significant challenge within Music Information Retrieval (MIR). While Deep Learning (DL) has shifted the field from rule-based methods to end-to-end transcription pipelines, the field lacks a comprehensive, high-quality benchmark corpus. Existing datasets suffer from critical limitations:

• Insufficient Size and Diversity: Early corpora like OMRTestCorpus are too small for modern DL training.
• Limited Scope: Datasets like CVC-MUSCIMA focus only on staff-line removal, while PrIMuS is restricted to monophonic, staff-region levels.
• Synthetic Bias: Datasets like GrandStaff are synthetically generated with limited composer diversity.
• Format Constraints: Recent datasets like OLiMPiC rely on Linearized XML, requiring additional processing to render full scores.
• Evaluation Deficiencies: The standard metric, Symbol Error Rate (SER), provides a global error count but fails to offer fine-grained diagnostic insights into specific notation errors (e.g., distinguishing between pitch errors, accidental misclassifications, or beam detection failures).

Consequently, rigorous performance assessment and qualitative comparison between OMR systems are hindered by the absence of a standardized, full-page, end-to-end dataset and a more informative evaluation metric.

2. Methodology and Contributions

2.1 The Sheet Music Benchmark (SMB) Dataset

The authors introduce SMB, a dataset comprising 685 pages (4,039 regions) of sheet music encoded in the Humdrum kern format. The dataset was constructed using the KernScores library, with all annotation steps performed by musically trained experts to ensure high-quality ground truth.

• Annotation Process: Scanned scores were uploaded to HumanSignal. Experts assigned texture tags (Monophony, Pianoform, Quartet, and Other) and delimited regions with bounding boxes. Annotations were provided at the staff level and as "diplomatic" full-page annotations suitable for layout analysis and full-page transcription.
• Encoding Strategy: The dataset utilizes kern for its efficiency and compatibility with the Humdrum ecosystem. To address the ambiguity of raw kern (where symbols can be represented in multiple ways), the authors implemented a post-processing pipeline using the kernpy package. This generated three formats:
  1. raw: Original fragmented encoding from labeling.
  2. kern: Standardized kern with proper headers and spine paths.
  3. ekern (Extended kern): A tokenized version where semantic categories (duration, pitch, accidental) are separated by delimiters ("@" and "·"), significantly reducing vocabulary size at the cost of increased sequence length.
• Data Composition: The dataset spans diverse periods (Baroque to Ragtime) and composers (e.g., Bach, Chopin, Mozart, Joplin). The distribution is dominated by Pianoform (469 pages), followed by Monophony (115), Quartet (79), and Other (22).

2.2 The OMR Normalized Edit Distance (OMR-NED)

To address the limitations of SER, the authors propose OMR-NED, a metric designed to provide both a single numerical performance score and a detailed error profile.

• Mechanism: OMR-NED is built upon an improved version of MusicDiff (originally Music-Score-Diff). Unlike traditional edit distances that operate on character strings, OMR-NED compares visual music symbols.
• Granularity: It categorizes errors into specific notation elements:
  • Notes/Rests: Pitch, accidental, tie, note head, flags/beams, dots, articulations, ornaments, and grace notes.
  • Non-Notes: Dynamics, clefs, key signatures, time signatures, slurs, ottavas, directions, arpeggios, chord symbols, lyrics, and endings.
• Calculation: The metric computes the average number of symbol insertions and deletions required to convert a predicted score to the reference score, normalized by the total symbol count of both scores. Crucially, a substitution is treated as a deletion of the old symbol and an insertion of the new one, rather than a single edit operation.
• Output: The tool generates a CSV report detailing edit distances for each category, allowing researchers to identify specific system weaknesses (e.g., poor beam detection vs. pitch errors).

3. Experimental Results

The authors established a baseline using the Sheet Music Transformer (an encoder-decoder architecture) trained on the SMB dataset using 5-fold cross-validation. The model was trained from scratch (no pre-trained weights) for 400 epochs on an NVIDIA RTX 4080.

• Performance: The results indicate that OMR remains a challenging task, with high error rates even for state-of-the-art models.
  • Monophony: Achieved the lowest OMR-NED (94.1% for kern, 98.8% for ekern) and SER (57.1%).
  • Pianoform: Showed significantly better performance than polyphonic textures, with an OMR-NED of 57.4% (kern) and SER of 31.4%.
  • Quartet & Other: Exhibited higher error rates, with Quartet OMR-NED reaching 92.3% (kern) and SER 39.8%.
• Encoding Impact: The ekern format generally resulted in higher error rates compared to kern in these baseline experiments, likely due to the increased sequence length and complexity of the tokenized vocabulary.

4. Case Study: OMR-NED vs. SER

The paper presents a comparative analysis of two transcription examples to demonstrate the utility of OMR-NED:

1. Monophonic with Lyrics: While SER provided a global error rate, OMR-NED revealed that 65% of errors were in Notes and 35% in Lyrics, allowing for targeted debugging.
2. Pianoform: OMR-NED showed that 88.2% of errors were in Notes, but also highlighted a 11.8% error rate in the Extra category (dynamics, clefs, key signatures). This level of detail is invisible to standard SER, which would only report a single aggregate percentage.

The authors note that OMR-NED tends to penalize errors more heavily than SER due to its fine-grained counting of individual symbols (e.g., treating a time signature change as two edits rather than one substitution).

5. Significance and Claims

The paper positions SMB and OMR-NED as foundational tools to advance the OMR field:

• Standardization: SMB provides the first publicly available, full-page, end-to-end dataset specifically designed for benchmarking modern DL-based OMR systems, addressing the "long-standing gap" in evaluation resources.
• Diagnostic Capability: OMR-NED moves beyond simple accuracy metrics to enable "fine-grained and detailed error analysis." This allows researchers to identify specific notation features (e.g., beams, accidentals) where models struggle, facilitating the development of more robust architectures.
• Community Framework: By providing standardized dataset splits and a novel metric, the authors aim to facilitate clear, comparable performance evaluations among OMR systems, similar to how benchmarks like MAPS or GTZAN have driven progress in other MIR subfields.

The authors conclude that while the baseline results show ample room for improvement, the combination of SMB and OMR-NED establishes a rigorous framework necessary for the systematic evaluation and future advancement of Optical Music Recognition.