MusicSynth: An Automated Pipeline for Generating Violin Fingerboard Animations from Sheet Music Using Optical Music Recognition

MusicSynth is an open-source, browser-based pipeline that automatically generates violin fingerboard animation tutorials from sheet music images or digital files by integrating optical music recognition, MusicXML parsing, and a custom note-to-position lookup table.

The Gist

Imagine trying to learn to play the violin. Unlike a piano, where you just press a specific key, or a guitar, where you press a string against a metal fret, the violin neck is completely smooth and blank. There are no markings to tell you where to put your fingers. If you are off by even a tiny fraction of a millimeter, the note sounds wrong. For a beginner, this is like trying to find a specific house in a city with no street signs or house numbers.

MusicSynth is a new, free web tool designed to act as those missing street signs.

Here is how it works, broken down into simple steps:

1. The Problem: The "Blank Map"

When a student looks at a sheet of music, they see notes, but they don't know where on the violin to put their fingers. Traditional books show the notes but not the hand position. Apps exist that listen to you play, but they can't look at a picture of a song and tell you how to play it.

2. The Solution: A "Magic Translator"

MusicSynth is like a translator that speaks two languages: Sheet Music and Finger Placement.

• You upload: A photo of a violin song (taken with your phone) or a digital file of the music.
• It processes: The system reads the music, figures out every single note, and then instantly calculates exactly which string and which finger you need to use.
• You get: A video that plays right in your web browser. It shows a diagram of the violin neck, lighting up the correct spot for every note as the music plays. You can download this video to practice anytime, anywhere.

3. How It's Built: The "Assembly Line"

The author didn't invent new technology from scratch. Instead, they built a pipeline that connects three existing tools, like connecting three different train cars to make one long train:

1. The Scanner (OMR): This is the "eyes." It looks at your photo of the sheet music and turns the picture into a list of notes.
2. The Brain (The Lookup Table): This is the "secret sauce" the author built. It's a massive rulebook that says, "If the note is 'A', put your first finger on the D string." It follows standard teaching rules used by violin teachers for years.
3. The Animator: This is the "artist." It takes the list of notes and draws a video frame-by-frame, showing a red dot for the current note and blue dots for the notes coming up next.

4. How Well Does It Work?

The author tested this tool on 110 different violin songs ranging from easy beginner pieces to harder advanced ones.

• If you upload a digital file: It is almost perfect. It gets the finger placement right 99.1% of the time.
• If you upload a photo: It is very good at reading clean, printed music, getting the notes right about 91% of the time.
• The "Look-Ahead" Feature: The video doesn't just show the current note; it shows the next few notes in small blue circles, giving you a "heads-up" so you can prepare your hand before the music gets there.

5. What It Can't Do Yet (The Limitations)

Like any new tool, it has boundaries:

• It's for beginners and intermediates: It covers the first few years of learning (the "first position"). If a song requires moving your hand way up the neck to play very high notes, the tool might skip those notes rather than guessing.
• It needs clear photos: If the sheet music is handwritten, very blurry, or has many notes playing at once (like a complex orchestra score), the "scanner" might get confused.
• It's not live: You can't play the violin while the computer watches you. You have to watch the video and practice along.

The Bottom Line

Before this tool, if you wanted to turn a picture of sheet music into a finger-placement guide, you would have to do it manually or use a combination of different, expensive programs. MusicSynth is the first free, browser-based tool that does the whole job in one click. It turns a confusing image of music into a clear, animated video tutorial, helping beginners bridge the gap between reading the music and actually playing the instrument.

Technical Summary

Technical Summary: MusicSynth

Problem Statement
Learning the violin presents a unique pedagogical challenge compared to fretted or keyed instruments: the violin neck lacks physical markers, requiring beginners to develop precise spatial intuition for finger placement simultaneously with score-reading skills. Existing educational tools fail to bridge this gap effectively. Traditional method books provide notation without visual finger guides, while feedback apps like Yousician require live audio input rather than processing arbitrary sheet music. Furthermore, no freely available, browser-based tool currently exists that can automatically convert a sheet music image into an animated fingerboard tutorial without requiring software installation or manual note entry.

Methodology
MusicSynth is an open-source web application that integrates three existing open-source libraries into a unified pipeline to automate the creation of violin fingerboard animations. The system accepts two input types: a photograph of sheet music (PNG/JPG) or a digital MusicXML file.

1. Optical Music Recognition (OMR): For image inputs, the system utilizes Oemer, an open-source OMR library. Oemer processes the image to detect staff lines, note heads, and rhythmic symbols, outputting the data in MusicXML format. If the image quality is insufficient, the system prompts the user to upload a cleaner scan or a digital file.
2. Parsing and Data Extraction: Whether the input is a raw MusicXML file or one generated by Oemer, a Python-based parser extracts note pitch, duration, and start time. Rests are processed to maintain accurate timing without generating visual output.
3. Fingerboard Lookup (Core Contribution): The system employs a custom-built lookup table mapping musical notes (MIDI range G3–G6) to specific violin strings (G, D, A, E) and finger positions (0 for open string, 1–3 for first position). This table adheres to standard violin pedagogy (e.g., Suzuki method), prioritizing fingerings that minimize string crossings and hand shifts. Flat notes are normalized to their sharp equivalents to simplify the lookup logic.
4. Video Rendering: Using the PIL (Python Imaging Library) and MoviePy, the system generates a frame-by-frame video. Each frame displays a fingerboard diagram with:
   • Four string lines.
   • Position markers.
   • A large red circle indicating the current note (with note name and position label).
   • Smaller cyan circles indicating upcoming notes (look-ahead cues).
   • A timestamp and title.
     The final output is an MP4 video encoded at 30 fps.
5. Deployment: The application is built with Streamlit and uses Supabase for user authentication. It manages temporary files via unique session IDs to prevent conflicts. In cloud environments with read-only filesystems, the OMR step is disabled, accepting only MusicXML inputs.

Key Contributions

• First-of-its-Kind Pipeline: MusicSynth is the first publicly available, browser-based tool to automatically generate animated violin fingerboard tutorials from sheet music images without manual intervention.
• Pedagogical Lookup Table: The project introduces a documented, extensible note-to-finger lookup table covering the standard beginner range (G3–G6) based on established violin pedagogy.
• End-to-End Integration: It successfully integrates OMR, MusicXML parsing, and video rendering into a single, user-friendly workflow.
• Benchmarking: The system provides a quantitative evaluation across 110 public-domain scores, establishing baseline accuracy metrics for image-to-fingerboard conversion.

Results and Evaluation
The system was evaluated on 110 public-domain violin scores across five categories: beginner, intermediate, and advanced printed music; scanned Suzuki book pages; and direct MusicXML uploads.

• Fingerboard Accuracy: When provided with clean digital MusicXML files (bypassing OMR), the system achieved 99.1% accuracy in assigning the correct string and finger position. The 0.9% error rate was attributed to rare edge cases involving complex key signatures.
• Note Recognition (OMR): For image inputs, note pitch accuracy varied by complexity:
  • Beginner printed music: 91.2%
  • Intermediate printed music: 84.1%
  • Advanced printed music: 76.8%
  • Scanned Suzuki pages: 79.3%
• Error Sources: The primary sources of OMR error were notes on ledger lines (common in high violin registers), accidental handling in sharp-heavy keys, and rhythm misinterpretation in fast, densely beamed passages.
• Processing Speed: On an Apple M2 machine, the total processing time for image inputs averaged 17.1 seconds (dominated by the ~14.8s OMR step). Direct MusicXML processing averaged 2.3 seconds.
• Coverage: The lookup table covers 91.0% of unique notes found in Suzuki Books 1–3, with 100% coverage for Book 1. The missing notes reside in higher positions (beyond the 7th position) not typically encountered by beginners.

Significance and Claims
The paper claims that MusicSynth addresses a specific, unmet need in music education by removing the barrier of manual transcription and software installation. Its significance lies not in the invention of new underlying technologies (OMR, parsers, and renderers already exist) but in the integration of these components into a specialized, violin-centric tool.

The author emphasizes that the tool is designed for asynchronous practice: users prepare videos in advance rather than using the system for real-time feedback. While the system has limitations—specifically regarding higher positions, complex polyphony, and handwritten scores—it provides a functional, free alternative for beginners to visualize finger placement for a wide range of standard repertoire. The project is open-source, with code and a live demo available under the MIT license.