A Visually Guided Sound Source Separation and Localization System for Musical Instrument Videos

Imagine walking into a crowded room where a dozen different musical instruments are playing at once. To the human ear, the result is a single, tangled wall of sound. If you were to record that noise on a single microphone, the recording would contain every note, every rhythm, and every instrument mixed together into one indistinguishable stream. For decades, scientists have struggled with a specific challenge: how to take that single, messy recording and pull out the individual instruments, isolating the violin from the drum or the piano from the flute, without ever having heard them alone before. This is the problem of sound source separation. While computers have become quite good at this when they can see the video of the performance, most existing methods rely on complex, all-in-one systems that are difficult to adapt to new or rare instruments.

A team of researchers at Yazd University in Iran has proposed a different approach, one that breaks the problem into two distinct steps to better handle real-world videos, including those featuring traditional Persian musical instruments. Their work, published in a recent study, focuses on creating a system that first "looks" at the video to identify which instruments are present and where they are located, and then uses that visual information to guide the process of separating the audio. Instead of trying to solve the visual and audio puzzles simultaneously with a single massive brain, their method uses a specialized visual detector to tell a separate audio processor exactly what to listen for. This modular design allows the system to be more flexible and effective, particularly for instruments that are rarely found in standard training data.

The researchers began by building a visual detection system using a model known as YOLOv8, a tool designed to recognize objects in images quickly and accurately. They trained this model on a collection of 358 annotated images featuring twelve different types of musical instruments, a mix of international staples like the guitar and piano, and traditional Persian instruments such as the Daf, Kamancheh, and Santur. The goal was to teach the computer to spot these instruments in video frames, drawing a box around them and naming them. Because the dataset was relatively small, the team used a technique called transfer learning, which allows a model trained on a vast library of general objects to be fine-tuned for these specific musical tools. When tested on new video frames, the system successfully identified the instruments with a precision of 75.1 percent, meaning that when it said it saw a specific instrument, it was usually correct. It also managed to find about 59 percent of all the instruments actually present in the scenes.

Once the visual system identifies which instruments are playing, the second part of the process begins. The researchers developed a separate audio processing network for each of the twelve instrument types. This means that if the visual detector spots a guitar and a drum, the system does not try to guess how to separate them from scratch. Instead, it activates the specific "Guitar" network and the specific "Drum" network. Each of these networks is designed to look at the mixed audio recording, which has been converted into a visual map of sound frequencies over time, known as a spectrogram. The network's job is to create a digital filter, or a mask, that highlights the parts of the sound belonging to its specific instrument and silences the rest.

The researchers tested this two-step system on 420 real-world video clips. They found that by using the visual detection to guide the audio separation, the system could effectively isolate the sounds of the different instruments. On average, the separation quality was strong, with the system achieving a score that indicates a clear distinction between the target sound and the background noise. For some instruments, like the clarinet and the guitar, the separation was exceptionally clean, while others with more complex or overlapping frequencies presented a greater challenge. The study also compared their method against other advanced techniques and found that their approach produced superior results in reducing interference from other instruments, achieving an average separation quality score of 8.75 decibels.

A key finding of this research is that the visual information acts as a powerful guide, narrowing down the possibilities for the audio system. By knowing exactly which instruments are in the scene, the system does not have to guess or search through all possible combinations of sounds. This is particularly valuable for Persian musical instruments, which are often underrepresented in global datasets and difficult for standard models to recognize. The study demonstrates that a modular system, where visual detection and audio separation work in tandem but remain distinct, is a practical and effective way to untangle complex musical performances. While the system currently requires a separate model for each instrument type and cannot yet learn to separate a brand-new instrument it has never seen before, the results show a clear path forward for improving how machines understand and isolate sound in the real world.