💻 computer science

Local Multimodal Music Alignment from Global Supervision

The paper introduces FuSiLi, a novel similarity score that enables multimodal music models to learn fine-grained local alignments between audio frames and sheet music patches using only coarse global supervision, while maintaining strong global retrieval performance.

Original authors: Irmak Bukey, Zachary Novack, Jongmin Jung, Dasaem Jeong, Chris Donahue

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Irmak Bukey, Zachary Novack, Jongmin Jung, Dasaem Jeong, Chris Donahue

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, magical library where every book has a matching soundtrack. Usually, if you wanted to teach a computer to understand how the music matches the pictures in the book, you'd need a human to sit down and painstakingly draw a line connecting every single note in the audio to every single tiny patch of ink on the page. It's like asking someone to match every single grain of sand on a beach to a specific wave in the ocean. It's possible, but it takes forever and costs a fortune.

The researchers in this paper asked a bold question: Can we teach the computer to make these tiny, local connections just by showing it the whole book and the whole song together? They call this "global supervision." It's like showing a student a whole movie and its script and asking them to figure out which line of dialogue happens at which second, without ever being told the exact answer.

The Problem with the Old Way

The most popular way to teach computers this kind of matching right now is called "contrastive learning." Think of this like a game of "Hot and Cold." The computer looks at a picture and a song and just says, "Yes, these two go together!" or "No, these don't." It crushes the picture and the song into a single, blurry summary blob and compares those blobs.

The paper shows that this "blob" approach is terrible at finding the specific details. When they tested it on a task where the computer had to point to the exact moment in the song that matches a specific spot on the sheet music, the old method only got it right 16% of the time. It was like trying to find a specific needle in a haystack by just looking at the whole haystack and guessing.

The New Magic Trick: FuSiLi

The team invented a new method called FuSiLi (which stands for Fused Sinkhorn-Localized Similarity). Instead of squishing the picture and song into a single blob first, FuSiLi keeps all the tiny details alive.

Here is how it works, using a playful analogy:
Imagine the sheet music is a grid of tiny squares (patches) and the audio is a stream of tiny time-slices (frames).

The Grid: The computer looks at every single square of the music and every single slice of the sound.
The Dance: It calculates how well every square matches every slice, creating a giant grid of "match scores."
The Sinkhorn Step: This is the secret sauce. The computer uses a mathematical dance called the Sinkhorn algorithm. Imagine a dance floor where every dancer (a music square) needs to find exactly one partner (an audio slice). The algorithm gently nudges them until everyone has a unique partner, creating a "soft alignment." It forces the computer to realize, "Okay, this specific note must go with this specific moment in time," even though it was only told, "This whole song goes with this whole page."
The Result: Only after this dance does the computer summarize the results into a final score.

What They Found

The results were pretty exciting. By using this new "dance" method:

Local Alignment: The computer's ability to match specific notes to specific times jumped from 16% to 30%. That's nearly double the accuracy!
Retrieval: Even though they focused on the tiny details, the computer didn't forget how to find the right song for the right page. It stayed just as good at the big-picture search as the old methods (around 43.5% to 43.8%).
The "Point-and-Retrieve" Test: They tested a real-world scenario: if a user clicks on a specific measure in a sheet music image, can the computer jump to the right second in the audio recording? The old method got this right only 1.33% of the time. FuSiLi got it right 13.91% of the time. That is more than a 10-fold improvement.

What They Ruled Out

The paper is very clear about what doesn't work or isn't needed:

No Manual Labels Needed: You don't need those expensive, human-drawn lines connecting notes to time. The method works with just the paired song and image.
No "Blob" Only: Simply averaging the features (the old "blob" method) is proven insufficient for finding local connections.
No Sequence-to-Sequence Overhead: There are other methods that try to translate music to audio like a translator translating a book sentence-by-sentence. The paper notes these are much slower and require the computer to do way more work (specifically, they need $|Q| \times |R|$ calculations, which is a lot of math) compared to FuSiLi's efficient approach.
Just Cosine isn't Enough: They tried a simpler version where they just added up the match scores without the Sinkhorn dance. That failed miserably, dropping local accuracy down to 2%. The "dance" step is essential.

How Sure Are They?

The authors are quite confident in these numbers because they tested them on real data.

They trained on a massive dataset of about 345,000 image-audio pairs.
They tested on specific datasets like MSMD (which has ground-truth labels to check against) and YTSV (real YouTube videos with sheet music).
The 10x improvement in the "point-and-retrieve" task was measured directly on these datasets.
They also noted that while the method is great, it's not a magic wand for everything. For example, on the "Same Piece" batch strategy (where the computer tries to distinguish between two very similar parts of the same song), the method worked best, but on random batches, the gains were smaller.

The Bottom Line

This paper suggests that you don't need a million human annotators to teach a computer the fine details of music. By using a clever mathematical dance (Sinkhorn) to force the computer to find one-to-one matches between tiny pieces of audio and tiny pieces of sheet music, you can learn these local connections just from the big picture. It's like teaching a student to read the fine print by only showing them the whole page, but giving them a special tool that helps them focus on the right words at the right time.

The authors believe this approach could be a game-changer for other fields where we have lots of big-picture data but very few detailed labels, opening the door to smarter, more precise AI without the massive cost of manual labeling.

Technical Summary: Local Multimodal Music Alignment from Global Supervision

Problem Statement
Multimodal music understanding often requires establishing localized correspondences between data modalities, such as mapping specific time frames in a performance audio recording to specific pixel patches in a sheet music score image. Ideally, training such systems would utilize two types of ground truth: global supervision (pairing the correct audio with the correct image) and local supervision (explicit mappings between audio time and image pixels). However, collecting local alignments at scale is prohibitively expensive, requiring significant manual effort from expert annotators. In contrast, global supervision in the form of audio-image pairs corresponding to the same musical piece is abundant and scalable (e.g., from IMSLP or YouTube segments). The central research question addressed is whether it is possible to learn fine-grained local alignments across modalities using global supervision alone.

Methodology: FuSiLi
The authors propose FuSiLi (Fused Sinkhorn-Localized Similarity), a novel similarity score for multimodal contrastive learning that operates directly on local features.

Paradigm Shift: Unlike standard contrastive learning (e.g., CLIP, CLAP) which follows a "pool-then-fuse" paradigm (pooling local features into global vectors before computing similarity), FuSiLi adopts a "fuse-then-pool" paradigm.
Local Feature Processing: Instead of pooling first, the model computes pairwise cosine similarity directly between frame-level features (image patches and audio frames) to create a frame-wise similarity matrix $F_{ij}$ .
Soft Alignment via Sinkhorn: To capture frame-level correspondences, the method applies the Sinkhorn algorithm to the similarity matrix. This transforms the matrix into a doubly stochastic soft-alignment matrix $A_{ij}$ . Unlike attention mechanisms that operate in a single direction, Sinkhorn enforces mutual constraints (row and column normalization), learning flexible correspondences without assuming monotonic temporal order (crucial for sheet music where layout does not strictly follow time).
Similarity Scoring: The final scalar similarity score $S^L_{ij}$ is computed as the element-wise product of the soft-alignment matrix and the original similarity matrix, effectively weighting frame pairs by their alignment probability.
Training Objective: The model is fine-tuned using a hybrid objective combining the proposed local loss ( $L_L$ ) with the conventional global contrastive loss ( $L_G$ ): $L = \alpha L_L + (1-\alpha)L_G$ . This allows the model to learn local alignments while retaining global retrieval capabilities.
Inference Efficiency: Although trained with iterative Sinkhorn steps, inference for local alignment does not require the Sinkhorn algorithm; direct cosine similarity between projected frame embeddings is sufficient, reducing inference time significantly (from 37ms to 5ms).

Key Contributions

FuSiLi Algorithm: A new inductive bias for contrastive learning that enables the emergence of fine-grained local alignments from global segment-level supervision.
Hybrid Training Strategy: A demonstration that combining local and global objectives preserves the strong cross-modal retrieval performance of standard models while significantly boosting local alignment accuracy.
Zero-Shot Capability: The ability to perform "point-and-retrieve" tasks (mapping a specific location in a score image to a timestamp in audio) without explicit local training labels.
Efficiency: The approach maintains the asymptotic inference efficiency of standard contrastive models ( $|Q| + |R|$ passes) compared to sequence-to-sequence or specialized alignment models that require $|Q| \times |R|$ passes.

Experimental Results
The authors evaluated FuSiLi on cross-modal retrieval and frame-level alignment tasks using datasets including PDMX (synthetic), MSMD (synthetic with note-level ground truth), and YTSV (real-world YouTube segments).

Local Alignment: FuSiLi achieved 30% top-1 accuracy on frame-level alignment, a substantial improvement over the 16% achieved by standard global contrastive baselines. In zero-shot point-and-retrieve tasks, FuSiLi showed over 10x improvement over the baseline (13.91% vs. 1.33% in Image-to-Audio).
Global Retrieval: Despite focusing on local structure, FuSiLi variants trained with the hybrid objective ( $\alpha=0.5$ ) remained competitive with global baselines on retrieval tasks (Mean Reciprocal Rank), demonstrating that global semantic similarity is preserved.
Comparison to Alternatives: FuSiLi outperformed the U-MusT sequence-to-sequence model in most local alignment settings with significantly lower inference costs. While U-MusT achieved 20% top-1 accuracy, it exhibited higher perplexity and required more complex preprocessing.
Ablation Studies: The study confirmed that the Sinkhorn-based similarity is essential; replacing it with a simple summed cosine similarity caused a collapse in local alignment performance (dropping to 2% accuracy) and degraded global retrieval. The "Same Piece" batch construction strategy (sampling pairs from the same musical piece) was found to be more effective than random sampling for learning fine-grained distinctions.

Significance and Claims
The paper claims that FuSiLi successfully bridges the gap between the scarcity of local annotations and the abundance of global data. By reformulating the similarity function to operate on local features before pooling, the model learns to predict local alignments without explicit supervision. The authors position this as a promising direction for other domains where fine-grained annotations are scarce but global pairings are available. The work is motivated by practical applications in music information retrieval, such as score following, automatic transcription, and optical music recognition, aiming to assist human musicians in creative workflows. The authors note that their specific focus carries no noteworthy ethical risks beyond the general landscape of music and AI.