⚡ electrical engineering

Multilingual Word-Level Forced Alignment with Self-Supervised Representations and Learned Dynamic Programming

This paper presents a multilingual word-level forced alignment method that integrates self-supervised representations from the MMS model and UnSupSeg into a learned dynamic programming framework, demonstrating superior performance over existing approaches on both trained and unseen languages.

Original authors: Roy Weber, Meidan Zehavi, Rotem Rousso, Joseph Keshet

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Roy Weber, Meidan Zehavi, Rotem Rousso, Joseph Keshet

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a recording of someone speaking and a written transcript of exactly what they said. The goal of "forced alignment" is to draw a perfect line connecting every single word in the text to the exact millisecond it was spoken in the audio. It’s like creating a perfectly synchronized subtitle track for a movie, but down to the individual word.

This paper introduces a new, smarter way to do this synchronization, especially for languages other than English. Here is how it works, explained through a few simple analogies.

The Problem: The Old Way is Clunky

Traditionally, computers have used older methods (like the Montreal Forced Aligner, or MFA) to sync audio and text. Think of these older methods like a rigid rulebook. They rely on strict rules about how sounds should fit together. If the speaker talks fast, mumbles, or uses a dialect the rulebook doesn’t know well, the synchronization gets messy. It’s like trying to fit square pegs into round holes using only a hammer.

The Solution: A Two-Part Team

The authors created a new system that acts like a team of two experts working together to find the exact boundaries of each word.

1. The "Vibe Check" Experts (The Encoder)
The system first looks at the audio through two different lenses to get a "feel" for where words start and end:

The Sound Detective (UnSupSeg): This part of the model doesn’t need to know what language is being spoken. It just listens for sharp changes in the sound waves—like the sudden silence between words or the burst of air in a consonant. It’s like a person who can hear the rhythm and breaks in speech without understanding the words.
The Language Expert (MMS): This part uses a massive, pre-trained model (Massively Multilingual Speech) that has heard over 1,100 languages. It tries to guess which words are likely being spoken at any given moment. It’s like a polyglot who recognizes the shape of the words even if they aren’t perfectly clear.

These two experts combine their insights. The "Sound Detective" says, "There’s a break here!" and the "Language Expert" says, "That sounds like the end of the word 'hello'." The system then calculates a probability score for every tiny slice of time (every 10 milliseconds), deciding how likely it is that a word boundary exists there.

2. The Puzzle Solver (The Decoder)
Having probabilities isn’t enough; you need a final, clean list of start and end times. This is where the "Decoder" comes in. Think of it as a puzzle solver that uses a smart checklist (Dynamic Programming) to make the final decision.

It doesn’t just pick the highest probability blindly. It checks for logic:

Consistency: Does this boundary make sense with the one before it?
Duration: Is the word too short or too long? (A word can’t be 1 millisecond long).
Agreement: Do the "Sound Detective" and the "Language Expert" agree?

It balances all these clues to find the most logical path through the audio, ensuring the final alignment is smooth and accurate.

Why It’s Better

The authors tested this new system on English datasets (TIMIT and Buckeye) and found it beat the old standard (MFA) and other modern tools.

But the real magic happens with unseen languages. The system was trained only on English. However, because it relies on the MMS model (which knows 1,100+ languages) and the UnSupSeg model (which just listens to sound patterns, not specific words), it can be applied to other languages without any extra training.

They tested it on Hebrew, German, and Dutch—languages it had never seen during training.

Hebrew: It performed significantly better than the baseline MMS model.
German: It matched or beat the traditional MFA tool.
Dutch: It performed reasonably well, though slightly lower than the others, likely due to the specific nature of the test data.

The Takeaway

In simple terms, this paper presents a new tool that syncs speech to text more accurately than previous methods. By combining a "sound rhythm" detector with a "multilingual word" recognizer, and then using a logical puzzle-solver to clean up the results, it creates a robust system.

The biggest win is that because it doesn’t rely on language-specific rules (like phoneme dictionaries), it can potentially work for any of the 1,100+ languages the underlying models support, without needing to be retrained for each one. It’s a universal translator for timing, not just words.

Technical Summary: Multilingual Word-Level Forced Alignment with Self-Supervised Representations and Learned Dynamic Programming

Problem Statement
Accurate word-level forced alignment—the precise temporal mapping between audio signals and textual transcriptions—is a foundational task in speech processing, critical for linguistic analysis and the development of technologies such as automatic speech recognition (ASR) and speech synthesis. While traditional Hidden Markov Model-Gaussian Mixture Model (HMM-GMM) frameworks, particularly the Montreal Forced Aligner (MFA), remain the industry standard, recent advances in self-supervised learning (SSL) have produced robust speech representations. However, leveraging these SSL representations for forced alignment remains challenging, particularly in multilingual contexts where resources for training language-specific models are scarce.

Methodology
The authors propose a framework for multilingual word-level forced alignment that integrates self-supervised representations with a learned alignment decoder. The system consists of two primary components: an alignment encoder and a learned dynamic programming (DP) decoder.

Alignment Representations: The method fuses two distinct representations:
- UnSupSeg: An unsupervised phoneme segmentation model trained via self-supervised contrastive learning. It identifies phoneme boundaries directly from raw speech, capturing fine-grained acoustic transitions without relying on annotated data.
- MMS (Massively Multilingual Speech): A self-supervised speech model. The authors derive a representation from MMS using CTC alignment, defined as the likelihood of a specific word starting at a given frame. This representation is upsampled to match the 10 ms temporal resolution of the UnSupSeg features.
Alignment Encoder: The normalized and concatenated outputs from UnSupSeg and MMS serve as input to an encoder ( $g_\theta$ ). The encoder’s objective is to estimate word-boundary probabilities over long temporal contexts. The authors evaluated VGG, Transformer, and Conformer architectures. Due to the class imbalance inherent in boundary detection (boundary frames are far fewer than non-boundary frames), the encoder is optimized using focal loss rather than standard cross-entropy. The Conformer architecture was selected for the final model due to its balance of computational efficiency and performance.
Alignment Decoder: The decoder ( $h_\psi$ ) is a learnable dynamic programming module that refines the raw boundary probabilities from the encoder into a final alignment sequence. It models the alignment score as a linear combination of feature functions ( $\phi_n$ ) weighted by parameters ( $\psi_n$ ). The feature functions capture structural aspects of alignment quality, including:
- Euclidean distance between UnSupSeg representations around candidate boundaries.
- Word transition scores from the encoder output.
- Normalized sums of encoder outputs within word segments.
- Likelihood of character emissions from the MMS emissions matrix.
  The optimal alignment sequence is found by maximizing this score subject to constraints on minimal word duration. The decoder parameters are optimized iteratively.

Key Contributions

A novel forced alignment framework that combines unsupervised phoneme boundary detection (UnSupSeg) with multilingual SSL representations (MMS).
The introduction of a learned dynamic programming decoder that integrates encoder probabilities with segmental features to infer final word boundaries.
Demonstration that the model can generalize to unseen languages without further training, leveraging the multilingual nature of MMS and the language-agnostic nature of UnSupSeg.

Results
The model was trained on English datasets (TIMIT and Buckeye) and evaluated on both English and unseen languages (Hebrew, German, and Dutch).

English Performance: On TIMIT and Buckeye, the proposed method (MWA) outperformed established baselines, including MFA, MMS-based alignment, WhisperX, and Nvidia-Canary-1B, across various tolerance thresholds (e.g., achieving 97.8% accuracy at $t \le 100$ ms on TIMIT, compared to 97.4% for MFA).
Unseen Languages: Without additional training, the model demonstrated strong generalization:
- Hebrew: Outperformed MMS-based alignment at stricter tolerance thresholds ( $t \le 25$ ms and $t \le 50$ ms).
- German: Performance was comparable to or exceeded MFA on the PHONDAT corpus.
- Dutch: Achieved performance on par with or better than existing approaches on the IFA Corpus.
Model Selection: The Conformer encoder consistently outperformed Transformer and VGG architectures in terms of alignment accuracy and robustness. Models trained on read speech (TIMIT) generally transferred better to unseen languages than those trained on conversational speech (Buckeye).

Significance
The paper claims that this approach offers a potential replacement for traditional HMM-GMM-based aligners like MFA, eliminating the need for grapheme-to-phoneme (G2P) conversions. By relying on robust multilingual representations, the method demonstrates the potential to scale to the 1,100+ languages supported by MMS without requiring language-specific training data, thereby enabling accurate and resource-efficient word alignment across a wide range of languages.

The Problem: The Old Way is Clunky

The Solution: A Two-Part Team

Why It’s Better

The Takeaway

More like this