⚡ electrical engineering

MAVL: A Multilingual Audio-Video Lyrics Dataset for Animated Song Translation

This paper introduces MAVL, the first multilingual audio-video benchmark for animated song translation, and proposes the SylAVL-CoT model that leverages multimodal cues and syllabic constraints to significantly outperform text-only approaches in generating singable and contextually accurate lyrics.

Original authors: Woohyun Cho, Youngmin Kim, Sunghyun Lee, Youngjae Yu

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Woohyun Cho, Youngmin Kim, Sunghyun Lee, Youngjae Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to translate a song from English to Korean, but you can't just translate the words. You have to translate the feeling, the rhythm, and make sure the new words fit perfectly into the melody, just like pieces of a puzzle snapping into place.

This is the challenge the paper "MAVL" tackles. Here is a simple breakdown of what the researchers did, using some everyday analogies.

The Problem: The "Literal" Trap

Imagine you are watching a cartoon where a character sees a butterfly and sings, "And there's a butterfly."

The Old Way (Text-Only): If you just ask a standard translator (like Google Translate) to translate this, it might say, "And there is a butterfly." In Korean, that sounds stiff and clunky. It's like trying to fit a square peg into a round hole. It might mean the right thing, but it doesn't sing right, and it doesn't match the happy, flapping motion of the butterfly on the screen.
The Real Challenge: To make a song translation work, you need to know:
1. What is being said? (The meaning)
2. How many beats does it take? (The rhythm/syllables)
3. What is happening on screen? (The visual context)

The Solution: MAVL (The New Recipe Book)

The researchers created a massive new "recipe book" called MAVL.

What is it? It's a dataset containing 228 songs from animated movies (like Frozen or Trolls) in five different languages (English, Spanish, French, Korean, and Japanese).
Why is it special? Unlike previous datasets that only had the lyrics (text), MAVL includes the audio (the singing) and the video (the animation).
The Analogy: Think of previous datasets as a sheet of music with just the notes. MAVL is the sheet music plus the recording of the orchestra and the video of the conductor. This allows the computer to "see" and "hear" the song, not just read it.

The New Tool: SylAVL-CoT (The Smart Translator)

To use this new recipe book, they built a smart AI system called SylAVL-CoT.

How it works: Instead of just guessing the translation, this AI acts like a careful chef following a strict recipe. It uses a "Chain-of-Thought" process (basically, it thinks out loud step-by-step):
1. Listen and Count: It listens to the audio to count exactly how many syllables (beats) the original singer used.
2. Watch and Feel: It looks at the video to understand the mood. Is the character sad? Are they running? This helps it pick words that fit the scene.
3. Adjust and Refine: It tries to write the new lyrics. If the new words are too long or too short, it rearranges them until they fit the beat perfectly, just like a tailor hemming a pair of pants to fit exactly.

The Results: Why It Matters

The researchers tested their new tool against standard translators and found:

Better Singability: The lyrics produced by SylAVL-CoT fit the music much better. They didn't sound like a robot reading a dictionary; they sounded like a natural song.
Better Context: Because the AI watched the video, it could choose words that matched the action on screen (like choosing a word for "flying" instead of just "being" for a butterfly).
Human-Like Quality: When real people listened to the translations, they rated the new tool's output as much closer to what a human professional translator would do, especially regarding how "singable" the words were.

The Bottom Line

This paper didn't just build a better translator; it built a multimodal one. It proved that to translate a song well, you can't just look at the text. You have to listen to the music and watch the movie. By giving the AI all three senses (text, audio, video) and forcing it to count the beats, they created a system that produces translations that are ready to be sung, not just read.

Technical Summary: MAVL and SylAVL-CoT

Problem Statement

Lyric translation is a specialized domain that prioritizes "singability"—the ability of translated lyrics to fit a melody while preserving semantic meaning, poetic structure, and musical rhythm. This task is significantly more complex than standard text translation because it requires balancing literal meaning with musical constraints (syllable count, rhythm, rhyme) and often necessitates non-literal adaptations.

The challenge intensifies in the context of animated musicals, where lyrics must align not only with the audio but also with visual cues (character actions, scene context). Existing approaches are largely limited to text-based or text-and-score-based methods, which fail to capture the full multimodal context required for natural, singable translations. Furthermore, previous datasets often lack alignment across multiple modalities (text, audio, video) and languages, making it difficult to train or evaluate models that can integrate these diverse inputs effectively.

Methodology

1. The MAVL Dataset

The authors introduce MAVL (Multilingual Audio-Video Lyrics Benchmark for Animated Song Translation), the first multilingual, multimodal benchmark designed specifically for singable lyrics translation in animated musicals.

Composition: MAVL comprises 228 songs from animated films (e.g., Frozen, Trolls, Encanto) aligned across five languages: English, Spanish, French, Korean, and Japanese.
Modality: Unlike previous datasets that rely solely on text or scores, MAVL integrates text, audio, and video.
Alignment Pipeline: The dataset construction involved a rigorous pipeline:
1. Collection: Gathering metadata and original English lyrics, then sourcing official dubbed lyrics from platforms like lyricstranslate.
2. Human Alignment: Verifying that candidate lyrics correspond to official, singable dubbed versions by cross-referencing with original audio/video. Non-singable dialogue or overlapping lyrics were excluded.
3. Audio-Video-Lyrics Alignment: Using stable-ts (based on Whisper) to generate stable timestamps, segmenting audio and video to match specific lyric lines.

2. SylAVL-CoT Model

To leverage MAVL, the authors propose SylAVL-CoT (Syllable-Constrained Audio-Video LLM with Chain-of-Thought). This approach utilizes existing Multimodal Large Language Models (MLLMs), specifically Gemini 2.0 Flash and Gemini 2.5 Flash, without fine-tuning, relying instead on prompt engineering and Chain-of-Thought (CoT) reasoning.

The model operates via a three-step CoT pipeline:

Identify Core Lyric & Syllable Segmentation: The model analyzes the provided audio snippet and text to identify the exact lyric segment. It segments the lyrics into syllables based on audible breaks and the known syllable count of the source, establishing a rhythmic template.
Generate Target-Language Syllable List (Video Context): The model processes visual cues from the video (e.g., character expressions, motion, cultural context) to refine the translation. It aims to capture imagery and mood while generating a target-language syllable list that matches the source syllable count.
Iterate and Refine: The model iteratively paraphrases and reorders words to ensure the translation is culturally appropriate, linguistically coherent, and strictly adheres to the target syllable count. This step addresses the difficulty of maintaining rhythm across languages with different grammatical structures.

3. Evaluation Metrics

The paper proposes a multi-faceted evaluation framework focusing on three principles:

Singability: Measured via Syllable Error (SE), Syllable Count Distance (SCD), and Error Rate. SE uses a penalty factor ( $\beta=2.0$ ) to heavily penalize exceeding the reference syllable count, as this negatively impacts singability.
Sense: Measured using Semantic Scores based on cosine similarity of multilingual embeddings (mpnet-base-v2) between the translation and both the original English lyrics and the human-translated dubbed lyrics.
Naturalness: Measured via Phonetic Distance (Levenshtein distance) between the International Phonetic Alphabet (IPA) transcriptions of the generated lyrics and the reference lyrics.

Key Results

Quantitative Performance

Singability: SylAVL-CoT significantly outperforms text-only baselines (including Google Translate, mBART-50, Qwen2.5-72B, GPT-4o, and Gemini 2.0) across all languages. It achieves the lowest Syllable Error, Syllable Distance, and Error Rate. For instance, in English-to-Korean translation, SylAVL-CoT reduced Syllable Error from ~10.3 (GPT-4o) to 0.695.
Sense: While text-only models often achieve higher semantic similarity to the original English lyrics (indicating literal translation), SylAVL-CoT shows lower similarity to the original but maintains high similarity to human-translated dubbed lyrics. This suggests SylAVL-CoT successfully adopts the creative paraphrasing strategies used by human professionals to maintain singability, rather than sticking to literal meanings.
Naturalness: SylAVL-CoT yields the lowest phonetic distance scores compared to both original and dubbed references, indicating its output shares articulatory and rhythmic traits with professional translations.

Ablation Studies

Multimodality: Removing video input (T+A) resulted in lower semantic similarity to dubbed lyrics compared to the full T+A+V setup, suggesting video context is crucial for generating contextually appropriate paraphrases.
CoT Steps: The "Syllable List" generation and "Refine" stages were critical. Removing these steps led to a significant increase in Syllable Error, confirming that the structured reasoning process is necessary for enforcing constraints.
Model Choice: While GPT-4o and Qwen2.5 performed well with simple constraints, Gemini demonstrated superior performance when utilizing the CoT approach, highlighting its ability to track multiple constraints simultaneously during reasoning.

User Study

A user study with native speakers (N=10 per language) rated translations on Singability, Sense, and Overall Quality. SylAVL-CoT achieved the highest Overall Quality scores across all languages, notably outperforming other models in Singability.

Significance and Claims

The paper claims that MAVL and SylAVL-CoT address a critical gap in lyrics translation by moving beyond text-only limitations.

First Multimodal Benchmark: MAVL is presented as the first dataset to support multilingual lyric translation across text, audio, and video, enabling the evaluation of models that integrate these modalities.
Effective Constraint Handling: The study demonstrates that leveraging CoT reasoning in existing MLLMs, combined with multimodal inputs, allows for the enforcement of strict syllabic constraints without the need for resource-intensive fine-tuning.
Human-Like Adaptation: By aligning with human-translated dubbed lyrics in terms of semantic adaptation and phonetic naturalness, the proposed approach bridges the gap between literal machine translation and the creative, musical requirements of professional song localization.

The authors conclude that their work paves the way for more effective automated lyrics translation systems, though they acknowledge limitations regarding the scope of genres (focused on animated musicals), line-by-line processing constraints, and the lack of specialized mechanisms for tonal languages.