💬 NLP

Are you speaking my languages? On spoken language adherence in multimodal LLMs

This paper addresses the issue of language misidentification in multimodal LLM-based Automatic Speech Recognition by defining "language adherence," introducing a metric to quantify violations, and evaluating soft prompting strategies like zero-shot guidance, supervised fine-tuning, and Chain-of-Thought reasoning to mitigate these errors while preserving transcription quality.

Original authors: Hyungwon Kim, Kandarp Joshi, Lillian Zhou, Pavel Golik, Petar Aleksic

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hyungwon Kim, Kandarp Joshi, Lillian Zhou, Pavel Golik, Petar Aleksic

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart, multilingual robot assistant that can listen to you speak and type out exactly what you said. This robot is great at understanding many languages at once, even if you switch between them mid-sentence (like saying "I need a café" in English but then switching to Spanish).

However, this robot has a funny quirk: sometimes, it gets confused about which language you are actually speaking. If you speak French, it might accidentally type out the words using German spelling rules, or if you speak Korean, it might throw in some random Japanese characters. To a human, this looks like a mistake; to the robot, it's just a guess.

This paper, titled "Are you speaking my languages?", is about fixing that confusion without making the robot too rigid. The researchers want the robot to listen to your hints about what language you're using, but they also want it to be smart enough to ignore those hints if they are wrong.

Here is a breakdown of their approach, using some everyday analogies:

The Problem: The "Wrong Recipe" Disaster

The researchers call this problem a "Language Adherence Violation."
Think of the robot like a chef. If you tell the chef, "Make me a French dish," but the ingredients on the counter are clearly Italian, a bad chef might still try to make a French dish using Italian ingredients, resulting in a weird, inedible mess. In the robot's case, if it thinks you are speaking Spanish but you are actually speaking Hindi, it might write Hindi words using Spanish spelling. This ruins the transcription and makes the robot look unprofessional or even insensitive.

The Solution: Three Ways to Guide the Chef

The team tested three different ways to help the robot listen to your hints better without forcing it to ignore the actual audio.

1. Zero-Shot Prompting (The "Polite Nudge")
This is like walking up to the chef and saying, "Hey, I think we're making French food today," before they start cooking.

How it works: The researchers simply added a sentence to the robot's instructions, like "Transcribe this in French."
The Catch: Sometimes you might give the wrong hint (e.g., you tell the robot "French" but you actually speak Spanish). The researchers found that if you phrase the hint gently (e.g., "This might be a mix of French and other languages"), the robot is smart enough to listen to the hint unless the audio clearly says otherwise. It's a "nudge" rather than a "shove."

2. Supervised Fine-Tuning (The "Intensive Training Camp")
This is like taking the chef to a special school where they practice making French dishes over and over again, specifically learning to follow the "French" instruction tag.

How it works: They retrained the robot on thousands of examples where the instruction matched the audio. They also threw in some "trick" examples where the instruction was wrong, to teach the robot to stay flexible.
The Result: The robot got very good at following instructions, but if you gave it no instruction at all, it sometimes forgot how to cook properly because it had been trained so heavily on following tags.

3. Chain-of-Thought (The "Thinking Pause")
This is like asking the chef to stop for a second, think out loud, and say, "Okay, I hear French words, so I will write in French," before they start cooking.

How it works: The researchers made the robot pause and explicitly state, "The language is French," before it started typing the transcription.
The Result: This forced the robot to commit to a language decision first. It worked well, but it added a tiny bit of extra thinking time (though the researchers say it's negligible).

The Big Findings

The researchers tested these methods on real-world data, including people switching between languages (code-switching). Here is what they discovered:

The "Correct Hint" is King: If you give the robot the right language hint (e.g., "This is French"), it performs perfectly, no matter which of the three methods you use.
The "Wrong Hint" is Tricky: If you give it a wrong hint (e.g., "This is Spanish" when it's actually French), the robot sometimes gets confused. However, the "Polite Nudge" (Zero-Shot) method was the most robust at ignoring the wrong hint and sticking to the audio.
The "Mix" is Okay: If you tell the robot, "This is French and maybe some Spanish," and it is actually just French, the robot handles it fine. It doesn't get confused by the extra option.
No Hint is Risky: If you give the robot no hint at all, it performs worse, especially if you have trained it heavily on following hints. It seems the robot gets a bit lost without a starting point.

The Bottom Line

The paper concludes that you don't necessarily need to retrain your robot or make it think in complex steps to fix this problem. Simply giving it a clear, polite hint about the language is often enough.

However, the most important takeaway is that getting the language hint right in the first place is crucial. If the system that tells the robot "You are speaking French" is wrong, the robot will struggle. The best strategy is to have a reliable system upstream that accurately guesses the language before the transcription even begins.

In short: Give the robot a good map, and it will find the right path. If the map is wrong, even the smartest robot might get lost.

Technical Summary: On Spoken Language Adherence in Multimodal LLMs

Problem Statement
While Large Language Model (LLM) based Automatic Speech Recognition (ASR) enables flexible, zero-shot multilingual transcription and code-switching, these models frequently misidentify the target language, particularly in short or noisy segments. This "lack of language adherence" degrades transcription fidelity, introducing errors that distort meaning and compromise downstream tasks such as machine translation and sentiment analysis. Beyond functional errors, erroneous foreign-language text creates a negative user experience, potentially appearing amateurish or culturally insensitive. The core challenge is to guide the model toward the expected language using user context (hints) without strictly constraining the model's ability to code-switch or handle imperfect signals.

Methodology
The authors propose a systematic approach to address language adherence, encompassing formalization, measurement, and three distinct mitigation strategies.

Formalization and Metric (LAVR):
The paper defines a "Language Adherence Violation" as occurring when the set of languages ( $L_{hyp}$ ) detected in the generated output is not a subset of the reference languages ( $L_{ref}$ ) present in the audio. To quantify this, the authors introduce the Language Adherence Violation Rate (LAVR).
- Operational Definition: "Language" is defined by its canonical script. A violation occurs if the ASR output contains characters inconsistent with the target language's standard orthography (e.g., a 'ç' in a German output).
- Scope: Basic ASCII punctuation and digits are treated as neutral. The metric prioritizes recall for unexpected characters that would draw user attention, accepting a trade-off where it may miss systematic errors (like dropping umlauts) if no "surprising" characters are present.
Mitigation Strategies:
The study evaluates three non-mutually exclusive strategies to improve adherence while maintaining flexibility:
- Zero-Shot Prompting (ZS): Leveraging in-context learning by providing language hints in the prompt. The authors tested three prompt styles (P1, P2, P3) and selected P3 ("Transcribe this speech segment. It may contain a mix of and other languages.") for its robustness against incorrect hints.
- Supervised Fine-Tuning (SFT): Training the model with language-adherence prompts to ingrain desired behavior. The training data included a mixture of correct hints, distractor hints (incorrect languages), mixed hints, and no-hint scenarios to ensure robustness.
- Chain-of-Thought (CoT): A reasoning step where the model is compelled to explicitly identify and declare the spoken language before transcribing. This narrows the sampling space for the transcription task.
Experimental Setup:
- Datasets: Monolingual datasets (real-world user queries) and synthesized code-switching datasets (English paired with French, Hindi, or Korean).
- Baseline: Gemini Flash Lite 2.0, a deep transformer-based LLM.
- Evaluation Scenarios: Four prompt conditions were tested: no-hint, correct (ideal oracle), distractor (incorrect language), and mix (correct + distractor).
- Metrics: LAVR and Word Error Rate (WER).

Key Results
The comparative analysis across English, French, Hindi, and Korean (both monolingual and code-switching) yielded the following findings:

Impact of Language Hints: The presence of at least one correct language hint ("correct" or "mix" conditions) significantly outperformed "no-hint" and "distractor" conditions across all methods and metrics. This underscores the critical importance of accurate upstream language estimation.
Strategy Comparison:
- When provided with identical language hints, the three strategies (ZS, SFT, CoT) demonstrated broadly comparable performance levels.
- Zero-Shot Prompting proved as effective as the more computationally intensive SFT and CoT approaches.
- SFT and CoT Degradation: Both SFT and CoT models exhibited elevated WER under the "no-hint" condition compared to the baseline. The authors attribute this to catastrophic forgetting, as the baseline was heavily trained on "no-hint" data, whereas the fine-tuned models saw a reduced proportion (10%) of such data during training.
Distractor Resilience: Including a distractor language alongside a correct hint ("mix" condition) had a negligible negative impact, performing significantly better than having no hint at all. Conversely, providing only a distractor hint generally resulted in performance inferior to the "no-hint" scenario.

Significance and Claims
The paper claims to provide the first systematic evaluation of language adherence in multimodal LLMs for ASR, introducing a standardized metric (LAVR) to quantify these violations.

The primary contribution is the demonstration that zero-shot prompting is as effective as fine-tuning or Chain-of-Thought reasoning for maintaining language adherence, provided that accurate language signals are available. The authors conclude that the most impactful factor for optimizing system performance is not the complexity of the mitigation strategy, but rather the accuracy of the language hint itself. Consequently, the paper advocates for the implementation of upstream mechanisms (such as dedicated Language ID models or metadata analysis) to predict spoken languages with high confidence, rather than relying solely on model-side architectural changes to correct for missing or erroneous signals.

The Problem: The "Wrong Recipe" Disaster

The Solution: Three Ways to Guide the Chef

The Big Findings

The Bottom Line

Technical Summary: On Spoken Language Adherence in Multimodal LLMs

More like this