💬 NLP

DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation

This paper introduces DialectGen, a large-scale benchmark revealing significant performance degradation in multimodal generative models when processing dialectal inputs, and proposes a novel encoder-based mitigation strategy that effectively restores dialect robustness to match Standard American English levels without compromising standard performance.

Original authors: Yu Zhou, Sohyun An, Haikang Deng, Da Yin, Clark Peng, Cho-Jui Hsieh, Kai-Wei Chang, Nanyun Peng

Published 2026-04-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yu Zhou, Sohyun An, Haikang Deng, Da Yin, Clark Peng, Cho-Jui Hsieh, Kai-Wei Chang, Nanyun Peng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-talented artist who can paint or film anything you describe. You tell them, "Draw a man driving his car," and they instantly create a perfect picture of a guy in a sedan. But then, you say, "Draw a man driving his whip."

If you speak African American English, you know "whip" is a cool slang word for a car. But this artist? They get confused. They might draw a man holding a leather whip, or a man driving a horse-drawn carriage, completely missing the point. They are so used to the "standard" dictionary that they don't understand the local flavor of your language.

This is exactly the problem DialectGen is solving. Here is the story of the paper in plain English:

1. The Problem: The Artist Only Speaks "Standard"

The researchers found that the world's best AI image and video generators (like DALL-E, Stable Diffusion, and Wan 2.1) are like that confused artist. They are trained mostly on "Standard American English" (SAE).

When you use a dialect word—like "ang pow" (Singaporean for red envelope), "brinjal" (Indian for eggplant), or "carnal" (Chicano for brother)—the AI often fails miserably.

The Result: The AI ignores the word or draws the wrong thing.
The Scale: The paper tested 17 different AI models. When a single dialect word was used, the AI's performance dropped by 32% to 48%. That's like a student who gets an A on a test but fails the moment you ask one question in a different accent.

2. The Solution: Building a "Dialect Gym" (DialectGen)

To fix this, the team built a massive new test called DialectGen.

The Dataset: They didn't just guess; they worked with real speakers of six different English dialects (African American, British, Chicano, Indian, Singaporean, and Standard American).
The Process: They created over 4,200 pairs of prompts. One pair might be "A man hiking with his brother" (Standard) and "A man hiking with his carnal" (Chicano).
The Filter: Real human speakers checked every single prompt to make sure it made sense and wasn't ambiguous. This is like having a strict coach ensuring the athletes are training with the right form.

3. Why Old Fixes Didn't Work

The researchers tried the usual tricks to fix the AI, but they were like trying to fix a flat tire with duct tape:

Prompt Rewriting: They tried using another AI to translate "whip" back to "car" before sending it to the artist. Result: It helped a tiny bit, but often the translator made mistakes or changed the vibe of the prompt.
Fine-Tuning: They tried teaching the AI's "painting brain" (the decoder) new words. Result: This actually made the AI worse at understanding standard English. It was like teaching a chef a new recipe but making them forget how to cook rice.

4. The Real Fix: Teaching the "Translator" (The Encoder)

The team realized the problem wasn't the painting; it was the translator inside the AI. In these models, there is a part that reads your text and turns it into a "meaning map" before the painting starts. This part was ignoring dialects.

They designed a new training method with three special exercises:

Dialect Learning: They taught the translator, "Hey, when you see 'whip' in this context, it means 'car'."
Polysemy Control: They taught the translator, "But wait! If the context is different, 'whip' does mean a leather strap. Don't get confused." (This is crucial because many dialect words have double meanings).
The Safety Net (KL Regularization): They made sure that while learning these new words, the AI didn't forget how to speak Standard English. It's like learning a new language without forgetting your native tongue.

5. The Magic Result

After this training, the AI models became bilingual (or multilingual!).

Before: The AI failed 40% of the time with dialect words.
After: The AI performed just as well with dialect words as it did with Standard English.
The Best Part: The AI didn't get worse at Standard English. It gained a new skill without losing the old one.

The Big Picture

Think of the AI as a global traveler. Before this paper, the traveler only spoke the "tourist language" (Standard English). If you spoke to them in a local dialect, they froze.

DialectGen is the map and the language school that finally teaches the traveler to understand the locals. It ensures that whether you ask for "sneakers" or "kicks," "bathroom" or "loo," the AI understands you perfectly and creates the art you actually wanted to see.

This is a huge step toward making AI fair and useful for everyone, not just people who speak the "standard" version of English.

1. Problem Statement

Multimodal generative models (Text-to-Image and Text-to-Video) exhibit significant performance degradation when processing inputs in non-Standard American English (SAE) dialects. While these models perform well on SAE prompts, they fail to correctly interpret semantically identical prompts containing dialect-specific lexemes (e.g., "brinjal" in Indian English vs. "eggplant" in SAE, or "whip" in African American English vs. "car").

Current pre-training paradigms often filter out lower-resource dialect data, leading to:

Lexical Misinterpretation: Models fail to generate content corresponding to dialect words.
Polysemy Errors: When a dialect word has a different meaning in SAE (e.g., "whip" meaning a tool vs. a car), models default to the SAE meaning even when context dictates the dialect meaning.
Performance Gaps: Preliminary observations suggest performance drops of 30–50% for dialect inputs compared to SAE.

2. Methodology

A. DialectGen Benchmark Construction

The authors constructed DialectGen, a large-scale benchmark covering six English dialects:

Standard American English (SAE)
African American English (AAE)
British English (BrE)
Chicano English (ChE)
Indian English (InE)
Singaporean English (SgE)

Data Pipeline:

Lexeme Selection: Extracted 1,126 dialect lexemes from regional dictionaries (e.g., Oxford, Dictionary of American Regional English).
Prompt Generation: Used GPT-4o to generate "Concise" (≤6 words) and "Detailed" (≥9 words) prompts for SAE lexemes.
Translation: Swapped SAE lexemes with their synonymous dialect equivalents.
Human Validation: Employed native dialect speakers via Amazon MTurk to verify three criteria:
1. Synonymy: The dialect prompt must mean exactly the same as the SAE prompt.
2. Validity: The prompt must be natural within the dialect context.
3. Non-Ambiguity: For polysemous words, the dialect meaning must be the only logical interpretation in the prompt context.
Final Dataset: 4,200 validated prompt pairs (SAE/Dialect) plus polysemous control prompts.

B. Evaluation Metrics

Automatic: Reference-free image-text alignment metrics: VQAScore and CLIPScore.
Human Evaluation: Annotators rated alignment between generated images/videos and the SAE prompt (0–10 scale).
Performance Drop Calculation: $Drop = \frac{SAE_{score} - Dialect_{score}}{SAE_{score}}$ .

C. Mitigation Strategy: Encoder-Based Learning

The authors propose a novel mitigation strategy that updates the text encoder rather than the decoder or the prompt. The method consists of three loss components:

Dialect Learning Loss ( $L_{DL}$ ):
- Aligns the embedding of a dialect prompt ( $\pi(p_d)$ ) with the frozen embedding of its synonymous SAE prompt ( $\pi_0(p_s)$ ).
- Minimizes cosine distance: $1 - \langle \pi(p_d), \pi_0(p_s) \rangle$ .
- Goal: Teach the model to recognize dialect lexemes.
Polysemy Control Loss ( $L_{PC}$ ):
- Prevents the model from losing the ability to distinguish SAE meanings of polysemous words.
- Aligns the target encoder's embedding of an SAE polysemous prompt with the frozen encoder's embedding of the same prompt.
- Goal: Ensure the model doesn't force a dialect interpretation when the context is clearly SAE.
KL Regularization ( $L_{KL}$ ):
- Preserves general SAE performance by regularizing the output distribution of the trainable encoder against a frozen reference encoder.
- Since encoders output continuous embeddings, the authors approximate logits by computing similarity scores between caption embeddings and a set of reference image embeddings (from MSCOCO).
- Goal: Prevent catastrophic forgetting of general SAE capabilities.

3. Key Contributions

DialectGen Benchmark: The first large-scale benchmark for dialect robustness in multimodal generation, covering 6 dialects and 4,200+ validated prompts.
Comprehensive Evaluation: Evaluated 17 state-of-the-art models (including Stable Diffusion 1.5/3.5, Flux.1, DALL-E 3, Wan 2.1, CogVideoX) and found severe performance drops (32–48%) on dialect inputs.
Baseline Analysis: Demonstrated that common mitigation methods (Decoder Fine-tuning, Prompt Rewriting/Translation) are ineffective, offering only marginal gains (<7%) or causing significant degradation in SAE performance.
Novel Mitigation Method: Introduced an encoder-based training strategy that simultaneously improves dialect robustness and preserves SAE performance.

4. Experimental Results

Benchmark Findings

Performance Drop: SOTA models showed performance drops ranging from 32.26% to 48.17% when a single dialect word was used.
Dialect Sensitivity:
- ChE and InE suffered the most severe drops (often >40%).
- BrE showed the least degradation (likely due to higher representation in training data).
- Concise prompts were significantly harder for models than detailed prompts, highlighting the lack of contextual inference for unknown lexemes.
Baseline Failure:
- Prompt Rewriting: Improved dialect performance by only ~6.8% but introduced semantic drift and failed to handle polysemy correctly.
- Decoder Fine-tuning: Improved dialect performance slightly (~5.7%) but caused substantial drops in general SAE performance and polysemy handling.

Mitigation Results (Stable Diffusion 1.5 & XL)

The proposed encoder-based method achieved:

Dialect Performance: Raised performance on 5 dialects to be on par with SAE (Average VQAScore improvement of +34.4%).
SAE Preservation: Incurring near-zero cost (<1% drop) on general SAE (MSCOCO) and polysemy tasks.
Comparison: Outperformed all baselines (Prompt Rewrite, Decoder Fine-tuning, Diffusion DPO) significantly.

5. Significance and Future Work

Equity in AI: This work highlights a critical "allocational harm" where dialect speakers receive lower-quality AI outputs, reinforcing the need for inclusive model development.
Technical Insight: It proves that text encoders are the bottleneck for dialect understanding in multimodal models, not the image/video decoders. Updating the encoder is more effective than fine-tuning the decoder.
Scalability: The proposed method is computationally efficient (training on a single GPU takes <1 hour) and generalizable across different model architectures (SD1.5, SDXL, Wan 2.1).
Future Directions: The authors suggest exploring cultural biases (e.g., skin tone shifts in generated images based on dialect), the impact of grammatical variations, and the compositional effects of multiple dialect lexemes in a single prompt.

In conclusion, DialectGen provides the first rigorous framework to measure and fix dialect bias in multimodal generation, offering a practical, high-performance solution that ensures AI models serve all English speakers equitably without sacrificing standard performance.