💬 NLP

DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation

The paper introduces DiaLLM, a framework that reveals a critical dissociation between dialectal robustness and generation in large language models, demonstrating that while continual pretraining and SFT improve understanding, explicit variety-targeted adaptation is necessary for authentic dialectal output, though current reward-based alignment methods often fail to align with human preferences.

Original authors: Jordan Painter, Dipankar Srirag, Adarsh Kappiyath, Diptesh Kanojia, Aditya Joshi, Lu Yin

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jordan Painter, Dipankar Srirag, Adarsh Kappiyath, Diptesh Kanojia, Aditya Joshi, Lu Yin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Bilingual" Robot That Only Speaks One Way

Imagine you have a very smart robot (a Large Language Model) that can understand every accent and dialect of English. If you speak to it in Australian slang, Indian English, or Northern British English, it understands you perfectly.

But here's the catch: No matter what you say, the robot always replies in a flat, standard American accent. It's like a person who can understand a joke told in a Scottish brogue but insists on telling the punchline in a generic TV news anchor voice.

The researchers call this the "Robustness-Generation Gap." The robot is robust (strong) at listening to dialects, but it fails at speaking them.

The Solution: DiaLLM (The Dialect Gym)

The team created a new training pipeline called DiaLLM to fix this. Think of it as a three-step gym routine to teach the robot how to actually speak with a local accent.

They tested this on three different robot "brains" (Llama, Qwen, and Gemma) and focused on three specific accents: Australian, Indian, and Northern British English.

Step 1: The Immersion Camp (Continual Pretraining)

First, they took the robots and sent them to an "immersion camp." They fed them a massive library of books and transcripts (the International Corpus of English) from 18 different English-speaking regions.

The Analogy: This is like moving the robot to a village where everyone speaks with a local accent. It absorbs the "vibe" and the vocabulary, but it hasn't learned how to chat with people yet.

Step 2: The Conversation Class (Supervised Fine-Tuning)

Next, they taught the robots how to follow instructions.

The "Implicit" Class: They taught the robots to be polite and helpful, but they didn't tell them to use a specific accent.
The "Explicit" Class: They gave the robots specific scripts written in the target dialect (e.g., "Write this story using Indian English slang"). This forced the robot to practice speaking that specific way.

Step 3: The Coaching Session (Alignment)

Finally, they used three different "coaching methods" (called DPO, GRPO, and GSPO) to refine the robot's speech. These methods act like a coach giving feedback: "That sounded good!" or "Try to sound more like a local."

The Twist: They tried to coach the robots using a "Reward System." They built a computer program that acts like a dialect detector. If the robot used a specific slang word or grammar rule, the detector gave it a high score (a reward).

The Surprising Discoveries

The researchers found two major things that surprised them:

1. The "Listening vs. Speaking" Split

They discovered that listening and speaking are controlled by different parts of the training.

The Analogy: Think of the "Immersion Camp" (Step 1) and the "Conversation Class" (Step 2) as the parts that teach the robot how to understand and generate the accent.
The Finding: The "Coaching Session" (Step 3) barely changed how well the robot performed on standard tests. However, it did change how the robot sounded. The tests (benchmarks) couldn't see the difference, but if you listened to the robot, you could hear it. The tests were blind to the accent change.

2. The "Reward Trap" (The Quality Gap)

This was the most interesting finding. The researchers tried to be very aggressive with the "Reward System." They told the robot, "Get as many points as possible for using dialect words!"

The Result: The robot that tried the hardest to get points (using a method called GRPO) actually sounded the least natural to human listeners.
The Analogy: Imagine a student trying to pass a test by memorizing a dictionary of slang words. They use the words correctly on paper, but when they speak, they sound like a robot reading a dictionary. They got the "points" (the reward), but they lost the "soul" of the conversation.
The Reality: Humans and other AI judges preferred the robot that was trained with a more balanced approach (SFTd) over the one that aggressively chased the reward points. The "chasing points" method actually made the dialect sound fake.

The Verdict: No "Magic Bullet"

The paper concludes that there is no single "best" way to teach a robot a dialect.

Explicit Targeting Works: If you want a robot to sound like it's from a specific place, you must train it specifically on that dialect (the "Explicit" path). Broad, general training doesn't work well.
The Reward System is Flawed: Just because a computer program says a robot is "using more dialect features" doesn't mean humans will think it sounds authentic. The "points" system used in this study didn't match human perception.

Summary

The DiaLLM project showed us that teaching an AI to speak a dialect is harder than just teaching it to understand one. You can't just "hack" it with a reward system that counts slang words; you have to carefully train the robot to understand the flow and feel of the language. If you push too hard for the "points," the robot sounds like a caricature rather than a real person.

The team released all their code and data so others can try to solve this puzzle better in the future.

Technical Summary: DiaLLM – An Investigation into the Robustness-Generation Gap in English Dialect Adaptation

1. Problem Statement

Large Language Models (LLMs) have demonstrated increasing capability in understanding dialectal English, yet they predominantly generate output in standard, US-leaning English. This creates a "robustness-generation gap": models can tolerate dialectal input (robustness) but fail to produce dialectally appropriate output (generation). Existing approaches, such as dialect adapters (e.g., TADA, HyperLoRA) or low-rank adaptations (LoRDD), primarily target Natural Language Understanding (NLU) robustness under dialectal input. None have systematically investigated full-pipeline adaptation—spanning continual pretraining, supervised fine-tuning (SFT), and alignment—nor have they determined whether models can genuinely produce dialectally marked text. Furthermore, current benchmarks often fail to capture the nuances of dialectal generation, focusing instead on input robustness.

2. Methodology

2.1 Base Models and Corpus

The study utilizes three open-weight LLM families spanning different architectures and parameter ranges: Llama 3.1-8B, Qwen 3-8B, and Gemma 3-4B-it.
The models are continually pre-trained (CPT) on the International Corpus of English (ICE), a collection of 18 English varieties (approx. 20 million tokens) covering both "Inner Circle" (e.g., UK, USA) and "Outer Circle" (e.g., India, Nigeria) varieties. CPT is performed using GaLore for memory-efficient full-parameter optimization.

2.2 Post-Training Paradigms

From the CPT checkpoints, the authors pursue two distinct adaptation threads:

Implicit Adaptation: Applies standard instruction-tuning (SFT) on general preference data (UltraFeedback) followed by broad alignment. This thread does not explicitly target specific dialects during SFT.
Explicit Adaptation: Applies dialect-specific SFT (SFTd) using a dataset where preferred completions are transformed into target dialects (Australian, Indian, Northern British English) using the Multi-VALUE library and eWAVE features. This is followed by variety-targeted alignment.

2.3 Alignment Strategies

Three alignment methods are compared across both paradigms:

Direct Preference Optimisation (DPO): An offline method optimizing directly from contrastive pairs (dialectal vs. standard).
Group Relative Policy Optimisation (GRPO): An online method generating multiple rollouts per prompt and estimating a baseline from group mean rewards.
Group Sequence Policy Optimisation (GSPO): Similar to GRPO but employs sequence-level reweighting instead of token-level importance sampling to avoid bias.

2.4 Reward Formulation

For online alignment (GRPO/GSPO), a composite reward function is used:
$R = \lambda \cdot \phi_{dial}(y) + \frac{1-\lambda}{2} \cdot \phi_{comet}(y, \hat{y}) + \frac{1-\lambda}{2} \cdot \phi_{cos}(y, \hat{y})$

$\phi_{dial}(y)$ : A log-sum reward derived from a multi-label classifier detecting 135 morphosyntactic and lexical features from the eWAVE typological database.
$\phi_{comet}$ and $\phi_{cos}$ : Metrics ensuring semantic fidelity and meaning preservation against the SFT chosen completion.
$\lambda$ is fixed at 0.80 to prioritize dialectal features.

2.5 Evaluation

Evaluation goes beyond standard benchmarks (BBH, GPQA, GLUE, VALUE, BESSTIE) to include:

Automatic Metrics: A variety classifier (DeBERTa-v3-base) to detect target dialects.
Human Preference: Native speakers perform pairwise or three-way comparisons of outputs.
LLM-as-Judge: Using Phi-4 to evaluate dialectal features, with and without explicit eWAVE feature lists in the system prompt.
Linguistic Analysis: A rule-based, independent analysis of surface markers (lexical, orthographic, morphosyntactic) to verify dialectal richness without relying on the training reward signal.

3. Key Contributions

DiaLLM Framework: A full-pipeline dialect adaptation framework providing a controlled comparison of CPT, SFT, and three alignment methods (DPO, GRPO, GSPO) across two paradigms and three model families. All code, checkpoints, and preference datasets are released.
Robustness-Generation Gap: The study establishes that benchmark performance (robustness) is primarily driven by CPT and SFT, while alignment methods have marginal effects on benchmarks but visibly reshape generation in ways benchmarks do not capture.
Reward-Quality Gap: The paper exposes a disconnect between optimizing dialectal reward signals and producing high-quality dialectal output. The method that most aggressively optimizes the dialectal reward (GRPO) is not preferred by human annotators or LLM judges, and independent linguistic analysis confirms it produces fewer recognizable surface markers than other methods.

4. Key Results

Dissociation of Robustness and Generation: Benchmarks are shaped by CPT and SFT. Alignment (DPO, GRPO, GSPO) produces inconsistent shifts in benchmark scores but significantly alters generation style.
Explicit Targeting is Preferred: Explicit variety-targeted adaptation (SFTd + variety-targeted alignment) produces output reliably recognized as dialectal and is preferred over broad alignment by both humans and LLM judges.
The Reward-Quality Gap:
- GRPO achieves the highest eWAVE feature density (the training objective) across all model families.
- However, GRPO is the least preferred method by human annotators and LLM judges.
- Independent linguistic analysis reveals that GRPO produces the fewest surface dialectal markers and lowest feature diversity for Llama and Qwen, despite maximizing the reward. This suggests the eWAVE-based reward signal encourages superficial feature density rather than authentic, holistic dialectal quality.
Variety-Specific Nuances:
- Indian English (en-IN) and Northern British English (en-UK) show clear preferences for explicit adaptation.
- Australian English (en-AU) is an exception; annotators show no clear preference for SFTd over the baseline, likely due to the subtlety of Australian features and their register flexibility, which casual prompts fail to elicit reliably.
Alignment Method Performance: No single alignment method dominates across all conditions. DPO often performs better than GRPO in terms of perceived quality, despite lower feature density scores.

5. Significance and Claims

The authors claim that DiaLLM provides a critical empirical account of dialect adaptation, challenging the assumption that maximizing a dialectal reward signal equates to generating authentic dialectal text.

Limitations of Benchmarks: The paper argues that relying solely on standard benchmarks gives an incomplete and potentially misleading picture of dialect adaptation, as they fail to capture the qualitative shifts in generation caused by alignment.
Reward Design: The findings highlight a specific limitation in using typological feature inventories (like eWAVE) as reward signals for generative training. These inventories document attested features but do not account for pragmatic conditioning, register, or the perceptual quality of dialect, leading to a "reward-quality gap."
Future Directions: The work suggests that closing the dialect gap requires richer reward designs that move beyond simple feature density and continued investment in dialectal resources. It positions DiaLLM not as a final solution, but as a foundational step toward understanding the current limits of dialect adaptation and the specific challenges in generating authentic, non-standard English varieties.

The authors emphasize that their goal was not to establish a single "winning recipe" but to empirically demonstrate how different pipeline components contribute to dialect adaptation and where current alignment methods fall short.