Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translation

1. Problem Statement

The paper addresses two critical challenges in translating Standard Korean into regional dialects (specifically Jeolla, Gyeongsang, and Jeju) using Large Language Models (LLMs):

• The Dialect Gap: LLMs struggle to generate authentic dialectal outputs, often defaulting to Standard Korean even when prompted for a specific dialect. This results in a "False Success" scenario where models appear to perform well because they simply copy the source text, rather than translating it.
• Evaluation Distortion: Traditional n-gram metrics (BLEU, chrF++) are structurally biased toward surface-level overlap with the source. Consequently, they reward "False Success" (copying the standard source) and penalize "True Attempts" (genuine dialectal generation that deviates lexically from the source). This makes it difficult to accurately assess dialect translation quality.

2. Methodology: The DIA-REFINE Framework

The authors propose DIA-REFINE (Dialect Refinement), an iterative framework designed to guide LLMs toward faithful target dialect outputs using external feedback.

A. Core Components

1. Ensemble Dialect Classifier ($C$):

   • The authors constructed a dataset of 50,000 samples covering five classes: Jeolla, Gyeongsang, Jeju, Standard, and Unknown (mixed dialects).
   • They fine-tuned five distinct Korean pre-trained language models (e.g., KcELECTRA, KoSimCSE-RoBERTa) and created an ensemble.
   • The best-performing ensemble achieved 94.92% accuracy, serving as an external "verifier" to check if an LLM's output matches the target dialect.

2. Iterative Refinement Loop:

   • Generation: The LLM generates a translation candidate ($y$).
   • Verification: The ensemble classifier $C$ predicts the dialect of $y$.
   • Feedback:
     • If $C(y) \neq$ Target Dialect, the system provides explicit feedback to the LLM, specifying the incorrect class predicted (e.g., "The output was classified as Jeolla, not Jeju").
     • Oscillation Handling: If the model oscillates between different error types on consecutive retries, a specific prompt flags this pattern to break the cycle.
   • Retry: The LLM re-translates based on the feedback. This loop allows up to two retries (3 total attempts).

3. Variants:

   • DIA-REFINE(S): Single-candidate approach (generate $\to$ verify $\to$ retry).
   • DIA-REFINE(M): Multi-candidate approach. The model generates $k$ candidates (starting at $k=3$), and the system selects the one with the highest posterior probability of being the target dialect before verification. This broadens the search space.

B. Experimental Setup

• Models Tested: HyperCLOVAX, EXAONE-3.5, EEVE, Llama-3.1, and Gemini-1.5.
• Baselines: Zero-shot (ZS) and In-Context Learning (ICL) without refinement.
• Data: Standard Korean sentences translated into Jeolla, Gyeongsang, and Jeju dialects.

3. Key Contributions: New Evaluation Metrics

To solve the evaluation distortion problem, the authors introduced two new metrics that replace reliance on n-gram overlap:

1. Dialect Fidelity Score (DFS):

   • A continuous metric measuring the linguistic shift of the hypothesis ($h$) toward the dialect reference ($r$) versus the standard source ($s$).
   • Calculated using the log-ratio of cosine similarities in an embedding space (using a dialect-aware model):
     $$DFS = \log \left( \frac{1 + \cos(e_h, e_r) + \epsilon}{1 + \cos(e_h, e_s) + \epsilon} \right)$$
   • Positive DFS: Indicates a shift toward the dialect.
   • Negative DFS: Indicates proximity to the standard language.
   • Significance: It correctly identifies "True Attempts" (high DFS) even when n-gram scores are low.

2. Target Dialect Ratio (TDR):

   • The proportion of generated outputs correctly classified as the target dialect by the ensemble classifier $C$.
   • Provides a direct measure of translation success rate.

4. Results and Analysis

A. Unmasking "False Success"

• N-gram Failure: In Zero-shot settings, models like HyperCLOVAX and Llama-3.1 achieved high BLEU/chrF++ scores (e.g., 60 chrF++) but had near-zero TDR (<0.03) and negative DFS (-0.37). This confirmed they were copying the standard source.
• Metric Validation: The proposed metrics successfully distinguished these failures. For instance, Gemini-1.5 had low BLEU scores in ZS but a TDR of 0.93, correctly identifying it as the best performer despite traditional metrics suggesting otherwise.

B. Performance of DIA-REFINE

• DIA-REFINE(M) Superiority: The multi-candidate variant consistently outperformed the single-candidate version.
  • ICL + DIA-REFINE(M) yielded the best results across all models.
  • Gemini-1.5 achieved a TDR of 0.99 and a positive DFS of 0.39 with ICL + DIA-REFINE(M), while maintaining reasonable n-gram scores.
  • EXAONE-3.5 showed the most dramatic improvement, transforming from a failing model (negative DFS) to a high-performing one (DFS 0.27, TDR 0.94) when combined with ICL and DIA-REFINE(M).
• Efficiency: DIA-REFINE(M) achieved higher success rates with fewer attempts (avg. 2.81 attempts) compared to DIA-REFINE(S) (avg. 2.95 attempts) in difficult scenarios.

C. Model Responsiveness

• Models exhibited varying degrees of responsiveness. Models prone to "False Success" (copying) were harder to steer than those making "True Attempts."
• However, even initially failing models showed substantial improvement when the framework was applied, proving the method's robustness.

5. Significance and Conclusion

• Framework Innovation: DIA-REFINE provides a robust, goal-directed mechanism for dialect translation that does not require fine-tuning the LLM itself, relying instead on external classifier feedback.
• Evaluation Paradigm Shift: The paper establishes that n-gram metrics are fundamentally flawed for dialect translation. The introduction of DFS and TDR offers a rigorous, reliable alternative that aligns with human intuition regarding dialectal fidelity.
• Inclusivity: By successfully translating under-researched Korean dialects (Jeolla and Gyeongsang) and providing tools to evaluate them, the work advances linguistic inclusivity and preserves regional linguistic heritage.
• Future Work: The authors plan to release code to facilitate further research, though they note limitations regarding semantic preservation and the need for native speaker verification for fluency.

In summary, this paper demonstrates that iterative refinement with external feedback effectively steers LLMs toward authentic dialect generation, and that embedding-based metrics (DFS/TDR) are essential for accurately evaluating these outputs, correcting the biases inherent in traditional machine translation evaluation.