When the Gold Standard Isn't Necessarily Standard: Challenges of Evaluating the Translation of User-Generated Content

Problem Statement
Evaluating the machine translation (MT) of User-Generated Content (UGC) presents a unique challenge because UGC is characterized by frequent non-standard language, including spelling errors, slang, character repetitions, emojis, and code-switching. Unlike traditional text, there is no single "correct" way to translate these features; a "good" translation depends entirely on the desired level of standardness in the output. Current evaluation practices often assume a single gold standard, yet human reference translations for UGC vary significantly in how they treat non-standard phenomena. This variability makes reference-based automatic evaluation (AE) unreliable, as metrics may penalize valid stylistic choices that deviate from a specific reference's normalization strategy. Furthermore, it is unclear whether large language models (LLMs) can be effectively guided to adhere to specific translation guidelines and whether current metrics can fairly assess such guideline-adherent outputs.

Methodology
The authors address two research questions: (1) What constitutes a "good" translation of UGC? and (2) How can UGC translation be evaluated fairly?

1. Taxonomy and Guideline Analysis: The study analyzes four parallel UGC datasets (RoCS-MT, FooTweets, MMTC, and PFSMB) covering English-French, English-German, and French-English translation. The authors extract human translation guidelines from these datasets to derive a taxonomy of twelve non-standard phenomena (e.g., grammar, spelling, word elongation, capitalization, informal abbreviations, hashtags, profanity) and five translation actions:

   • NORMALISE: Correcting or omitting non-standard features to produce standard language.
   • COPY: Preserving the feature exactly as is.
   • TRANSFER: Mapping the feature to an equivalent non-standard form in the target language.
   • OMIT: Ignoring the feature entirely.
   • CENSOR: Replacing offensive language with milder terms.
     The analysis reveals that these datasets represent a spectrum of standardness, with RoCS-MT being the most standardizing and PFSMB the most preservation-oriented.

2. Experimental Setup: The authors evaluate six instruction-tuned LLMs (Gemma-2-9B, Granite-4.1-8B, LLaMA-3.1-8B, Mistral-0.3-7B, Qwen-2.5-7B, and Tower-0.2-7B) and one encoder-decoder baseline (NLLB-3B). The models are tested under three prompting conditions:

   • No-guidelines: Default zero-shot translation.
   • Matching guidelines: Prompts containing the specific guidelines of the target dataset.
   • Mismatching guidelines: Prompts containing guidelines from a different dataset.

3. Evaluation: Translation quality is assessed using reference-based metrics (COMET-22, BLEU) and reference-less metrics (COMET-Kiwi). Additionally, a human evaluation is conducted on RoCS-MT and PFSMB, where annotators compare default outputs against guideline-guided outputs based on overall quality and adherence to guidelines.

Key Results

• Guideline Sensitivity: LLMs are highly sensitive to explicit translation instructions. For most models (Gemma, Granite, Mistral, Qwen), providing corpus-specific guidelines significantly improves COMET scores when the guidelines match the dataset's underlying philosophy.
• Model Behavior:
  • LLaMA-3.1-8B frequently refuses to translate content deemed offensive or explicit, a behavior exacerbated by guidelines that instruct preserving profanity.
  • Tower-0.2-7B shows minimal sensitivity to guidelines, tending to stick to its inherent translation style regardless of prompts.
  • Gemma-2-9B and others successfully adapt their style to match guidelines, though they occasionally struggle with the composability of conflicting instructions (e.g., normalizing spelling while transferring character repetitions).
• Metric Limitations: Reference-based metrics (COMET, BLEU) are implicitly style-aware; they reward outputs that align with the reference guidelines and penalize mismatches. In contrast, the reference-less metric COMET-Kiwi exhibits a bias toward standardised outputs, assigning lower scores to preservation-oriented translations even when they align with the intended guidelines.
• Human Evaluation: Human annotators generally prefer outputs generated with matching guidelines over default outputs, particularly for preservation-oriented datasets like PFSMB (76% preference for quality, 90% for adherence). Agreement between human preferences and automatic metrics is moderate to substantial, though metrics remain influenced by the level of standardness favored by the evaluation setup.

Significance and Contributions
The paper argues that the "gold standard" for UGC translation is not absolute but context-dependent, varying from highly standardizing to preservation-oriented approaches. The authors demonstrate that fair evaluation requires both models and metrics to be aware of the specific translation guidelines governing the dataset.

Key contributions include:

1. A unified taxonomy of non-standard phenomena and translation actions derived from existing datasets.
2. Empirical evidence that LLMs can be steered to follow specific translation guidelines, but that their performance and the resulting evaluation scores are highly dependent on the alignment between the prompt, the model, and the reference style.
3. A call for clear guidelines during dataset creation and the development of controllable, guideline-aware evaluation frameworks. The authors suggest that future evaluation should either use reference-less metrics or reference-based metrics with multiple reference versions spanning different standardness levels, rather than relying on a single reference that may not represent the desired translation style.