Beyond "To whom it may concern": Tailoring Machine Translation to Audience and Intent

Technical Summary: Beyond "To whom it may concern": Tailoring Machine Translation to Audience and Intent

Problem Statement

Traditional Machine Translation (MT) systems and their evaluation metrics treat translation as a fixed mapping from source to target, optimizing for semantic adequacy against a single "correct" reference. This approach ignores the pragmatic reality that translation quality is relative to purpose, as argued by Skopos theory. A single source text may require vastly different translations depending on the intended audience, tone, register, and communicative intent (e.g., translating "Excuse me" as a formal apology versus a casual interruption). While Large Language Models (LLMs) theoretically enable users to specify these contextual parameters via prompts, it remains unclear whether current models can systematically leverage such specifications to produce purpose-adapted translations across diverse languages, model sizes, and domains. Furthermore, existing research on context-aware MT has been limited in scope, often focusing on single language pairs or narrow phenomena like formality, and traditional reference-based metrics (e.g., COMET, ChrF) may fail to capture or even penalize successful stylistic adaptation.

Methodology

The authors introduce a systematic evaluation framework for purpose-driven MT, conditioning translation on both the source text and a natural-language instruction describing the intended purpose.

• Datasets and Scope: The study evaluates performance across 50+ languages, 5 model sizes (Gemma-3-4B/12B/27B, Gemma-4-31B, and Qwen3.5-27B), and 8 text domains. The primary benchmark is BOUQuET (504 instances per language), supplemented by WMT24++ (997 instances per language) for validation. The evaluation focuses on English-to-target language translation.
• Instruction Generation: For BOUQuET, instructions are drafted by an LLM (Gemini 3 Flash) based on metadata (domain, register, context comments) and manually revised by human annotators. These instructions specify the target audience, tone, and purpose.
• Experimental Conditions: The study compares four translation conditions:
  1. Baseline: Sentence-level translation with a fixed 3-shot prompt, no instructions.
  2. Instructions: The baseline prompt augmented with the user's specific instruction.
  3. Few-Shot: The baseline prompt augmented with 5 semantically similar translation pairs retrieved from a held-out set.
  4. Paragraph: The full source paragraph provided as context (with markers for the target sentence or as a numbered list).
• Evaluation Metrics:
  • Translation Rating: A 0–100 score measuring meaning preservation, following the Error Span Annotation (ESA) protocol.
  • Adaptedness: A novel 0–100 score measuring how well the translation matches the instruction's intended purpose (tone, register, audience).
  • LLM-as-Judge: A reference-free evaluation pipeline using an LLM (Gemini-3-Flash) to score both metrics, validated against human annotations from native speakers.
  • Traditional Metrics: XCOMET-XL and ChrF++ are used to benchmark against standard MT evaluation.

Key Contributions and Results

1. Instructions Substantially Improve Adaptedness
Explicit user instructions significantly enhance the "adaptedness" of translations.

• Scaling: Gains in adaptedness scale with model size. Larger models (12B, 27B, 31B) show consistent improvements (e.g., +10.2 to +17.8 adaptedness points), whereas smaller models (4B) exhibit mixed results, sometimes suffering from critical failures (wrong language or no translation) when instructions are added, particularly in low-resource languages.
• Domain Sensitivity: The benefits are most pronounced in informal domains (conversation, social media), where adaptedness gains reach +16.5 to +26.1 points. Formal domains (news, how-to) show smaller gains.
• Language Resource Level: Higher-resource languages benefit more from instructions. Additionally, "register-rich" languages (those with grammaticalized social distinctions, e.g., Japanese, Arabic) show the largest gains.

2. Instructions Outperform Alternative Context Methods
Explicit instructions are superior to both semantic few-shot examples and paragraph-level context.

• vs. Few-Shot: Instructions yield significantly higher adaptedness gains than semantically matched few-shot examples (e.g., +11.1 vs. +2.9 for the 12B model).
• vs. Paragraph Context: Providing raw paragraph context often fails to improve adaptedness and can even degrade translation quality (negative deltas) for models ≤27B. Explicit instructions are the most reliable method for adaptation.

3. Traditional Metrics Fail to Capture Adaptation
There is a significant divergence between LLM-as-judge scores and traditional metrics.

• When instructions are added, LLM-as-judge adaptedness scores increase, but XCOMET-XL and ChrF++ scores often decrease.
• This divergence is concentrated in informal domains (21–50% disagreement rate), suggesting that traditional metrics penalize necessary stylistic shifts (e.g., making text more colloquial) because they rely on reference translations that may be overly formal or lack the specific stylistic nuance requested.

4. Self-Instruction from Document Context
When curated instructions are unavailable, models can generate their own instructions from surrounding document context.

• Effectiveness: Self-generated instructions close up to 80% of the adaptedness gap compared to curated gold instructions (e.g., 71% for Gemma-3-27B, 80% for Gemma-4-31B).
• Mechanism: This approach transforms potentially harmful raw paragraph context (which can lower adaptedness by -2.3 to -10.6 points when fed directly) into a beneficial brief. The model effectively "guesses" the context and tone, recovering most of the benefits of a human-curated brief.

5. Mitigating Small Model Failures
For smaller models (4B) on low-resource languages, instructions can trigger catastrophic failures (e.g., 45.5% failure rate on Javanese). The authors demonstrate that Parameter-Efficient Fine-Tuning (PEFT) via LoRA distillation from a larger teacher model (27B) can virtually eliminate these critical failures (reducing rates to ~4.0%) while preserving the ability to adapt to instructions.

Significance and Claims

The paper establishes that purpose-adapted MT is a viable and measurable capability of current LLMs. The authors argue that:

• Evaluation Paradigm Shift: Translation evaluation must move beyond static, reference-based metrics to include purpose-aware metrics (like adaptedness) that account for audience and intent.
• Practical Deployment: For models with ≥12B parameters, adding user instructions is a reliable mechanism to improve translation quality, particularly for informal content.
• Autonomous Adaptation: When explicit user instructions are absent, capable models can self-generate effective instructions from document context, offering a practical path for document-level MT that does not require manual briefs.
• Limitations: The authors note that their findings are based on specific model families (Gemma, Qwen) and that the "instruction-following" capability competes with multilingual generation capacity in smaller models, requiring techniques like distillation to stabilize performance.

The work concludes that while LLMs have opened new, iterative translation capabilities, realizing their full potential requires evaluation frameworks that can measure these adaptive qualities, moving the field toward a more user-centered paradigm.