💬 NLP

Fabricator or dynamic translator?

This paper explores strategies for detecting and categorizing the diverse overgenerations of Large Language Models in machine translation—ranging from confabulations to helpful explanations—and presents results from their implementation in a commercial setting.

Original authors: Lisa Vasileva, Karin Sim

Published 2026-04-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lisa Vasileva, Karin Sim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hired a brilliant, hyper-creative translator to convert a book from English to another language. In the past, these translators were like photocopiers: they were very strict, sometimes making weird mechanical errors (like repeating a word 50 times), but they generally stuck to the original text.

Now, we have Large Language Models (LLMs). These are like improvisational jazz musicians. They are incredibly talented, fluent, and can sound very human. But because they are so creative, they sometimes decide to add their own "solo" to the song. They might explain why they chose a word, apologize for a difficult phrase, or even invent a detail that wasn't in the original story just to make the sentence flow better.

This paper, written by experts from Language Weaver, is a report on how to catch these "jazz solos" when they are actually mistakes, without accidentally punishing the translator for being helpful.

Here is the breakdown of their findings using simple analogies:

1. The Problem: "The Over-Explainer"

The authors noticed that while old translation models made obvious mistakes (like a robot stuttering), these new AI models make subtle mistakes. They call these "Overgenerations."

Think of it like a tour guide:

The Good Guide: "Here is the Eiffel Tower." (Accurate translation).
The Bad Over-Generator: "Here is the Eiffel Tower, which is 330 meters tall, built in 1889, and by the way, I think the weather is nice today."
- Sometimes the extra info is helpful (like expanding an acronym).
- Sometimes it's a lie (confabulation) or a refusal to translate ("I can't translate this because it's rude").
- Sometimes it's just a tiny, unnecessary word added to make the sentence sound smoother.

The problem is that these extra words are so fluent and natural that it's hard to tell if they are a mistake or good translation.

2. The Four Types of "Jazz Solos"

The authors categorized these extra additions into four levels of "detachment" from the original text:

The Broken Record (Oscillatory): The AI gets stuck and repeats a word like a skipping CD. Easy to spot.
The Ghost Writer (Detached): The AI ignores the source text entirely and writes a whole new paragraph or refuses to translate. Easy to spot.
The Chatty Neighbor (Partially Detached): The AI translates the text but adds a prefix like, "Sure! Here is the translation:" or adds a long explanation in the middle. Moderately easy to spot.
The Silent Editor (Minimally Detached): This is the hardest one. The AI translates perfectly but sneaks in one or two extra words that make the sentence sound more natural but weren't in the original.
- Example: Source says "NSW government." Target says "New South Wales government."
- Is this a mistake? No, it's actually helpful for the reader! But an automated checker might think it's an error because the words don't match 1-to-1.

3. The Detective Tools

The team tried two different ways to catch these "over-generations":

Tool A: The Quality Inspector (MTQE)

This is like a school teacher grading an essay. It looks at the whole translation and asks, "Does this look like a high-quality translation?"

Pros: It's good at spotting the "Ghost Writer" (refusals) and the "Chatty Neighbor."
Cons: It's often fooled by the "Silent Editor." If the sentence sounds perfect, the teacher gives it an A, even if it added extra words.

Tool B: The Word-for-Word Tracker (CheckAlign)

This is like a spot-checker with a highlighter. It tries to match every word in the source to a word in the target. If it finds words in the target that don't have a partner in the source, it flags them.

Pros: It catches the "Silent Editor" and the "Chatty Neighbor" very well.
Cons: It gets confused by good translation habits. Sometimes, a translator should add words to make sense (like expanding "NSW" to "New South Wales"). The tracker thinks this is a mistake, but it's actually Explicitation (making things clearer for the reader).

4. The "Ensemble" Solution

Since neither tool is perfect, they combined them into a Team of Detectives.

If the Teacher or the Tracker flags a sentence, they mark it for review.
This combination catches almost all the bad stuff, but it still has a problem: False Alarms.

5. The Big Dilemma: Mistake vs. Helpfulness

The most interesting part of the paper is the realization that sometimes the AI is right to add words.

Imagine the source text says, "Al's was around the corner."

Literal Translation: "Al's was around the corner." (Confusing: Al's what? A house? A person?)
AI Translation: "Al's shop was around the corner."

The AI added the word "shop."

The Tracker says: "Error! 'Shop' wasn't in the original!"
The Human says: "Actually, that's great! It helps the reader understand."

The paper concludes that the hardest job for AI is distinguishing between:

Confabulation: Lying or adding nonsense.
Explicitation: Adding helpful context (like a good human translator would do).

The Takeaway

We have moved from a world where translation errors were obvious (like a robot stuttering) to a world where errors are sneaky and subtle.

The authors found that while we can catch the big, obvious mistakes easily, catching the tiny, "helpful-sounding" additions is very hard. The future isn't just about building better detectors; it's about teaching them the difference between a creative mistake and creative helpfulness.

In short: The AI is becoming such a good translator that it's starting to act like a human editor. The challenge is figuring out when to let it be creative and when to tell it to stick to the script.

1. Problem Statement

The paper addresses the emergence of overgenerations in Large Language Model (LLM) based Machine Translation (MT). Unlike traditional Neural Machine Translation (NMT), which primarily suffered from "neurobabble" (repetitions, incoherent strings) or clear hallucinations, LLMs exhibit a broader spectrum of overgenerations. These include:

Self-explanations: The model adds instructions or apologies (e.g., "I apologize, but I cannot translate...").
Confabulations: Fluent, on-topic content that is not present in the source text but is fabricated by the model.
Explicitation: The model expands acronyms or cultural references to aid target audience comprehension (a behavior often desirable in human translation but problematic for strict MT evaluation).

The Core Challenge: Detecting these overgenerations is difficult because they are often fluent and semantically coherent. Existing Machine Translation Quality Estimation (MTQE) models and hallucination detectors (designed for NMT) fail to distinguish between erroneous confabulations and beneficial explicitation, leading to high false positive rates or missed detections.

2. Methodology

A. Taxonomy of Overgenerations

The authors propose a refined categorization of overgenerations, expanding on previous NMT taxonomies:

Oscillatory: Repetitive tokens or symbols (neurobabble).
Detached: Content entirely absent from the source (e.g., refusal to translate, long explanations).
Partially Detached: Fluent output with chunks of unaligned content (e.g., prefixes like "Translation into Italian:").
Minimally Detached: The most challenging category. Fluent, on-topic output containing only a few extra tokens (e.g., adding a word for clarity or a minor confabulation).

B. Dataset Construction

Due to the lack of public datasets specifically annotated for LLM overgenerations, the authors constructed a multi-source dataset:

Open Source: WMT24 AOC task data (filtered for low scores) and the Deepspin NMT dataset.
Internal Data:
- R&D Testset: Observed failures in generative models.
- APE Testset: Errors reported during Automatic Post-Editing.
- Prototype & Synthetic: Customer issues and artificially generated prefixes.
- Minimally Detached Set: A specific English-Italian set annotated by linguists to capture subtle confabulations.

C. Detection Strategies

The authors evaluated two distinct strategies and an ensemble approach:

MTQE (Quality Estimation):
- Approach: Fine-tuned an internal multilingual encoder-based regression model (based on XLM-R Large) on synthetic data containing overgenerations.
- Goal: Predict a quality score (0.0–2.0) to flag low-quality segments containing hallucinations.
CheckAlign (Alignment-based):
- Approach: Uses AwesomeAlign (leveraging multilingual BERT embeddings) to align source and target tokens.
- Logic: Identifies "detached chunks" where $n$ consecutive target tokens have no alignment to the source.
- Threshold: Set to $n=2$ or $n=3$ to catch minimally detached confabulations while avoiding false positives from single-word alignment variations.
Ensemble:
- Combines MTQE and CheckAlign. Any segment flagged by either model is marked as an overgeneration.

3. Key Results

The performance varied significantly depending on the type of overgeneration and the dataset:

Oscillatory & Detached Overgenerations: Both MTQE and CheckAlign performed with high accuracy (F1 > 0.85). These are easier to detect as they are either incoherent or clearly unaligned.
Partially Detached: The Ensemble approach showed robust performance, leveraging the strengths of both models.
Minimally Detached (The Critical Challenge):
- MTQE: Struggled significantly (Precision ~0.18, Recall ~0.18 in some internal sets), failing to detect subtle additions.
- CheckAlign: Achieved higher recall (0.77) but very low precision (0.22).
- Analysis of False Positives: The low precision was not just due to alignment errors but because the model flagged valid explicitations.
  - Example: Expanding "NSW" to "New South Wales" or "Al's" to "Al's shop" to aid comprehension.
  - The alignment-based model correctly identified these as "unaligned" but incorrectly classified them as errors, whereas human translators view them as good translation practices.

4. Key Contributions

Refined Taxonomy: Introduced the category of "Minimally Detached Overgenerations," distinguishing subtle confabulations from beneficial explicitation, a nuance often missed in prior NMT research.
Dataset Creation: Released a new dataset (via GitHub) containing annotated examples of LLM overgenerations, specifically addressing the scarcity of labeled data for this phenomenon.
Dual-Strategy Evaluation: Demonstrated that a single detection method is insufficient. While MTQE handles general quality drops, alignment-based methods are necessary for detecting specific unaligned chunks, though they require careful tuning to avoid penalizing good translation practices.
Explicitation vs. Confabulation: Provided empirical evidence that the line between "hallucination" and "explicitation" is blurry. The paper highlights that LLMs often perform "dynamic translation" (adding context) which mimics confabulation but serves a functional purpose.

5. Significance and Future Work

Commercial Impact: For Language Service Providers (LSPs), blindly filtering LLM output based on current detectors risks removing high-quality, fluent translations that simply added necessary context.
The "Fabricator or Translator" Dilemma: The paper concludes that LLMs are not just "fabricators" of errors; they are "dynamic translators" that sometimes over-generate to improve comprehension.
Future Directions: The authors propose a Human-in-the-Loop pipeline. By collecting false positives (where the model flagged an error but a human confirmed it was a valid explicitation), they aim to fine-tune models to distinguish between erroneous confabulation and beneficial explicitation.

In summary, the paper argues that detecting LLM overgenerations requires moving beyond simple "source-groundedness" metrics. Future systems must understand the intent of the addition—whether it is a risky hallucination or a helpful clarification for the target audience.