Semi-Synthetic Parallel Data for Translation Quality Estimation: A Case Study of Dataset Building for an Under-Resourced Language Pair

Imagine you have a robot translator that speaks English and Hebrew. It's getting pretty good, but sometimes it makes silly mistakes, like getting the gender of a person wrong or mixing up singular and plural words.

The problem is: How do you teach a computer to know when the robot translator is making a mistake, without a human having to read every single sentence?

This is what "Quality Estimation" (QE) is. It's like a traffic cop for machine translation. Instead of checking every car (translation) against a perfect map (a human translation), the cop just looks at the car and says, "That looks safe, drive on!" or "That looks dangerous, stop and fix it!"

The paper you shared is about building a better "training school" for this traffic cop, specifically for the tricky English-to-Hebrew route. Here is the story of how they did it, explained simply.

1. The Problem: The "Empty Classroom"

Usually, to teach a computer how to grade translations, you need a huge library of examples where humans have already graded them. But for Hebrew, this library is tiny. It's like trying to teach a student to drive a race car, but you only have three practice tracks. The student (the AI model) gets confused and can't learn the nuances.

Hebrew is also a "morphologically rich" language. Think of English words as Lego bricks that stay mostly the same. Hebrew words are like shapeshifting clay; they change their shape completely depending on who is doing the action (gender), how many there are (number), and when it happened (tense). If the robot gets the shape wrong, the sentence breaks.

2. The Solution: Building a "Fake" Library

Since they didn't have enough real examples, the researchers decided to build a giant, semi-fake library. They called it a "Semi-Synthetic Dataset."

Here is how they built it, step-by-step:

• Step 1: The Blueprint (The Prompt): They took simple English sentences from a learner's dictionary (like "The men wore carnations"). They asked an AI (ChatGPT) to write new sentences that followed the same rules but were longer and more complex.
• Step 2: The Translation Factory: They fed these new sentences into three different robot translators (Google, Yandex, Microsoft).
• Step 3: The Quality Filter: They checked if the robots agreed. If two robots gave very different translations, they threw that sentence away. If they agreed, they kept it, assuming it was probably good.
• Step 4: The Human Inspector: A real human linguist read thousands of these and gave them a score from 1 (garbage) to 5 (perfect).

The Result: They had a decent pile of data, but it was still too small and unbalanced. The AI models trained on it were terrible. They kept guessing "3" (average) for everything because they hadn't seen enough "bad" examples to learn what really bad looks like.

3. The "Mad Scientist" Phase: Creating Mistakes on Purpose

This is the most creative part. The researchers realized, "We need more examples of bad translations to teach the AI what to avoid."

So, they became digital mad scientists. They took their good translations and intentionally broke them in specific ways:

• The Gender Bender: They took a sentence about a "he" and changed the Hebrew verb to look like it was about a "she."
• The Number Mixer: They changed "one cat" to "many cats" but kept the verb singular.
• The Word Jumble: They shuffled the words in the Hebrew sentence so it made no sense.

Every time they broke a sentence, they lowered its score.

• 1 mistake = Score drops by 1.
• 2 mistakes = Score drops by 2.
• Total gibberish = Score 0.

They also added a trick: They took an English sentence and paired it with a completely random Hebrew sentence that had nothing to do with it. They labeled these "Score 0" to teach the AI: "If the sentences don't match at all, that's the worst possible grade."

By doing this, they exploded their dataset from a few thousand examples to 400,000 examples.

4. The Training: Finding the Sweet Spot

Now they had a massive library. But how should they feed it to the AI? They tried three different "diets":

1. The Natural Diet (Normal Distribution): They gave the AI a mix that looked like real life (mostly good translations, fewer bad ones).
   • Result: The AI got okay (65% accuracy). It was still a bit lazy and guessed "average" too often.
2. The Balanced Diet (Uniform Distribution): They forced the AI to see an equal number of perfect, average, and terrible translations.
   • Result: Much better (88% accuracy). The AI learned to distinguish the "meh" from the "bad" because it saw them equally often.
3. The Feast (Large Scale): They just threw 1 million to 4 million examples at the AI, letting the natural mix happen but with huge volume.
   • Result: The Winner! (92% accuracy).

The Lesson: It turns out, volume is king. Even if the data is a bit messy or "fake," having so much of it that the AI sees every possible type of mistake over and over again makes it a genius.

5. The Future: The Idiom Challenge

The paper ends with a look ahead. The AI is now great at grammar and word order, but it still struggles with idioms.

• Example: If you say "It's raining cats and dogs," a literal translator might think animals are falling from the sky.
• The Plan: They are now building a special "Idiom Library" to teach the AI that sometimes, you have to translate the feeling, not the words.

Summary Analogy

Imagine you are teaching a child to spot a fake painting.

• Old Way: You show them 10 real paintings and 2 fake ones. The child gets confused and thinks everything is real.
• New Way (This Paper): You take 10 real paintings, photocopy them, and then use a marker to draw mustaches on 500 of them, scribble on 200, and burn 100. You show the child 1,000 paintings, half real and half "ruined."
• The Outcome: The child becomes a master art critic because they have seen every possible way a painting can go wrong.

This paper shows that for languages like Hebrew, where the rules are complex, you can't just rely on what nature gives you. You have to manufacture your own training data by breaking things on purpose, so your AI learns how to fix them.