💬 NLP

TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation

This paper introduces TQLite, a novel distillation framework that leverages a multi-LLM jury to generate high-quality synthetic training data, enabling small language models to achieve real-time, scalable translation quality evaluation performance comparable to computationally expensive large reasoning models.

Original authors: Bhavin Jawade, Cameron R. Wolfe

Published 2026-08-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bhavin Jawade, Cameron R. Wolfe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, global translation service. Every day, millions of sentences cross your desk, turning one language into another. But how do you know if the translation is actually good? In the old days, you'd hire a human expert to read every single sentence, but that's slow and expensive. Recently, scientists discovered that giant computer brains, called Large Language Models (LLMs), are surprisingly good at playing the role of these experts. They can read a translation, spot mistakes, and give it a grade, just like a teacher. However, these giant brains are like supercomputers: they are incredibly powerful but also hungry, expensive to run, and slow to answer. On the other end of the spectrum, there are "Small Language Models" (SLMs). These are like nimble, efficient pocket calculators. They are fast and cheap, but they often struggle with the complex reasoning needed to grade a translation accurately. The big question for the tech world is: Can we make these nimble calculators as smart as the supercomputers without losing their speed?

This is exactly what the team at Netflix set out to solve in their new paper, "TQLite." They realized that while the giant models are great at grading, they are too heavy to use for everything. So, they invented a clever trick called "distillation." Think of it like a master chef (the giant model) teaching a sous-chef (the small model) how to cook a perfect dish. Instead of just letting the sous-chef guess, the team gathered a "jury" of the smartest, most powerful AI chefs available. They asked this jury to grade thousands of translations and then had them vote on the final score. If all the chefs agreed on a grade, that became a "gold standard" example. They then used these high-quality examples to train the small, nimble models. The result? They created a system called TQLite, where the small models learned to grade translations almost as well as the giant, expensive ones, but much faster and cheaper.

The researchers started by testing the current "giants" to see who was the best at grading. They found that the most advanced "Reasoning Models" (LRMs)—a special type of AI that takes extra time to think through its answers—were the absolute champions. One specific model, when given a high level of "reasoning effort," achieved a system-level accuracy of 92.34%, which is a new high score. However, these models are slow and cost a lot of money to run. The team also noticed that when they asked these powerful models to be very strict about how they formatted their answers (like writing in a specific list format), they sometimes made more mistakes. But when they let them write in a slightly more flexible "structured text" format, the results were much better.

To build their "TQLite" system, the team didn't just pick one smart model to teach the small ones. Instead, they created a "Multi-LRM Jury." Imagine a panel of three different expert judges. For every translation, they asked all three to give their opinion. If the judges all agreed (or mostly agreed) on the score, the team knew that answer was trustworthy. They used this agreement as a filter, throwing out any examples where the judges were arguing with each other. They then took these "agreed-upon" examples and used them to train small, open-source models called Gemma-12B-it and Gemma-3-4B-it.

The results were impressive. The small models, after being trained on this high-quality "jury" data, jumped from a segment-level accuracy of about 52.6% (when they were just regular, untrained models) to 55.03%. This might not sound like a huge jump, but in the world of AI grading, it's a massive leap. It means the small models are now beating other open-source models that are much larger and more complex. In fact, the team found that the small, distilled models performed better than all the other open-source "Reasoning Models" they tested, and they got very close to the performance of the most expensive, closed-source giants.

Perhaps the most exciting part of their discovery is the trade-off between speed and smarts. The team plotted a graph showing how long it takes to grade a translation versus how accurate the grade is. The giant, expensive models are at the top for accuracy but take a long time to think. The untrained small models are fast but not very accurate. The TQLite model, however, sits in the sweet spot: it is significantly faster than the giants while maintaining a level of accuracy that is nearly as good. It's like having a race car that drives as fast as a motorcycle but handles the turns as well as a luxury sedan.

The researchers were careful to note that this isn't a magic wand that solves everything. The small models still don't quite reach the very peak performance of the absolute best, most expensive closed-source models. Also, their training data was mostly made up of examples where the jury of experts agreed perfectly. This means the small models are great at handling clear-cut cases, but they might struggle with the weird, edge-case translations where even the experts disagree. They also tested this on languages like English, German, Chinese, and Russian, but they haven't yet proven it works for languages with very few speakers or very complex grammar structures.

In the end, TQLite offers a practical path forward. It shows that we don't necessarily need to run massive, expensive supercomputers to get high-quality translation grading. By using a "jury" of the smartest models to create a perfect textbook, we can teach smaller, efficient models to do the heavy lifting. This means companies can grade translations in real-time, saving money and time, without sacrificing too much quality. It's a reminder that sometimes, the best way to get smarter isn't to build a bigger brain, but to teach a smaller one how to think like a team of geniuses.

Technical Summary: TQLite – Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation

1. Problem Statement

Translation Quality (TQ) evaluation is critical for machine translation systems, with the Multidimensional Quality Metrics (MQM) framework serving as the gold standard. While Large Language Models (LLMs) and Large Reasoning Models (LRMs) have demonstrated state-of-the-art performance in MQM-based evaluation, they face significant scalability challenges:

Computational Cost: Frontier-level LLMs and LRMs (e.g., GPT-4, o1, o3) are computationally expensive to deploy at scale, making them impractical for real-time or high-volume evaluation.
Performance Gap: Small Language Models (SLMs), while efficient, struggle with the complex reasoning required for accurate MQM evaluation, often lagging significantly behind their larger counterparts.
Closed-Source Dependence: Current best-performing evaluators are predominantly closed-source, limiting reproducibility and increasing costs.

The core challenge is to develop a TQ evaluation system that retains the high accuracy of frontier LLMs/LRMs while achieving the efficiency and openness of SLMs.

2. Methodology: The TQLite Framework

The authors propose TQLite, a knowledge distillation framework designed to transfer the evaluation capabilities of a "jury" of high-performing LRMs into smaller, open-source SLMs. The methodology consists of three primary stages:

A. Benchmarking and Jury Selection

Before distillation, the authors conducted an extensive benchmark of LLMs and LRMs on the WMT22 metrics test set (covering zh-en, en-de, en-ru).

Findings: LRMs (particularly closed-source models like o3 with high reasoning effort) achieved the highest system-level accuracy. However, they exhibited variability at the segment level.
Jury Construction: The authors selected a diverse panel of top-tier LRMs (including OpenAI's o1, o3, o4-mini, and Google's gemini-2.5) to serve as the "teacher" jury.
Prompt Engineering: The study refined the standard Gemba-MQM prompt to resolve ambiguities in output formatting. They determined that while strict structured output (JSON or structured text) can degrade performance for smaller models, it is necessary to avoid heuristic parsing errors. They adopted a single-turn prompt with structured text output as the optimal balance.

B. Synthetic Data Generation via Multi-LRM Jury

To create high-quality training data without human annotation, the authors employed a multi-step aggregation process:

Multi-Model Inference: For each translation segment, the jury of LRMs generates error span annotations (ESAs) and MQM scores.
Agreement Filtering: The system calculates the score range ( $\Delta_i$ ) across the jury. Segments are categorized by agreement levels (Perfect, High, Moderate, Low, No agreement). Only segments with Perfect or High agreement are retained as high-confidence candidates.
Meta-Judge Refinement: A meta-judge LLM (GPT-4o-mini) processes the outputs of the agreeing LRMs. It identifies overlapping error spans with different boundaries, merges them into canonical annotations, and ensures semantic consistency in error categories and severities. This produces a clean, consolidated set of synthetic labels.

C. Student Model Distillation

The filtered synthetic dataset is used to train smaller open-source models (specifically Gemma-3-4b-it and Gemma-3-12b-it) via Supervised Fine-Tuning (SFT).

Training Setup: The student models are trained to replicate the structured output format (error spans, categories, severities) of the jury.
Parameter Efficiency: The authors utilized LoRA (Low-Rank Adaptation) for efficient fine-tuning, finding that a LoRA rank of 128 yielded optimal performance for the 12B model.

3. Key Results

The experiments were conducted on the WMT22 metrics test set, measuring both system-level and segment-level pairwise accuracy.

Distillation Efficacy: TQLite-distilled SLMs significantly outperformed their pre-trained counterparts and all open-source LRM baselines.
- The distilled Gemma-3-12b-it achieved an average segment-level accuracy of 55.03%, surpassing open LRM baselines (e.g., Qwen-3-32b-thinking at 52.74%) and approaching the performance of top closed-source models.
- System-level accuracy for the distilled 12B model reached 85.04%.
Data Quality Impact: The study demonstrated a strong correlation between jury agreement and model performance. Training exclusively on "Perfect Agreement" samples yielded the highest accuracy, while including lower-agreement samples degraded performance.
Scaling Laws: Performance improved consistently with the number of training samples, rising from ~52.6% accuracy at 5k samples to ~55.3% at 99k samples, with no signs of plateauing within the tested range.
Efficiency vs. Performance: TQLite models offer a superior accuracy-to-inference-time trade-off. While frontier LRMs (like o3) achieve slightly higher accuracy, TQLite-based SLMs provide competitive performance at a significantly lower inference cost and latency.

4. Contributions

Comprehensive Benchmark: The paper provides a detailed benchmark of modern LLMs and LRMs for TQ evaluation, establishing that while closed-source LRMs lead in performance, open-source models can be competitive with proper prompting and distillation.
TQLite Framework: Introduction of a novel distillation pipeline that leverages a multi-LRM jury and meta-judge refinement to generate high-quality synthetic training data.
Scalable SLM Judges: Demonstration that SLMs, when distilled via TQLite, can approach the MQM evaluation capabilities of the best LRM-based evaluators, offering a cost-effective and open alternative for real-time TQ evaluation.

5. Significance and Claims

The authors claim that TQLite offers a scalable and cost-effective alternative to the current reliance on expensive, closed-source LLMs and LRMs for translation quality evaluation. By distilling the consensus of a diverse jury of powerful models into efficient open-source SLMs, the framework bridges the gap between high-performance evaluation and practical deployment constraints.

The paper emphasizes that while distilled SLMs do not yet fully match the absolute ceiling of the best closed-source frontier models, they represent a significant leap forward for open-source TQ evaluation, enabling high-quality, real-time assessment without prohibitive computational costs.

6. Limitations

The authors acknowledge several limitations:

Performance Ceiling: SLM-based evaluators still fall short of the absolute best LRM-based judges.
Bias Inheritance: Synthetic data generated by LLM juries may cause student models to inherit systematic biases from their teachers, despite aggregation efforts.
Coverage: The filtering process for high-agreement samples may reduce coverage of edge cases and difficult translation errors.
Language Scope: Experiments were limited to high- and medium-resource language pairs (WMT22); effectiveness for low-resource or morphologically complex languages remains untested.