Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation

1. Problem Statement

The generation of high-quality Supervised Fine-Tuning (SFT) data for less-resourced languages is a critical bottleneck in developing inclusive multilingual Large Language Models (LLMs). While synthetic data generation using a "teacher" LLM to train a smaller "student" model is a common solution, current practices suffer from two main issues:

• Ad Hoc Teacher Selection: Researchers typically default to the largest available models (e.g., Llama 3.1 70B) assuming size correlates with capability. However, these models often exhibit significant capability gaps in non-English languages, leading to poor-quality synthetic data that propagates weaknesses rather than strengths.
• Lack of Systematic Evaluation: There is no standardized metric to evaluate how effective a specific teacher model is for generating synthetic data in a specific target language. Existing evaluations often focus solely on English tasks or intrinsic data quality without measuring downstream student performance.

2. Methodology

The authors propose a comprehensive evaluation framework called POLYGLOT SCORE (PG-SCORE) to systematically assess language models as multilingual teachers.

A. Experimental Setup

• Teacher Models: 10 state-of-the-art LMs were evaluated, varying in scale (0.4B to 104B parameters) and family (Gemma 3, Llama 3.1, Aya Expanse, Command A, IBM Granite, GPT-4o mini).
• Target Languages: 6 typologically diverse languages were selected: Arabic, Czech, German, Spanish, Indonesian, and Japanese.
• Data Generation: Three distinct methods were tested for each teacher-language pair:
  1. Generate: Few-shot generation of new prompt-response pairs.
  2. Translate: Forward-translating English prompts to the target language and generating responses.
  3. Respond: Generating responses to existing prompts in the target language.
• Scale: Over 1.4 million synthetic SFT examples were generated, and 240 student models (based on OLMo 3 7B, and later Llama 3.1 8B, Gemma 3 4B, Qwen 3 8B) were fine-tuned.

B. The POLYGLOT SCORE (PG-SCORE)

The metric combines intrinsic data quality and extrinsic student performance into a single normalized score:

1. Intrinsic Metrics ($Intr_{T,\ell}$):
   • Diversity: Cosine distance of prompt and response embeddings (using Llama-Embed-Nemotron-8B).
   • Perplexity (PPL): Fluency and naturalness of the response.
   • Reward Score: Quality assessment by a multilingual LLM-as-a-judge (M-Prometheus 14B) based on fluency, naturalness, and instruction following.
2. Extrinsic Metrics ($Extr_{T,\ell}$):
   • Performance Gap Recovered (PGR): Measures the improvement of the student model over a base model relative to a reference model on three benchmarks:
     • Cultural/Factual Knowledge: Global-MMLU Lite.
     • General Chat: M-RewardBench.
     • Mathematical Reasoning: M-GSM.
3. Final Score: $PG\text{-}SCORE = z\text{-}score(Intr + Extr)$.

3. Key Contributions

1. Systematic Evaluation: The first large-scale study evaluating 10 teacher models across 6 diverse languages, generating 1.4M+ examples and training 240 student models.
2. PG-SCORE Metric: Introduction of a holistic metric that balances intrinsic data quality with extrinsic downstream performance, addressing the gap in existing evaluation protocols.
3. Counter-Intuitive Findings: Demonstrated that model scale alone is not a predictor of teacher effectiveness.
4. Actionable Recipe: Provided a practical framework for generating multilingual synthetic data, including specific recommendations on teacher selection, model family matching, and data generation methods.

4. Key Results

A. Teacher Effectiveness

• Top Performers: Gemma 3 27B and Aya Expanse 32B consistently ranked as the most effective teachers across different student base models and languages.
• Scale vs. Performance: The largest model, Llama 3.1 70B, ranked near the bottom (9th out of 10), proving that larger models do not necessarily make better multilingual teachers.
• Family Matching: Matching the teacher and student model families (e.g., Gemma teacher to Gemma student) yielded significantly higher PG-SCOREs (at least +20.5% improvement) compared to mismatched pairs, likely due to shared tokenization schemes and pretraining distributions.

B. Drivers of Quality

• Intrinsic Predictors: Principal Component Analysis (PCA) revealed that prompt diversity, prompt length, response fluency, and response distinctiveness captured 93.3% of the variance in data quality.
• Predictive Power: These intrinsic metrics could predict extrinsic student performance with an $R^2$ of 0.664, suggesting practitioners can screen teachers using cheaper intrinsic metrics rather than full student training runs.
• Language Dependence:
  • High-Resource Languages (e.g., German): The Generate method (few-shot) performed best.
  • Less-Resourced Languages (e.g., Arabic, Indonesian): The Respond or Translate methods outperformed the Generate method, likely because seed data quality is lower in these languages.

C. Case Study (Tagalog)

A case study on Tagalog (a held-out language) validated the proposed "recipe":

• Using a high PG-SCORE teacher (Gemma 3 27B) and family-matching (Gemma 3 4B student) resulted in a model competitive with larger, non-synthetic baselines on the FILBENCH benchmark.
• This confirmed that the evaluation protocol generalizes to unseen languages.

5. Significance and Implications

• Paradigm Shift: The paper challenges the "bigger is better" heuristic in multilingual synthetic data generation, advocating for data-centric selection criteria based on actual generation quality in the target language.
• Cost Efficiency: By showing that intrinsic metrics predict downstream performance, the work offers a cost-effective way to select teachers without the computational expense of training multiple student models.
• Equity in AI: The findings highlight a structural gap where languages with low representation in pretraining data (CommonCrawl) suffer from lower synthetic data quality. The proposed "recipe" offers a pathway to improve models for less-resourced languages through careful teacher selection and family matching.
• Resource Release: The authors released the POLYGLOT Collection (synthetic datasets and student models), code, and evaluation tools to facilitate future research.

In conclusion, this work provides a rigorous, data-driven framework for selecting multilingual teachers, demonstrating that Gemma 3 27B and Aya Expanse 32B are currently superior choices for synthetic data generation, and that family alignment is a critical heuristic for maximizing student performance.