Technical Summary: Better as Generators Than Classifiers

Problem Statement

Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, yet their utility in low-resource scenarios remains debated. While LLMs are frequently employed directly as classifiers via zero-shot or few-shot prompting, existing literature suggests that in data-scarce settings (e.g., 4–64 samples), prompting often outperforms fine-tuning smaller models. Conversely, with sufficient labeled data (e.g., ~100 samples), fine-tuned smaller models typically match or exceed the performance of massive LLMs.

However, the behavior of these models in low-resource languages and niche tasks is not well understood, as most studies focus on high-resource languages like English. Furthermore, the direct use of massive LLMs as classifiers incurs significant computational costs. This work addresses the question: Are large language models better utilized as generators of synthetic data to train smaller, efficient models, rather than as direct classifiers in low-resource multilingual settings?

Methodology: Data-Driven Distillation

The authors propose a "data-driven distillation" framework where a large multilingual LLM acts as a teacher to generate synthetic samples, which are then used to train or improve smaller models. The methodology consists of two primary phases:

1. Synthetic Data Generation

• Generator Model: The state-of-the-art LLaMA-3 70B (instruction-tuned) is used to generate synthetic data.
• Generation Strategy: To ensure quality, the generator is provided with 10 representative human-labeled examples per label and language as few-shot prompts. This approach, identified in prior work (Anikina et al., 2025; Cegin et al., 2024b), significantly improves generation quality.
• Filtering: A self-revision mechanism is employed where the generator acts as a judge to filter out low-quality samples or those containing incorrect languages.
• Volume: The process generates up to 200 synthetic samples per label.

2. Training Smaller Models

The generated synthetic data is utilized in three distinct ways to improve smaller models:

1. Fine-tuning: Training a multilingual encoder (XLM-RoBERTa Large) on the synthetic data.
2. In-Context Learning (ICL): Using synthetic samples as few-shot examples for smaller LLMs (LLaMA-3.1-8B, Gemma-3-4B, Qwen-2.5-7B) during inference.
3. Instruction-Tuning: Fine-tuning the smaller LLMs using the synthetic data (via LoRA) and then using them with ICL.

These approaches are systematically compared against the large generator model (used in zero-shot mode) and, in specific sections, against models trained on human-labeled data.

Experimental Setup

• Languages: 11 typologically diverse languages categorized by resource availability:
  • Low-resource: Azerbaijani, Romanian, Slovenian, Telugu, Welsh.
  • Medium-resource: Hebrew, Indonesian, Swahili, Thai.
  • High-resource: English, German.
• Tasks: Four classification tasks:
  • Sentiment Classification (Binary).
  • Topic Classification (7 classes, from SIB-200).
  • Intent Recognition (10 classes, from MASSIVE).
  • Sarcasm Detection (Binary, English only, from SemEval 2021).
• Robustness: To account for randomness in subsampling and model initialization, all experiments were run 20 times, reporting mean accuracy and standard deviation.

Key Results

1. Generators Outperform Classifiers in Low-Resource Settings

The central finding is that smaller models trained on synthetic data consistently outperform the large generator model itself, particularly in low-resource languages.

• Sample Efficiency: Smaller models require as few as 50 synthetic samples to surpass the performance of the 70B generator.
• Performance Gains:
  • Low-resource languages: Fine-tuning yielded a 13% average improvement, ICL 15%, and instruction-tuning 18% over the generator.
  • High-resource languages: Gains were smaller (5–11%), likely due to the over-representation of these languages in the pre-training data of the large models.
• Task Dependency: The benefits are most pronounced for tasks with lower representation in pre-training data (e.g., Intent Recognition, Topic Classification). For highly represented tasks like Sentiment Classification, synthetic samples often failed to outperform the generator unless instruction-tuning was used, which incurs high computational costs.

2. Comparison with Human-Labelled Data

• Resource-Constrained Settings: When sample sizes are small (up to ~100), synthetic data provides performance comparable to human-labeled data.
• Stagnation: As the number of samples increases, the performance of models trained on human data continues to improve, while performance on synthetic data stagnates.
• Cause: The authors attribute this stagnation to the lower diversity and informativeness of generated samples compared to human data.
• Variance: Training on synthetic data exhibits higher sensitivity to hyperparameters and randomness, leading to more "failed runs" (performance worse than random chance) compared to human-labeled data.

3. Computational Efficiency

• Using smaller models (XLM-RoBERTa or 4B–8B LLMs) is significantly more computationally efficient than running the 70B generator for inference.
• While instruction-tuning is the most expensive training method, the resulting smaller models offer a scalable path for deployment in resource-constrained environments.

Significance and Claims

The paper claims that the optimal utilization of Large Language Models in low-resource multilingual contexts is not as direct classifiers, but as generators of synthetic data.

• Paradigm Shift: The authors argue that leveraging LLMs to "distill" their knowledge into smaller, efficient models via synthetic data generation is a more effective strategy than relying on the LLMs for inference.
• Practical Impact: This approach enables the creation of high-performing, low-cost classifiers for low-resource languages and niche tasks where human labeling is scarce or expensive.
• Limitations: The authors modestly note that while synthetic data is effective for prototyping and low-resource scenarios, it does not yet fully replace human data for large-scale training due to diversity issues. They suggest that future work should focus on generation strategies that enhance sample diversity.

In conclusion, the study demonstrates that a "generate-then-train" pipeline allows smaller models to achieve performance on par with or superior to massive LLMs, offering a computationally efficient solution for multilingual NLP in resource-constrained settings.