Goldfish: Monolingual Language Models for 350 Languages

The paper introduces Goldfish, a suite of over 1,000 small monolingual language models for 350 languages that outperform large multilingual models in grammaticality and perplexity, particularly for low-resource languages where existing models often struggle.

The Gist

Imagine the world of artificial intelligence as a giant library. For a long time, the librarians (AI researchers) have been building one massive, super-detailed encyclopedia that tries to cover every language on Earth at once. These are the "multilingual models" like BLOOM or XGLM. They are like a single, enormous chef trying to cook a perfect meal for 7,000 different cultures simultaneously.

The problem? When this giant chef tries to cook a meal for a small, remote village (a low-resource language), the food often comes out burnt or bland. The chef is so busy juggling the huge, popular cuisines (like English, Spanish, or Chinese) that they don't have enough time, ingredients, or attention to get the small village dishes right.

Enter "Goldfish."

This paper introduces a new project called Goldfish. Instead of one giant chef, the researchers decided to hire thousands of tiny, specialized chefs. Each Goldfish model is a small, single-language expert dedicated to just one language.

Here is the breakdown of what they did and why it matters, using some simple analogies:

1. The "Big Fish" vs. The "Goldfish"

• The Big Fish (Multilingual Models): These are the massive AI models everyone talks about. They are huge (billions of parameters) and trained on terabytes of data. But because they are so big, they often treat low-resource languages as an afterthought. It's like a giant ocean liner trying to dock at a tiny, rocky pier; it just doesn't fit well, and the passengers (the AI's predictions) get seasick.
• The Goldfish (Monolingual Models): The researchers created over 1,000 tiny models. Each one is small (only 125 million parameters) and focuses on a single language. They are like a fleet of small, agile boats. Even though each boat is small, they are perfectly designed for their specific harbor.

2. The "Memory" Problem

The paper makes a funny joke about the name "Goldfish." Real goldfish are famous for having short memories. The researchers say their models are like goldfish: they are small, there are many of them, and they don't have "memories" of the entire world's data.

But here's the twist: For low-resource languages, having a short memory is actually a superpower.

• The giant multilingual models try to remember everything, which confuses them when they try to speak a rare language.
• The Goldfish models only remember the specific language they were trained on. Because they aren't distracted by 500 other languages, they actually speak the target language much more clearly and grammatically.

3. The "Recipe" (Data)

Imagine you want to learn to cook a specific dish, say "Quechua Stew."

• The Old Way: You give a master chef a giant bag of ingredients containing 10,000 pounds of beef, 5,000 pounds of potatoes, and only 10 grams of the special Quechua spices. The chef makes a stew, but it tastes mostly like beef.
• The Goldfish Way: You give a small, dedicated cook a small bag containing only the perfect amount of Quechua spices and ingredients. Even though the bag is small, the cook makes a perfect stew because they aren't distracted by the other 9,990 pounds of beef.

The researchers found that for many languages, these small, dedicated models actually performed better than the giant ones, even though the giant ones had way more total data. In fact, for some languages, the Goldfish models were so good they beat the giant models, and sometimes even beat simple "Bigram" models (which are like a toddler guessing the next word based only on the word immediately before it).

4. The "Grammar" vs. "Reasoning" Test

The researchers tested the Goldfish in two ways:

1. Grammar (Can it speak correctly?): The Goldfish passed with flying colors. They generated text that sounded natural and grammatically correct for their specific languages.
2. Reasoning (Can it solve logic puzzles?): The Goldfish struggled here. They were about as smart as a random guess.

The Analogy: Think of the Goldfish as a local tour guide.

• If you ask the guide, "What is the best way to say 'hello' in this village?" or "How do I order coffee here?", they will give you a perfect, grammatically correct answer.
• But if you ask the guide, "Explain the theory of relativity" or "Solve this complex math problem," they will be lost. They know their language perfectly, but they haven't been trained to be a genius philosopher.

Why This Matters

For a long time, if you spoke a language with few speakers (like a language with only 100,000 speakers), you were ignored by AI. The big models didn't care enough about you.

Goldfish changes the game by saying: "We don't need a giant brain to speak your language well. We just need a small, focused one."

They released these models for 350 different languages, many of which had never had a dedicated AI model before. This is like handing a dictionary and a grammar book to 350 different communities that were previously left out of the digital conversation.

The Bottom Line

The paper proves that bigger isn't always better. For the world's smaller languages, a small, specialized, monolingual "Goldfish" model is often a much better speaker than a giant, distracted multilingual "Whale." It's a step toward making AI fairer and more inclusive for everyone, not just the speakers of the world's biggest languages.

Technical Summary

Here is a detailed technical summary of the paper "Goldfish: Monolingual Language Models for 350 Languages."

1. Problem Statement

The paper addresses a critical gap in Natural Language Processing (NLP) for low-resource languages:

• Reliance on Multilingual Models: Currently, most low-resource language modeling relies on massive multilingual models (e.g., BLOOM, XGLM, MaLA-500) trained on hundreds of languages simultaneously.
• Performance Deficits: Despite their scale, these models struggle with basic grammatical text generation (next-token prediction) in low-resource languages. The authors find that for many languages, these large multilingual models perform worse than simple bigram models.
• Data Imbalance: Multilingual datasets are heavily skewed toward high-resource languages (e.g., BLOOM uses 0.07MB of Akan data out of 1.61TB total), leading to severe under-performance for minority languages.
• Lack of Baselines: There is a scarcity of dedicated, comparable monolingual models for low-resource languages, hindering research and exacerbating inequities in NLP.

2. Methodology

The authors introduce Goldfish, a suite of over 1,000 small, monolingual language models trained specifically for 350 diverse languages.

Data Collection and Processing

• Sources: The training data merges three massive multilingual text datasets: Chang et al. (2024a), Glot500, and MADLAD-400.
• Byte Premium Scaling: To ensure fair comparison across languages with different character encodings, the authors apply byte premium scaling. They estimate the number of UTF-8 bytes required to encode content-matched text in a specific language relative to English. Dataset sizes are scaled based on this metric to represent "equivalent" English text bytes.
• Dataset Sizes: Models are trained on four specific data scales (where available): 5MB, 10MB, 100MB, and 1GB. If a 1GB dataset is unavailable, a "full" model is trained on all available data.
• Contamination Control: The authors explicitly exclude FLORES-200 and AmericasNLI from training data to prevent evaluation contamination. A contamination analysis confirms that <10 FLORES sequences appear in the training data for 98% of languages.

Model Architecture and Training

• Architecture: Autoregressive GPT-2 style Transformers trained from scratch.
  • Small Models (5MB/10MB): 39M parameters (4 layers, 512 hidden size).
  • Large Models (100MB/1GB/Full): 125M parameters (12 layers, 768 hidden size), equivalent to GPT-1/BERT-base.
• Tokenization: Custom monolingual Unigram tokenizers (SentencePiece) trained on the same dataset size as the model, with a vocabulary size of 50K.
• Training Regime: Models are trained for 10 epochs. Batch sizes vary by dataset size (4 for 5MB up to 64 for 1GB).
• Compute Efficiency: The total training compute is $1.65 \times 10^{20}$ FLOPs, which is less than 1/1900th of the compute used for the original GPT-3.

3. Key Contributions

1. Goldfish Suite: The release of 1,154 monolingual models covering 350 languages. For 215 of these languages, Goldfish represents the first publicly available monolingual text-generation model.
2. Demonstration of Inefficiency in Large Models: The paper provides empirical evidence that large multilingual models often fail at basic next-token prediction in low-resource settings, performing worse than simple statistical baselines (bigrams).
3. Scalable Low-Resource Baselines: The authors demonstrate that small, monolingual models (125M parameters) trained on limited data (1GB or less) significantly outperform massive multilingual models in terms of perplexity and grammaticality for low-resource languages.
4. Open Resources: Full release of models, training data (via Hugging Face), and code to facilitate future research.

4. Results and Evaluation

Perplexity (FLORES-200)

• Comparison: Goldfish models were compared against XGLM (4.5B/7.5B), BLOOM (7.1B), and MaLA-500 (10B).
• Performance: Goldfish achieved lower perplexity than all four comparison models on 98 out of 204 shared FLORES languages.
• Win Rates: Goldfish outperformed MaLA-500 on 63% of languages and XGLM 4.5B on 59% of languages.
• Bigram Baseline: Surprisingly, simple bigram models outperformed large multilingual models (BLOOM 7.1B) on 43% of languages, highlighting the severity of the multilingual models' failure in these domains.

Grammaticality (MultiBLiMP)

• Evaluated on a massively multilingual grammaticality benchmark covering 74 Goldfish languages.
• Goldfish models achieved higher average accuracy than any tested multilingual model (BLOOM 560M, XGLM, Gemma 3, Llama 3.2).
• Goldfish held the highest accuracy for 25 out of 74 languages.

Reasoning Benchmarks

• Evaluated on Belebele, XCOPA, and XStoryCloze.
• Result: Goldfish models performed at chance levels (similar to other small multilingual models).
• Implication: While small monolingual models excel at capturing linguistic structure and grammar, they lack the capacity for complex reasoning tasks at this scale. This suggests that reasoning capabilities may require larger-scale multilingual pretraining or different architectural approaches.

5. Significance and Discussion

• Paradigm Shift: The paper challenges the assumption that "bigger is always better." For low-resource languages, small, dedicated monolingual models are more effective for basic text generation than massive multilingual models.
• Accessibility: Goldfish models are computationally accessible to labs with limited budgets, democratizing NLP research for low-resource communities.
• Future Directions: The results suggest a dichotomy in language modeling:
  • Grammar/Structure: Best served by small, monolingual models trained on high-quality, language-specific data.
  • Reasoning/Abstraction: May still require large-scale multilingual pretraining to learn language-agnostic heuristics.
• Limitations: The models are limited by data availability (capped at 1GB) and do not yet support complex reasoning. The authors acknowledge that perfect comparability across diverse languages is difficult due to varying data cleanliness and domain coverage.

In conclusion, Goldfish establishes a new baseline for low-resource language modeling, proving that targeted, small-scale monolingual training is a superior strategy for achieving grammatical fluency in languages that are currently underserved by the state-of-the-art multilingual giants.