La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America

The paper introduces La Leaderboard, the first open-source, community-driven initiative that evaluates 50 generative LLMs across 66 datasets covering Spanish varieties and the languages of Spain and Latin America to establish evaluation standards and promote linguistic diversity.

The Gist

Imagine you have a giant library of books written in many different languages. For a long time, the "smartest" books in the library were only written in English. If you wanted to know which book was the best, you'd ask a librarian who only spoke English. If you asked about a book in Spanish, Basque, or Catalan, the librarian might just guess, or worse, translate the English questions poorly, missing the local jokes, cultural references, and unique ways of speaking.

"La Leaderboard" is a new, community-built library guide designed to fix this. It's the first open-source "scoreboard" specifically for testing how well Artificial Intelligence (AI) understands and speaks the many varieties of Spanish and the other languages of Spain (like Basque, Catalan, and Galician).

Here is a breakdown of how it works, using some everyday analogies:

1. The Problem: The "One-Size-Fits-All" Trap

Think of current AI models like a universal translator that studied hard in English but only skimmed the other languages.

• The Issue: Most AI tests are like a driving test written in English and then poorly translated into Spanish. The questions might make sense grammatically, but they miss the local road signs, the slang, and the cultural context.
• The Result: An AI might get a perfect score on a translated test but fail miserably when a real person in Mexico City or Buenos Aires asks it a question about local laws or humor.

2. The Solution: A "Taste-Test" for AI

The creators of La Leaderboard decided to stop using translated tests. Instead, they organized a massive taste-test.

• The Ingredients: They gathered 66 different "recipes" (datasets). Some were donated by researchers, and some were cooked up specifically for this event.
• The Menu: The menu covers everything from medical advice (like a doctor diagnosing a patient) to legal questions (like a lawyer arguing a case), humor (can the AI get a joke?), and reading comprehension.
• The Languages: The test isn't just in "Spanish." It's in the specific dialects of Spain, Mexico, Argentina, Chile, and Uruguay, plus Basque, Catalan, and Galician. It's like testing a chef not just on "Italian food," but specifically on Neapolitan pizza vs. Roman pasta.

3. The Contestants: The AI Models

They invited 50 different AI models to take the test.

• The Big Giants: Some are the famous, heavy-hitters from big tech companies (like Meta's Llama or Google's Gemma). These are like professional athletes who have trained on every sport in the world.
• The Local Heroes: Others are smaller, specialized models built by European and Spanish researchers (like Salamandra or EuroLLM). These are like local champions who know the neighborhood streets better than anyone else.
• The Results: The scoreboard shows who wins. Surprisingly, the big giants often do well, but the local heroes sometimes beat them in specific areas, proving that you don't always need a giant engine to win a local race.

4. The "Eco-Friendly" Twist

Usually, testing AI is like burning a ton of coal to light a single candle. It takes massive amounts of electricity and time.

• The Innovation: The team decided to be smarter. Instead of asking the AI to read a long list of examples before answering (like a student cramming for a test), they often asked them to answer without any examples (zero-shot) or with very few.
• The Benefit: This saves a huge amount of energy (like turning off the lights when you leave the room) and makes it easier for smaller researchers to run their own tests without needing a supercomputer.

5. Why This Matters

Think of AI as a new employee joining a global company.

• If you only test them on English, they might seem smart, but they will fail when talking to the team in Madrid, Mexico City, or San Salvador.
• La Leaderboard ensures that the AI is culturally aware. It checks if the AI understands that a "joke" in Argentina might be different from a "joke" in Spain, or that a legal term in Mexico has a specific meaning.

The Bottom Line

La Leaderboard is a community-driven project that says: "We want AI that speaks our language, understands our culture, and respects our diversity."

It's not just about who is the "smartest" AI in the world; it's about who is the most helpful and respectful AI for the 600 million people who speak Spanish and the other languages of the Iberian Peninsula. By making this scoreboard open to everyone, they hope to inspire other communities (like those speaking French, Arabic, or Indigenous languages) to build their own scoreboards, ensuring no one is left behind in the AI revolution.

Technical Summary

Here is a detailed technical summary of the paper "La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America."

1. Problem Statement

The evaluation of Large Language Models (LLMs) has historically been dominated by English-centric benchmarks. While Spanish is a major global language with over 600 million speakers, existing multilingual leaderboards suffer from significant limitations:

• Lack of Cultural and Linguistic Nuance: Many benchmarks rely on machine-translated datasets from English, which fail to capture the idiomatic richness, cultural context, and specific dialectal variations of Spanish varieties (e.g., Spain, Mexico, Argentina) and co-official languages (Basque, Catalan, Galician).
• Representation Gap: No existing leaderboard comprehensively evaluates LLMs across the specific combination of languages spoken in Spain and Latin America simultaneously.
• Resource Inefficiency: Standard evaluation practices often use excessive few-shot examples, increasing computational costs and environmental impact without necessarily improving evaluation accuracy for all tasks.
• Reproducibility Issues: Many existing Spanish benchmarks are private or lack open-source evaluation pipelines, hindering community-driven progress.

2. Methodology

The authors introduce La Leaderboard, a community-driven, open-source initiative designed to establish a standardized evaluation framework for generative LLMs in Spanish and its related languages.

A. Data Collection and Composition

• Scope: The leaderboard aggregates 66 datasets covering four languages: Spanish (including varieties from Spain, Mexico, Argentina, Chile, and Uruguay), Catalan, Basque, and Galician.
• Sources: Datasets were donated by 13 research groups. Of these, 7 were specifically created for La Leaderboard (e.g., AQuAS, ClinTreatES, SpaLawEx), while others include established benchmarks like Belebele.
• Quality Control:
  • Origin: 55% of datasets are originally created in the target language.
  • Translation: 38% are human-translated by professionals to preserve cultural nuance. Only 7% are machine-translated, and only after rigorous review by native speakers.
  • Native Annotation: All datasets are annotated or revised by at least one native speaker.
• Task Diversity: The evaluation covers diverse domains (medical, legal, press) and task types:
  • MCQA (59%): Multiple-choice question answering.
  • Text Generation: Summarization, long-form QA, and counter-narrative generation.
  • Domains: Ethics (stereotype detection), reasoning, linguistic acceptability, and reading comprehension.

B. Evaluation Configuration and Efficiency

• Few-Shot Strategy: The authors deviate from standard literature by using fewer few-shot examples (0 to 3 shots) to reduce environmental impact and improve reproducibility.
  • Rationale: Base models are often evaluated fairly with fewer shots, while instruct models do not require excessive prompting.
  • Bias Mitigation: For MCQA tasks, the system ensures all correct options appear in the few-shot context (rotated) to prevent order bias.
  • Context Limits: A maximum of 2,048 tokens is enforced for few-shot examples to accommodate models with smaller context windows.
• Metrics:
  • MCQA: Evaluated using log-probabilities (LOGPROBS) of the correct option.
  • Generation: Evaluated using BLEU, ROUGE, Semantic Answer Similarity (SAS), and an automated LLM-based Judge.
• Normalization: Results are normalized against a random baseline to ensure fair comparison across tasks with different difficulty levels.

C. Infrastructure

• Backend: Built on a fork of the lm-evaluation-harness.
• Frontend: Hosted on Hugging Face Spaces using Gradio, providing transparency with open evaluation commands and normalization formulas.
• Hardware: Evaluations were conducted on NVIDIA H100 GPUs (MareNostrum 5), tracking energy consumption (kWh) and CO2 emissions.

3. Key Contributions

1. First Open-Source Multilingual Leaderboard for the Region: La Leaderboard is the first platform to evaluate generative LLMs across Spanish, Catalan, Basque, and Galician simultaneously, addressing a critical gap in NLP resources.
2. Community-Driven Methodology: The project establishes a scalable framework for dataset donation and evaluation, fostering collaboration among 13 research groups and the broader community.
3. Efficient Evaluation Protocol: The paper proposes a resource-efficient few-shot methodology that balances fairness, cognitive bias mitigation, and environmental sustainability.
4. Comprehensive Analysis of SOTA Models: The authors provide a detailed analysis of 50 open-weight models, offering insights into how different training strategies (pre-training vs. fine-tuning) affect performance in low-to-mid resource languages.

4. Evaluation Results and Analysis

The authors evaluated 50 models (ranging from 1B to 32B parameters, including quantized versions) across the 66 datasets.

• Top Performers:
  • Gemma-2-9B (base and instruct) and Meta-Llama-3.1-8B-IT consistently ranked in the top 10 across all languages.
  • Qwen2.5 (quantized 14B and 32B versions) showed strong performance, particularly in Spanish.
  • EuroLLM-9B and Salamandra-7B (European/Spanish specific models) demonstrated competitive results, especially in Basque and Galician, though they sometimes lagged behind in Spanish compared to models trained on massive English-heavy corpora.
• Performance by Language:
  • Spanish: Generally the highest performing language due to data availability.
  • Basque & Catalan: Models like Salamandra-7B and Latxa-7B excelled here, showing the value of targeted pre-training.
  • Galician: Consistently showed lower scores across most models, highlighting it as a critical area for future improvement.
• Performance by Task:
  • Strong: Natural Language Inference (NLI) and paraphrasing.
  • Weak: Summarization and mathematical reasoning (often due to strict exact-match metrics).
• Training Strategy Insights:
  • Massive Multilingual Pre-training (e.g., Llama-3, Qwen) offers the most consistent coverage across languages.
  • Continuous Pre-training (adding target language data to existing models) yields peak scores in specific languages (e.g., Latxa in Basque).
  • Fine-tuning only improves fluency but rarely boosts reasoning or QA capabilities significantly.
• Energy Efficiency:
  • There is a strong correlation (Pearson ~0.50) between model performance and energy consumption.
  • Instruction-tuned models generally consumed less energy than their base counterparts due to reduced verbosity.
  • Total energy for the evaluation was ~582 kWh (92 kg CO2).

5. Significance and Future Work

Significance:
La Leaderboard sets a new standard for multilingual LLM evaluation by prioritizing cultural awareness and linguistic diversity over simple translation. It provides a transparent, reproducible benchmark that encourages the development of models capable of serving the diverse Spanish-speaking world, rather than just English-centric adaptations.

Future Work:

• Indigenous Languages: Expanding the leaderboard to include indigenous languages of Latin America (e.g., Quechua, Guarani, Náhuatl) using newly donated datasets.
• Cultural Adequacy: Organizing a hackathon to create benchmarks specifically measuring cultural appropriateness in different Spanish-speaking countries.
• Model Expansion: Incorporating larger open models and proprietary models to cover the full spectrum of state-of-the-art capabilities.
• Contamination Analysis: Implementing rigorous checks to detect potential data contamination in training sets.

In conclusion, La Leaderboard is a pivotal step toward democratizing LLM evaluation, ensuring that AI systems are not only linguistically competent but also culturally relevant to the 600 million Spanish speakers worldwide.