CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility

1. Problem Statement

Despite the proliferation of Large Language Model (LLM) leaderboards, significant gaps remain in the evaluation of European Portuguese (PT-PT):

• Lack of Dedicated Benchmarks: Existing leaderboards either focus on English (which is often saturated) or Brazilian Portuguese (PT-BR). There is no dedicated leaderboard for the European variant.
• Missing Evaluation Dimensions: Current benchmarks for Portuguese fail to address critical aspects of model performance, specifically cultural alignment (adherence to local norms and knowledge) and model safeguards (ability to refuse harmful requests).
• Evaluation Methodology Limitations: Many existing evaluations rely on logit-based ranking (selecting the most probable token), which is incompatible with closed-source models or API-based deployments. There is a need for a purely generative evaluation approach that mirrors real-world chatbot interactions.

2. Methodology

The authors developed CLARIN-PT-LDB, an open leaderboard hosted on HuggingFace Spaces. The system employs a purely generative evaluation approach, where models must generate text answers rather than selecting options based on internal logits. This ensures compatibility with both open and closed models.

The Benchmark Suite

The leaderboard comprises 10 benchmarks, categorized as follows:

A. Novel Contributions (Specific to PT-PT):

1. Tuguesice-PT (Cultural Alignment):
   • Creation: Manually created from scratch by 9 annotators.
   • Design: Contains 327 question-answer pairs implicitly testing knowledge of Portuguese culture (Society, History, Geography, Cuisine, etc.). Questions are phrased to assume local context (e.g., "What is the northernmost city in the country?" rather than "What is the capital of Portugal?").
   • Evaluation: Generative mode; correctness is determined by case-insensitive containment of the gold answer.
2. DoNotAnswer-PT (Safeguards):
   • Creation: A curated translation of the English Do-Not-Answer benchmark into European Portuguese.
   • Content: 939 requests across 61 harm categories (e.g., violence, racism, illegal acts).
   • Evaluation: Uses a Judge Model (Llama 3.3 70B Instruct) to classify model responses into categories like "refusal," "contest," "disclaimer," or "answer." Success is defined as the model refusing harmful requests.

B. Translated Benchmarks (Reasoning & NLU):
The remaining 8 benchmarks were translated from English to European Portuguese using the Google Cloud Advanced Translation API (specifically configured for PT-PT, avoiding PT-BR).

• MuSR (Chain-of-Thought Reasoning): 756 instances involving complex narratives (e.g., murder mysteries) requiring multi-step reasoning.
• AA-Omniscience-Public (NLU Q&A): 600 instances testing knowledge and hallucination propensity. Evaluated via a Judge Model (Gemini 2.5 Flash).
• GPQA Diamond (NLU Q&A): 198 graduate-level, "Google-proof" multiple-choice questions in Chemistry, Physics, and Biology.
• MMLU & MMLU Pro (NLU Q&A): 14,042 and 12,032 instances respectively, covering 57+ topics. MMLU Pro features harder questions with 10 options.
• ExtraGLUE (PT-PT): Includes CoPA (Reasoning, 500 instances), MRPC (Semantic Similarity, 1,730 instances), and RTE (Textual Entailment, 3,000 instances).

Technical Infrastructure

• Backend: Built on a fork of Eleuther's LM Evaluation Harness. It uses YAML configuration files to define prompts, output parsing (regex/pattern matching), and generation parameters (temperature, max tokens).
• Hardware: Powered by the PORTULAN CLARIN Research Infrastructure, utilizing two NVIDIA L40S nodes (10x 40GB GPUs each) and two NVIDIA RTX Pro 6000 nodes (1x 96GB GPU each).
• Frontend: A HuggingFace Space interface allowing users to view results, filter by model size/architecture, and submit open models for evaluation.

3. Key Contributions

1. First PT-PT Leaderboard: Fills the void of a dedicated evaluation platform for European Portuguese, distinguishing it from Brazilian Portuguese evaluations.
2. Cultural & Safety Benchmarks: Introduces the first metrics for cultural alignment (Tuguesice-PT) and safeguard adherence (DoNotAnswer-PT) in the Portuguese language context.
3. Generative-Only Evaluation: Establishes a standard for evaluating LLMs based on text generation rather than logit ranking, ensuring results are applicable to API-based and closed models.
4. Open Infrastructure: Provides a fully open, reproducible pipeline for benchmarking, supported by significant compute resources.

4. Results

The paper presents initial results for several models (Gervásio, Llama, and Mistral) across different parameter sizes (8B, 24B, 70B):

• Model Size Scaling: Performance generally increases with parameter count (e.g., Llama 70B outperforms Llama 8B across all metrics).
• Fine-Tuning Impact: Gervásio (a model fine-tuned on European Portuguese data) significantly outperforms its base model Llama on the Tuguesice-PT benchmark (39.76% vs. 25.69%), demonstrating the importance of cultural data for alignment.
• Benchmark Difficulty: The leaderboard successfully differentiates model capabilities, showing expected performance drops on harder benchmarks (e.g., MMLU Pro vs. MMLU) and on cultural alignment tasks compared to standard NLU tasks.

5. Significance

• Scientific Rigor: By addressing data contamination and saturation issues common in English benchmarks, this leaderboard offers a fresh, rigorous evaluation ground for non-English LLMs.
• Cultural Preservation: It highlights that "Portuguese" is not monolithic; models trained on Brazilian data may fail to align with European cultural nuances, necessitating specific evaluation tools.
• Safety & Ethics: By introducing safeguard testing, the paper pushes the community to consider safety and refusal mechanisms as core metrics for LLM deployment, not just accuracy.
• Community Resource: The open availability of the leaderboard and its codebase encourages reproducibility and rapid iteration in the European Portuguese NLP community.

The leaderboard is accessible at: https://huggingface.co/spaces/PORTULAN/portuguese-llm-leaderboard.