💬 NLP

P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs

The paper introduces P3B3, an expert-curated benchmark and evaluation framework that reveals a significant bias toward Brazilian Portuguese over European Portuguese in current Large Language Models, highlighting the urgent need for more balanced multilingual representation.

Original authors: Rafael Ferreira, Inês Vieira, Inês Calvo, James Furtado, Iago Paulo, Diogo Tavares, Diogo Glória-Silva, David Semedo, João Magalhães

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rafael Ferreira, Inês Vieira, Inês Calvo, James Furtado, Iago Paulo, Diogo Tavares, Diogo Glória-Silva, David Semedo, João Magalhães

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a group of very smart, well-read robots (Large Language Models, or LLMs) that speak Portuguese. The problem is that Portuguese isn't just one single voice; it's like a family with two very distinct cousins: European Portuguese (spoken in Portugal) and Brazilian Portuguese (spoken in Brazil).

While they share the same family name, they speak differently. They use different words for the same things (like "bus" vs. "autocarro"), talk to each other differently, and even arrange their sentences in unique ways.

The Problem: The "Brazilian Bias"

The researchers in this paper noticed something odd. Because the internet is flooded with text from Brazil, these robots have read way more Brazilian Portuguese than European Portuguese. It's like if a student only read textbooks written in one dialect; when they try to speak, they accidentally sound like they're from that region, even if they are talking to someone from the other side of the ocean.

The team wanted to know: Do these robots have a favorite cousin? And if we tell them, "Please speak like a European," can they actually do it?

The Solution: P3B3 (The "Accent Test")

To find out, the researchers created a special test called P3B3. Think of this as a "blind taste test" for language.

The Setup: They didn't just ask the robots, "What is a bus?" (which might trigger a specific answer). Instead, they created natural, multi-turn conversations about everyday topics like transportation and beauty products.
The Trick: The questions were designed to be "variety-agnostic." This means the questions didn't give any hints about which accent to use. It was like asking, "How do I get to the store?" without saying "in Lisbon" or "in São Paulo."
The Goal: They wanted to see what the robots would naturally say. Would they default to the Brazilian accent because that's what they've read the most? Or could they switch gears if asked?

How They Measured It

The researchers used two methods to grade the robots' answers:

The "Grammar Police" (Classifiers): Computer programs trained to spot specific words and grammar rules that belong to either Portugal or Brazil.
The "Expert Judge" (LLM-as-Judge): They used a super-smart AI (Gemini) to read the answers and act like a human linguist, giving a score from 0 (Pure Brazilian) to 10 (Pure European) and explaining why (e.g., "They used the word 'ônibus' instead of 'autocarro'").

What They Found

The results were a bit like a reality TV show where the contestants have a hard time hiding their true origins:

The Default Mode: When left alone (no instructions), almost every robot naturally spoke in Brazilian Portuguese. It was their "comfort zone." Even robots that were specifically trained to be European (like AMALIA) were the only ones that consistently stuck to the European accent.
The "Switch" Test: When the researchers explicitly told the robots, "Please speak Brazilian," most of them did a great job. But when they said, "Please speak European," it was much harder. Many robots struggled to stay consistent. They would start speaking European but then accidentally slip back into Brazilian words or grammar after a few sentences.
Size Matters: The newer, bigger, and more powerful robots were better at following instructions and switching accents than the older, smaller ones. However, even the best ones sometimes drifted back to their Brazilian roots during long conversations.

The Big Picture

The paper concludes that while these AI models are getting better, they still have a strong "Brazilian bias" because of the data they were trained on. They are like actors who are great at playing a Brazilian character but struggle to stay in character as a European one for a whole play, even when the director asks them to.

The researchers built this test (P3B3) so that in the future, developers can check if their robots are becoming more balanced and fair to all Portuguese speakers, ensuring that a user in Lisbon gets the same high-quality, culturally accurate experience as a user in Rio de Janeiro.

Technical Summary: P3B3 – A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs

Problem Statement

As Large Language Models (LLMs) become integral to everyday communication, their ability to capture regional linguistic variation is critical for equitable language use and cultural preservation. While Portuguese is a pluricentric language with over 250 million speakers, the divergence between European Portuguese (pt-PT) and Brazilian Portuguese (pt-BR) is substantial, encompassing lexical choices (e.g., autocarro vs. ônibus), forms of address (tu vs. você), and syntax (e.g., clitic placement).

Despite these differences, most training and evaluation frameworks treat Portuguese as a uniform language. Given the overwhelming prevalence of Brazilian data in large-scale datasets, models often exhibit a "silent bias," defaulting to pt-BR even when the context suggests pt-PT. This behavior risks degrading user experience and contributing to linguistic leveling where regional variations are underrepresented. The paper addresses two primary questions: (i) to what extent do current LLMs exhibit a preference for one Portuguese variant, and (ii) can models be effectively steered toward a target variant.

Methodology

The P3B3 Benchmark

The authors introduce P3B3, an expert-curated benchmark consisting of 74 multi-turn dialogues (203 turns total) designed to elicit variety-specific linguistic markers without explicit priming. The benchmark adheres to three core guidelines:

Variety-Agnostic: Prompts contain no explicit references to specific variants to prevent priming effects.
Variety Sensitivity: Prompts are designed to naturally elicit lexical and grammatical differences inherent to pt-PT and pt-BR.
Conversational: Multi-turn structures accumulate dialogue history to observe how biases evolve over context, rather than relying on isolated single-turn inputs.

The dialogues cover diverse everyday domains (e.g., transportation, beauty products) and were manually constructed by two linguistics experts to ensure validity and subtlety in variation cues.

Experimental Setup

The study evaluates a diverse range of multilingual instruction-tuned LLMs (including open-weight models like Llama, Gemma, Qwen, and specialized models like AMALIA and Sabiá) under three prompting settings:

No-Prompt: No explicit variety specification, measuring inherent bias.
pt-BR Prompt: Explicit instruction to respond in Brazilian Portuguese.
pt-PT Prompt: Explicit instruction to respond in European Portuguese.

Evaluation Framework

To assess variant alignment and controllability, the authors employ a dual-method evaluation strategy:

Encoder Classifiers: Using fine-tuned BERT-based models (PeroVaz and PtBrVId) to score outputs on a probability scale.
LLM-as-Judge: Utilizing Gemini-3-Flash to assign a score from 0 (pt-BR) to 10 (pt-PT) with chain-of-thought reasoning. This method was validated against human annotations by linguistic experts, showing strong agreement (Weighted $\kappa$ of 0.81 for the best configuration).

Key Results

Inherent Bias (No-Prompt Setting)

Most evaluated models exhibit a strong inherent bias toward pt-BR, likely reflecting training data imbalances.

AMALIA-9B, a model specifically pre-trained and fine-tuned for pt-PT, is the only model consistently biased toward pt-PT (score: 91.2).
Gervásio, a LLaMA-based model fine-tuned for pt-PT, still exhibits a pt-BR bias, suggesting that pre-training data and scale play a critical role in capturing variety-specific behavior.
Larger and more recent models (e.g., Qwen-3.5 27B, Gemma-4-31B, Gemini-3-Flash) show more balanced behavior, with scores ranging from 44.5 to 63.7, though still leaning toward pt-BR.

Controllability

pt-BR Steering: When explicitly prompted for pt-BR, most models maintain or increase their use of the variant, indicating strong instruction-following capabilities, particularly in larger models.
pt-PT Steering: Results are more variable. While models like EuroLLM, Apertus-70B, and AMALIA show strong alignment (≥86), LLaMA-based models struggle to produce consistent pt-PT outputs. Notably, Qwen-3.5 shows a substantial improvement over Qwen-3 (32.7 to 86.1), suggesting training changes significantly improved pt-PT handling.
Turn-Level Analysis: In the pt-PT setting, models with lower initial alignment tend to drift toward pt-BR over subsequent turns. This indicates that a single initial instruction is often insufficient to sustain language variety adherence across multiple turns, suggesting controllability degrades in longer contexts.

Evaluation Metrics

The study validates that while classifiers provide a low-cost complementary signal, LLM-as-Judge methods yield more aligned and interpretable results, performing best when using category-based prompts and full dialogue history.

Significance and Contributions

The paper claims three primary contributions:

P3B3 Benchmark: A novel, expert-curated, variety-agnostic multi-turn dialogue benchmark designed to systematically evaluate Portuguese variety bias and controllability.
Evaluation Framework: A human-validated methodology combining classifiers and LLM-as-Judge to assess variant alignment, demonstrating that LLM judges can reliably measure subtle linguistic differences.
Empirical Evidence: Extensive evaluation revealing a pervasive bias toward pt-BR across diverse models and highlighting the challenges in steering generation toward pt-PT, particularly in multi-turn contexts.

The authors conclude that while larger and newer models show improved controllability, pt-PT performance remains less consistent. They emphasize the need for more balanced multilingual representation in training data and evaluation frameworks to support the development of equitable multilingual systems. The work also notes limitations, such as the focus on everyday domains rather than technical ones, and suggests future work extending the benchmark to other Portuguese varieties (e.g., Angola, Mozambique, Cape Verde).