Digital Linguistic Bias in Spanish: Evidence from Lexical Variation in LLMs

Technical Summary: Digital Linguistic Bias in Spanish

Title: Digital Linguistic Bias in Spanish: Evidence from Lexical Variation in LLMs
Author: Yoshifumi Kawasaki (University of Tokyo)

1. Problem Statement

Large Language Models (LLMs) are increasingly used in applications interacting with diverse linguistic populations. However, because these models are trained on massive web-scale datasets, they often exhibit Digital Linguistic Bias (DLB). In the context of Spanish—a language with immense geographic lexical variation across 21 countries and 8 major dialectal areas—there is a risk that LLMs may privilege hegemonic or standardized varieties (e.g., Peninsular Spanish) while failing to represent regional nuances. Previous research has suggested that model performance correlates with the volume of digital resources available for a specific region, but this study seeks to challenge and refine that assumption.

2. Methodology

The study treats LLMs as "virtual informants" to probe their dialectal knowledge through a structured, survey-style evaluation.

• Reference Dataset: The researchers utilized VARILEX, an expert-curated database of Spanish lexical variation. It contains 934 lexical items and 9,057 variants across 21 countries, using a binary scheme ("+" for predominant usage, "–" for marginal/absent).
• Evaluation Formats:
  • Yes–No Question Format (YNQF): Models are asked if a specific variant is used in a specific country (e.g., "Is 'coche' used in Spain to refer to a vehicle?").
  • Multiple-Choice Question Format (MCQF): Models are presented with a list of potential variants and asked to select all that are typically used for a concept in a given country.
• Models Evaluated: The study focused on the GPT family via official APIs, specifically gpt-4o, gpt-5.1, and gpt-5.2.
• Metrics:
  • F1 Score: Used for YNQF to measure alignment with gold annotations.
  • Adjusted Jaccard Coefficient ($J_{adj}$): Used for MCQF. This is a crucial technical choice; it is a chance-corrected measure that accounts for the probability of overlap occurring by random chance, making it robust across items with different numbers of variants.

3. Key Contributions

1. Large-Scale Empirical Evaluation: Provides one of the most fine-grained assessments of Spanish geographic lexical variation to date.
2. Novel Evaluation Framework: Bridges traditional dialectology (survey-style elicitation) with LLM benchmarking.
3. Refutation of the "Data Volume" Hypothesis: Challenges the notion that dialectal knowledge in LLMs is purely a function of the quantity of available digital text.

4. Results

• Uneven Representation: LLMs demonstrate systematic differences in accuracy. Varieties associated with Spain, Equatorial Guinea, Mexico & Central America, and the La Plata River are recognized with high accuracy.
• The "Chilean Gap": Chilean Spanish proved the most difficult for models to distinguish, performing significantly lower than other varieties in both YNQF and MCQF.
• Performance Trends: Newer models (gpt-5.1/5.2) showed incremental improvements in instruction-following and dialectal recognition compared to gpt-4o, though gpt-4o actually held a slight edge in overall YNQF F1 score.
• Decoupling Data Volume from Knowledge: Crucially, the study found no significant correlation between the volume of digital resources (measured in tokens) and model performance.
  • Example of Success: Equatorial Guinea (low data volume) performed well due to lexical affinity with Spain.
  • Example of Failure: Chile (high data volume) performed poorly, suggesting that web data may not capture the specific lexical distinctiveness of certain dialects or may be dominated by standardized forms.

5. Significance

The findings have profound implications for the development of linguistically fair AI. They suggest that simply increasing the quantity of data from a specific region is insufficient to mitigate Digital Linguistic Bias. Instead, developers may need to employ more targeted strategies, such as fine-tuning on high-quality, dialect-rich corpora or addressing the "standardization pressure" present in web-scraped data. This work highlights the necessity of moving beyond "data quantity" metrics toward a more nuanced understanding of how linguistic diversity is encoded in neural networks.