\textit{Versteasch du mi?} Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Language

1. Problem Statement

The paper addresses the systemic marginalization of non-standard language varieties (dialects, sociolects, and under-resourced languages) by Large Language Models (LLMs) and Generative AI (GenAI). While LLMs have advanced rapidly, they are fundamentally biased toward standardized, high-resource languages (e.g., Standard English, Standard German). This creates a "digital language divide" where:

• Epistemological Bias: LLMs reinforce the ideology of language as a "monolithic, monolingual, syntactically standardized system," treating variation as noise rather than a constitutive feature of communication.
• Technical Exclusion: Non-standard varieties lack the training data, orthographic standardization, and institutional support required for effective AI processing.
• Algorithmic Discrimination: Users of non-standard varieties face a "tokenization tax," where their inputs require more tokens to process, leading to higher costs, degraded performance, and shorter context windows compared to standard language speakers.
• Evaluation Failure: Current benchmarks (e.g., MMLU) are culturally and linguistically biased toward Western, standardized norms, failing to measure true capability in dialects or under-resourced languages.

2. Methodology

The authors employ an interdisciplinary approach, synthesizing Critical Sociolinguistics and Computational Linguistics.

• Theoretical Framework: The paper critiques the historical roots of linguistic standardization (linked to colonialism and nationalism) and analyzes how these ideologies are encoded into modern AI architectures (tokenization, training data selection, and benchmarking).
• Case Studies: Two specific linguistic complexes are analyzed to illustrate the spectrum of marginalization:
  1. South Tyrolean German: A dialect continuum widely used in informal digital communication in Italy but lacking a specific ISO-639 code and formal standardization.
  2. Kurdish Varieties: A dialect continuum (including Kurmanji, Sorani, Southern Kurdish, Hawrami, Zazaki) characterized by political suppression, orthographic diversity (Latin, Arabic, Cyrillic scripts), and varying degrees of institutional support.
• Technical Analysis: The authors examine specific NLP components:
  • Tokenization: Analyzing how Byte Pair Encoding (BPE) fragments non-standard words, increasing token counts and computational costs.
  • Resource Auditing: Reviewing the availability of ISO codes, training corpora (OPUS, VLO), pre-trained models, and machine translation services.
  • Benchmark Review: Evaluating the inclusion (or exclusion) of these varieties in major benchmarks like FLORES-200, SIB-200, Global MMLU, and DialectBench.

3. Key Contributions

• Identification of the "Tokenization Tax": The paper quantifies how BPE tokenization disproportionately penalizes non-standard languages. Because dialects often lack subword entries in the model's vocabulary, they are broken into smaller, meaningless fragments (e.g., individual characters), increasing the token count by 2x to 15x compared to English. This results in higher financial costs for users and reduced context window efficiency.
• Critique of "Scriptism": The authors argue that the AI field's reliance on written text ("scriptism") inherently excludes oral traditions and non-standardized dialects, as LLMs prioritize standardized orthographies found in digitized books and academic papers.
• Resource Gap Mapping:
  • South Tyrolean: Identified as having zero specific ISO code, making it invisible to major NLP resources. It is often subsumed under broader "Bavarian" codes, masking its specific resource needs.
  • Kurdish: Demonstrated a stark hierarchy where Central (Sorani) and Northern (Kurmanji) Kurdish have some resources, while Southern Kurdish, Hawrami, and Zazaki are virtually absent from all datasets, models, and benchmarks.
• Policy Framework Proposal: The paper proposes a shift from "extractive" data practices to "democratic and decolonial digital strategies." This includes:
  • Dialect Gap Reporting: Mandating Big Tech to report performance disparities between standard and non-standard varieties.
  • Data Sovereignty: Ensuring communities control their own data (e.g., via Community Science initiatives like Mozilla Common Voice).
  • Culturally Grounded Benchmarks: Moving away from translated benchmarks (which create a "translation pivot trap") toward native, culturally specific evaluations (e.g., IRLBench for Irish, DialectBench).

4. Results

• Technical Performance: LLMs struggle significantly with non-standard varieties due to morphological complexity and orthographic variation. For example, a single Kurdish sentence may be compressed into one complex word, but the tokenizer splits it into multiple meaningless tokens, degrading reasoning capabilities.
• Benchmark Exclusion:
  • South Tyrolean is absent from all major multilingual benchmarks (FLORES, SIB-200, Global MMLU).
  • Kurdish Varieties: Only Central and Northern Kurdish appear in translation benchmarks (FLORES-200, NTREX-128). Southern Kurdish, Laki, Zazaki, and Hawrami are completely absent from all surveyed benchmarks.
• Institutional Support: Institutional support for Kurdish is highly uneven. Central and Northern Kurdish have full support in education, media, and official domains, whereas varieties like Hawrami and Laki have little to no institutional backing, directly correlating with their lack of digital resources.
• The Vicious Cycle: A feedback loop exists where historical suppression leads to low digital footprints, which leads to exclusion from web-scraped training data, which further entrenches technological marginalization.

5. Significance

• Political Implications: The paper argues that the ability of LLMs to handle non-standard language is not merely a technical issue but a political one. It impacts digital sovereignty, social justice, and the preservation of cultural identity.
• Economic Impact: The "tokenization tax" creates an economic barrier for speakers of non-standard languages, effectively pricing them out of full participation in the AI-driven digital economy.
• Future Directions: The authors call for a reorientation of AI development toward Linguistic Agency. This requires:
  • Big Tech Accountability: Using Corporate Social Responsibility (CSR) credits to incentivize dialect support.
  • State Intervention: Utilizing public supercomputing (e.g., EuroHPC) to subsidize training on under-resourced dialects.
  • Community Stewardship: Empowering civil society organizations to act as data stewards and algorithmic auditors to prevent "hallucinated standardization."

In conclusion, the paper posits that without a fundamental shift in how data is collected, tokenized, and evaluated, GenAI will continue to exacerbate linguistic hierarchies, threatening the vitality of the world's linguistic diversity in digital spaces.