Decolonizing Linguistic Policies in Automated Speech Recognition: A Framework for Cross-Culturally Competent Speech AI

Technical Summary: Decolonizing Linguistic Policies in Automated Speech Recognition

Problem Statement
The paper argues that persistent failures in Automatic Speech Recognition (ASR) and ASR-mediated voice interfaces for low-resource, Indigenous, and non-standard language varieties are not merely technical bugs or data scarcity issues. Instead, these failures are the predictable outcomes of implicit linguistic policies embedded within the design, evaluation, and deployment of speech AI. These policies reproduce colonial language hierarchies by defining which voices are "machine-legible" and which are unintelligible. Current evaluation regimes, dominated by Word Error Rate (WER), fail to capture the structural nature of these failures, which range from denied access to public services and healthcare to the indignity of being rendered unintelligible. The paper posits that ASR systems operationalize linguistic capital (valuing prestige varieties over marginalized ones) and raciolinguistic ideologies (judging speech through perceived speaker identity) through three technical layers: data curation, metric selection, and model priors.

Methodology and Theoretical Framework
The paper employs a theoretical synthesis rather than a new experimental architecture. It integrates four intellectual traditions to analyze the political architecture of ASR:

1. Linguistic Capital Theory: Explains how institutional power assigns value to specific language varieties.
2. Raciolinguistic Ideology: Analyzes how social judgments about race and identity shape listening practices and system training.
3. Language Policy Research: Examines how institutions manage language use, adapted here to "computational linguistic policy."
4. Decolonial Computing: Critiques how technical infrastructure extends colonial relations and epistemologies.

Using this synthesis, the authors propose two primary analytical tools:

• The Seven-Layer Situatedness Model: A diagnostic map (Figure 1) that moves from the surface level of "Language" (supported labels) down through "Nation," "Region," "Ethno-linguistic Identity," and "Sociolinguistic Ideology," up to "Social Justice Correlations" and "Socio-technical Implications." This model illustrates how ASR systems often ignore situated speech practices (e.g., code-switching, tonal variations, regional dialects) in favor of standardized norms.
• The Three Harms (3M) Taxonomy: A classification of failure modes that extends beyond transcription accuracy:
  • Misrecognition: Incorrect transcripts, dropped words, or total refusal to recognize a language variety.
  • Misalignment: Plausible output that misinterprets culturally situated meaning (e.g., losing honorifics, misreading pragmatics, or failing to handle code-switching).
  • Mistrust: The relational harm where users disengage due to perceived unreliability, disrespect, or surveillance risks.

Key Contributions
The paper makes three integrated contributions:

1. Theoretical Account: It reframes ASR failures as structural linguistic policies enacted through data, metrics, and model priors. It demonstrates how technical decisions (e.g., transcription guidelines, benchmark composition, fallback behaviors) function as policy even without formal documentation.
2. The 3M Taxonomy and Audit Protocol: It introduces the Misrecognition, Misalignment, and Mistrust framework as an operational diagnostic. Crucially, it proposes a Minimum 3M Audit Protocol that includes:
   • Defining context and risk.
   • Constructing culturally grounded test sets (including idioms, honorifics, and code-switching).
   • Specifying evaluator roles (requiring fluent community evaluators).
   • Annotating failure modes using a shared codebook.
   • Adjudicating disputes via community evaluators rather than external majority votes.
   • It also advocates for metrics beyond WER, such as Tone Error Rate (TER) for tonal languages and click-specific error rates for languages like Xhosa and Zulu.
3. Participatory Framework for Culturally Competent ASR: The paper proposes a four-pillar framework (Figure 2) that reassigns design and evaluation authority to affected communities:
   • Participatory Auditing: Communities lead the identification of failures using the 3M taxonomy.
   • Community Co-Design: Communities define "correct" behavior, annotation guidelines, and data governance (including the right to withdraw data).
   • Equitable Deployment: Communities determine deployment conditions, fallback rules, and risk safeguards.
   • Feedback Integration: A loop where community-identified failures trigger repair obligations and system updates.

Results and Findings
As a conceptual and methodological paper, it does not report new experimental results or benchmark scores. Instead, its "results" are the identification of specific mechanisms through which harm occurs:

• Data Curation: Training data selection mirrors colonial hierarchies, treating "low-resource" languages as inherently resource-poor rather than politically marginalized.
• Metrics: WER is shown to be insufficient for tonal languages (where tone carries lexical meaning) and languages with non-standard orthographies, often collapsing distinct phonological features into single error scores.
• Model Priors: Language models penalize code-switching and non-standard sequences, effectively forcing speakers to adapt to prestige norms to be recognized.
• Case Evidence: The paper cites existing literature (e.g., Koenecke et al. on African American English, Mensah et al. on Akan ASR) to demonstrate that domain mismatch leads to misrecognition, misalignment, and mistrust, validating the need for the proposed framework.

Significance and Claims
The paper claims significance not by offering a new ASR architecture, but by providing a human-centered evaluation framework to identify culturally situated harms that WER-centered regimes under-specify.

• It argues that cultural competence cannot be reduced to aggregate accuracy or language-level coverage; it requires addressing pragmatic alignment and trust.
• It positions refusal and non-recognition as legitimate design outcomes. The authors argue that communities should have the agency to decide when their speech should not be machine-readable (e.g., for privacy, cultural protocol, or resistance to surveillance), framing "selective intelligibility" as a form of governance rather than a technical failure.
• The work serves as a call to shift ASR development from extractive data collection to participatory governance, where affected communities act as co-designers, evaluators, and governance partners to repair the "failed legitimacy relationship" between speech systems and marginalized speakers.

The paper concludes that for speech AI to fulfill its promise of cross-cultural connection, it must reckon with the colonial legacies that determine who gets to speak, who gets understood, and who builds the systems that listen.