Technical Summary: Building Community-Centred NLP Resources for Puno Quechua

Problem Statement

The preservation and revitalization of under-resourced indigenous languages, specifically Puno Quechua (ISO 639-3: qxp), are hindered by a lack of digital tools tailored to their speakers. While approximately 465,000 people speak Puno Quechua in the Puno region of Peru, formal education is conducted almost exclusively in Spanish, resulting in low literacy rates in the native language. This literacy gap excludes speakers from text-based AI ecosystems.

Existing Automatic Speech Recognition (ASR) resources for the Quechua language family suffer from four primary deficiencies:

1. Data Scarcity and Aggregation: Corpora often aggregate distinct, non-mutually intelligible varieties under a single macrolanguage label (que), or fail to formally identify specific variants.
2. Morphological Complexity: The highly agglutinative nature of Quechua leads to poor word-level Word Error Rate (WER) metrics.
3. Low Written Literacy: Community-driven transcription is difficult due to low literacy rates in Quechua.
4. Lack of Benchmarks: No prior work has established a consistent, variety-specific evaluation benchmark for Puno Quechua.

Methodology

1. Participatory Data Collection

The authors constructed the largest speech corpus for a single Quechua variety through a four-phase participatory design campaign involving the National University of Altiplano Puno and the local community Illariy Ch'aska.

• Scope: The campaign ran from January to February 2026, engaging 151 contributors.
• Data Composition: The resulting dataset comprises 66 hours of audio, split into:
  • Scripted Speech (SCS-25): 30.5 hours of manually validated data.
  • Spontaneous Speech (SPS-3): 5.5 hours of manually validated data, supplemented by 30 hours of "silver" (automatically generated) transcriptions.
  • Out-of-Domain (OOD): ~16 minutes of radio and social media data, manually transcribed and validated.
• Governance: Data was released under a CC0-1.0 license via the Mozilla Data Collective.

2. Model Architectures and Training

The study evaluated and fine-tuned three foundation model architectures:

• Whisper-base (74M parameters): An encoder-decoder Transformer. Fine-tuned with transcript prefixes set to Spanish.
• wav2vec2-base (95M parameters): A self-supervised convolutional-Transformer pre-trained on Librispeech.
• XLS-R-300M (315M parameters): A multilingual wav2vec2 model pre-trained on 436k hours across 128 languages. This was selected as the strongest baseline for low-resource languages with complex phonology (e.g., ejectives, uvulars).

Training Strategies:

• Fine-tuning: Models were trained on two configurations:
  • V (Validated-only): 36 hours of manually transcribed data.
  • V+S (Validated + Silver): 66 hours, including the 30 hours of silver transcriptions.
• Continued Pre-Training (CPT): Specifically for XLS-R-300M, the model underwent CPT on 65 hours of unlabelled Puno Quechua audio prior to fine-tuning to adapt acoustic representations to the target language's phonology.
• Baselines: The study compared these results against state-of-the-art (SOTA) off-the-shelf models, including the omniASR family (CTC and LLM decoders up to 7.8B parameters) and MMS-1b-all (configured for Cuzco Quechua).

Key Results

Performance on Scripted Speech

• Continued Pre-Training (CPT) Impact: CPT yielded consistent gains on scripted speech. XLS-R-300M with CPT trained on validated data (V) achieved a 1.19% WER, a 42% relative improvement over the non-CPT version (2.06% WER).
• Best Overall Performance: The XLS-R-300M model with CPT, fine-tuned on the V+S dataset, achieved the lowest error rates on scripted speech: 2.11% WER and 0.30% CER.

Performance on Spontaneous Speech

• The Role of Silver Data: The inclusion of silver transcriptions (V+S) was the decisive factor for spontaneous speech performance. For XLS-R-300M with CPT, adding silver data reduced spontaneous WER from 13.6% to 3.15% (a 77% relative reduction).
• Model Comparison: While large off-the-shelf models like omniASR LLM_7B_v2 performed well (11.1% WER), the fine-tuned XLS-R-300M (V+S) significantly outperformed them on spontaneous speech (3.15% WER).

Out-of-Domain (OOD) Generalization

• Generalization Gap: All fine-tuned models exhibited a significant performance drop on OOD data compared to in-domain data.
• Trade-off: The omniASR LLM_7B_v2 (7.8B parameters) achieved the best OOD performance (23.7% WER), outperforming the best fine-tuned system (XLS-R + CPT, V+S: 27.4% WER). However, the fine-tuned XLS-R model requires only ~317M parameters and ~2GB VRAM, making it deployable on commodity hardware, whereas the omniASR model requires ~30GB VRAM.

Significance and Contributions

The paper claims three primary contributions:

1. Largest ASR Corpus: The creation of the largest ASR corpus for any single Quechua variety (66 hours), collected via a participatory design process that ensures community relevance and data sovereignty.
2. First Systematic Benchmark: The establishment of the first variety-specific ASR benchmark for Puno Quechua, evaluating SOTA models and fine-tuned foundation models across scripted, spontaneous, and OOD domains.
3. Open Release: The open-sourcing of all datasets, fine-tuned models (including CPT checkpoints), and code, enabling further research and application development.

The authors emphasize that while large-scale models offer better generalization, the proposed lightweight, community-centered approach provides a viable path for deploying ASR in low-resource settings where computational resources are limited. The study highlights that silver transcriptions are critical for overcoming data scarcity in spontaneous speech and that continued pre-training is essential for adapting models to the specific phonological features of Puno Quechua.