💬 NLP

DATASHI: A Parallel English-Tashlhiyt Corpus for Orthography Normalization and Low-Resource Language Processing

The paper introduces DATASHI, a new parallel English-Tashlhiyt corpus designed to address low-resource challenges in Amazigh language processing by enabling orthographic normalization studies and demonstrating that state-of-the-art large language models, particularly Gemini-2.5-Pro, can effectively handle the language's complex phonological features through few-shot prompting.

Original authors: Nasser-Eddine Monir, Zakaria Baou

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nasser-Eddine Monir, Zakaria Baou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of stories written in a beautiful, ancient language called Tashlhiyt (a type of Amazigh/Berber spoken in Morocco). But here's the problem: everyone writes it differently. Some people use the official alphabet, some use a Latin alphabet with weird shortcuts (like using the number "7" to sound like a specific throat sound), and others just mash words together or spell them however they feel like on social media.

It's like trying to read a recipe where one person writes "cup," another writes "c," and a third writes "1 cup of flour" but spells "flour" as "flor." Computers get totally confused by this mess. They can't translate, search, or understand the language because the "rules" are broken.

This paper introduces DATASHI, a new project designed to fix this mess and teach computers how to speak Tashlhiyt properly.

1. The Problem: The "Wild West" of Spelling

For a long time, Tashlhiyt has been the "forgotten cousin" of the digital world. While big languages like English or French have perfect dictionaries and spell-checkers, Tashlhiyt is like a city with no traffic lights.

The Chaos: People write the same word in 10 different ways.
The Gap: Computers need a "standard" version to learn. Without it, AI tools for translation or voice assistants just crash or give nonsense answers.

2. The Solution: The "Translator's Bridge"

The authors built DATASHI, which is essentially a giant bridge between English and Tashlhiyt.

The Collection: They gathered 5,000 sentences.
The "Real" Side: They asked regular people (non-experts) to write these sentences in their own natural, messy, "text-speak" style. This captures how people actually talk and type.
The "Gold" Side: They asked language experts to rewrite those same 1,500 sentences into perfect, standardized Tashlhiyt.
The Result: Now, computers have a "Before" (messy) and "After" (clean) version of the same text. It's like having a "Fix-It" guide that shows the computer exactly how to turn "ghakudan" into the correct "G akud ann."

3. The Test: Can AI Learn to Clean Up?

The researchers didn't just build the bridge; they tested the world's smartest AI robots (like GPT-5, Gemini, and Claude) to see if they could learn to clean up the messy writing just by looking at the examples.

Think of it like a game of "Spot the Difference":

The Challenge: The AI sees a messy sentence and has to guess the clean version.
The Results:
- Some AIs were like toddlers trying to solve a puzzle; they made lots of mistakes, deleting important letters or adding random ones.
- The Winner: Gemini-2.5-Pro was the star student. It learned the rules the fastest and made the fewest mistakes. It was especially good at handling the tricky parts of the language, like "gemination" (doubling consonants, like tt or ll), which are like the secret spices of Tashlhiyt that change the meaning of a word.

4. Why This Matters: Beyond Just Text

Why do we care if a computer can spell Tashlhiyt correctly?

Voice Assistants: If a computer can't read the text, it can't understand the voice. By fixing the spelling, this project paves the way for Siri or Alexa to eventually understand a grandmother speaking Tashlhiyt in the Souss region of Morocco.
Preservation: It helps save the language in the digital age, ensuring it doesn't get lost because it's "too hard" for computers to handle.
Fairness: It gives Tashlhiyt speakers the same digital tools that English speakers take for granted.

The Big Picture

In short, DATASHI is a dictionary, a training manual, and a bridge all rolled into one. It takes the chaotic, beautiful, real-world way people write Tashlhiyt and teaches the world's smartest computers how to understand it. It's a small step for a computer, but a giant leap for the Tashlhiyt language, turning a "low-resource" language into one that can finally thrive in the digital world.

1. Problem Statement

The paper addresses the critical scarcity of computational resources for Tashlhiyt (also known as Shilha), one of the three major Amazigh (Berber) varieties spoken in Morocco. Despite having millions of speakers, Tashlhiyt is severely under-represented in Natural Language Processing (NLP) compared to Standard Moroccan Tamazight.

Key challenges identified include:

Orthographic Diversity: There is no single, universally applied writing standard. Users employ a mix of Neo-Tifinagh (official), Latin, Arabic scripts, and non-standard "chat" orthographies (using numerals like '3', '7', '9' for specific phonemes).
Data Scarcity: Existing datasets are small, thematically narrow, or lack parallel alignment with major languages, hindering the training of robust models.
Normalization Gap: The lack of standardized text prevents effective tokenization, machine translation, and speech recognition. There is a specific methodological gap in addressing orthographic normalization for Tashlhiyt using modern Large Language Models (LLMs).

2. Methodology

A. Corpus Creation (DATASHI)

The authors created DATASHI, a parallel English–Tashlhiyt corpus designed to capture both authentic user variation and expert-standardized forms.

Scale: 5,000 sentence pairs covering 20 thematic domains (e.g., daily life, technology, religion).
Data Collection:
- Non-Standard Side: 34 native speakers (non-experts) transcribed English sentences using their spontaneous, non-standard writing conventions. This captures real-world orthographic noise, including inconsistent spacing, spelling, and script mixing.
- Standardized Side: A subset of 1,500 sentences was manually standardized by 7 Amazigh language experts using the Berber Latin script (aligned with IRCAM conventions). This serves as the "Gold Standard" for evaluation.
Linguistic Design: The corpus prioritizes lexical authenticity over prescriptive neologisms, retaining common loanwords (Darija, French, English) and archaic forms that are still in use, ensuring the data reflects actual communicative practices.

B. Normalization Framework

The study defines a normalization process that operates on three levels:

Character Level: Mapping numerals and digraphs to standard phonemes (e.g., 7 $\to$ h., gh $\to$ G) and correcting vowel representations influenced by French.
Morpho-syntactic Level: Correcting dialectal variations (e.g., regional suffixes) and enforcing grammatical rules (e.g., negative verb forms).
Phonological Level: Resolving assimilation and fusion (e.g., ayt dari $\to$ ayddari) to preserve canonical forms.

C. Experimental Setup

The authors evaluated the performance of five state-of-the-art Large Language Models (LLMs) on the normalization task without fine-tuning:

Models: GPT-5, Claude-Sonnet-4.5, Gemini-2.5-Pro, Mistral-Large-2411, and Qwen3-Max.
Settings:
- Zero-Shot: Models received instructions and rules but no examples.
- Few-Shot: Models received the same instructions plus 30 curated demonstration examples.
Metrics: Performance was measured using Word Error Rate (WER) and Levenshtein Distance (LD), with a granular analysis of edit operations (deletions, substitutions, insertions) across specific phonological classes (geminates, emphatics, uvulars, pharyngeals).

3. Key Contributions

First Parallel Corpus: DATASHI is the first large-scale (5,000 pairs), parallel English–Tashlhiyt corpus specifically designed to bridge the gap between non-standard user writing and expert standardization.
Dual-Representation Design: By providing both raw user-generated text and expert-standardized versions, the corpus enables systematic study of orthographic diversity and serves as a benchmark for normalization models.
Phonological Granularity: The corpus includes detailed statistics on complex Tashlhiyt features, such as consonant gemination, labialization, and emphatic/pharyngeal consonants, which are often ignored in general NLP datasets.
LLM Benchmarking: The paper provides the first comprehensive evaluation of how top-tier LLMs handle orthographic normalization for a low-resource, morphologically rich language, offering diagnostic insights into their strengths and weaknesses.

4. Results

Overall Performance

Best Performer: Gemini-2.5-Pro achieved the lowest error rates in both settings (Zero-shot WER: 37.8%; Few-shot WER: 35.5%).
Impact of Few-Shot: All models showed improvement with few-shot prompting, confirming that in-context learning significantly aids normalization tasks for low-resource languages.
Worst Performer: Qwen3-Max struggled the most, particularly in zero-shot scenarios (WER 63.4%), indicating difficulty with the morphological complexity without examples.

Fine-Grained Phonological Analysis

Gemination: This was the most challenging category for all models. Mistral and Qwen3-Max exhibited massive deletion and substitution errors regarding geminated consonants (e.g., tt, ll), failing to preserve consonant length contrasts.
Emphatics & Pharyngeals: Models generally performed better on these, but Mistral and GPT-5 showed higher substitution rates compared to Gemini and Claude.
Error Types: Substitution errors were more frequent than deletions or insertions across all models, suggesting models tend to "guess" the wrong character rather than omitting or adding them. However, Qwen3-Max showed a tendency for excessive insertions, particularly in geminated segments.

5. Significance and Future Directions

NLP Advancement: DATASHI fills a critical resource gap, enabling the development of tokenizers, translators, and normalization tools specifically for Tashlhiyt and related Amazigh varieties.
Multimodal Potential: The standardized text design facilitates cross-domain transfer to speech technologies. Consistent text normalization is a prerequisite for building pronunciation models, forced alignment, and speech recognition systems for low-resource languages.
Model Diagnostics: The study reveals that while foundation models have strong cross-lingual generalization, they struggle with specific typological features of Amazigh (particularly gemination and non-concatenative morphology). This highlights the need for subword-level modeling and orthography-aware training.
Future Work: The authors plan to expand DATASHI with read-speech data, fine-tune multilingual models on the normalized text, and integrate the corpus into broader cross-Amazigh benchmarks to promote resource equity.

In conclusion, DATASHI represents a foundational step toward democratizing NLP for Amazigh languages, providing both the data and the methodological framework necessary to tackle the unique challenges of orthographic normalization in low-resource settings.