AfriSUD: A Dependency Treebank Collection for Evaluating Models on African Languages

This paper introduces AfriSUD, the first large-scale collection of native-speaker verified syntactic treebanks for nine diverse African languages using the SUD framework, and demonstrates that current NLP models still face significant limitations in capturing the structural diversity of African-language syntax.

The Gist

Imagine the world of language technology as a massive library. For decades, this library has been filled with books about English, French, and Chinese, but the shelves dedicated to African languages have been almost empty. This paper, AfriSUD, is an attempt to finally fill those empty shelves with high-quality, organized books so computers can learn to understand African languages properly.

Here is a breakdown of what the researchers did, using simple analogies:

1. The Problem: The "Missing Map"

Think of a Dependency Treebank as a detailed map of a city. It doesn't just list the buildings (words); it draws the roads connecting them (grammar) and explains how they relate to each other (who is the boss, who is the helper, who is the object).

For a long time, computers trying to understand African languages were like tourists dropped in a city without a map. They could guess the names of the buildings, but they kept getting lost trying to figure out the streets. While there are some maps for a few African cities, they are scattered, inconsistent, or missing entirely.

2. The Solution: Building the "AfriSUD" Collection

The team created AfriSUD, which is like building a brand-new, standardized set of maps for nine different African cities (languages) that are very different from one another.

• The Cities: They chose languages from different "neighborhoods" (language families), including agglutinative ones (where words are like Lego blocks that snap together to make long, complex words) and isolating ones (where words stand alone).
• The Blueprint: They used a specific blueprint called SUD (Surface-Syntactic Universal Dependencies). Think of this as a universal architectural style that ensures all the maps look consistent, even if the buildings (words) look very different.
• The Builders: They didn't just use robots. They hired native speakers (local experts) to draw these maps by hand. This ensures the "roads" make sense to the people who actually live there.

3. The Test: Can the Computers Read the Maps?

Once the maps were built, the researchers asked: Can modern AI computers actually read and understand these new maps?

They tested three types of "students":

1. The Traditional Student (Stanza): A classic, rule-based computer program.
2. The Multilingual Student (Pre-trained Models): AI that has read books in many languages but isn't an expert on African ones yet.
3. The Super-Intelligent Student (LLMs): The newest, massive AI models (like the ones you might chat with) that are very smart but haven't been specifically trained on this data.

4. The Results: The "Head vs. Heart" Gap

The results were a mix of good news and a reality check:

• The Good News: The computers got much better at identifying the "Head" of a sentence. Imagine a sentence is a family tree. The computers were pretty good at pointing to the "Parent" word (the main verb or noun).

• The Bad News: They struggled with the "Relationship." While they knew who the parent was, they often got the relationship wrong.

  • Analogy: The computer might correctly identify that "Dog" and "Bark" are the main words, but it might think the Dog is barking at the tree, when actually the Dog is barking because of the tree.
  • The Gap: There is a huge gap between knowing the structure (UAS) and knowing the specific label (LAS). The computers are better at seeing the skeleton of the sentence than understanding the specific grammar rules holding it together.

• Who Won?

  • The Traditional Student (Stanza) actually performed the best overall. It's like an old-school mechanic who knows the specific quirks of these engines better than the new, flashy AI.
  • The Super-Intelligent Students (LLMs) were very smart but needed help. When the researchers gave them a few examples to look at first (called "few-shot prompting"), they got much better. However, even with help, they still couldn't beat the traditional student in accuracy.

5. Where Did They Get Stuck?

The researchers looked closely at where the computers failed. They found the AI struggled most with:

• Serial Verb Constructions: When a sentence uses a chain of verbs to tell a story (common in African languages), the AI often flattened the chain, missing the nuance.
• Tense and Mood Chains: Complex chains of words that indicate when and how something happened were often misunderstood.
• Possessives: Figuring out who owns what was tricky.

The Bottom Line

This paper is a foundational step. It's not about building a perfect translator or a clinical tool yet; it's about building the library.

The researchers have successfully created the first large-scale, high-quality "grammar maps" for nine African languages. They proved that while our current AI is getting smarter, it still lacks the deep, structural understanding needed to truly master the complex and beautiful grammar of African languages. The maps are ready; now, the AI needs to learn how to read them.

Technical Summary

Technical Summary: AfriSUD – A Dependency Treebank Collection for Evaluating Models on African Languages

1. Problem Statement

Despite accounting for nearly a third of global linguistic diversity, African languages remain significantly underrepresented in Natural Language Processing (NLP) research and resources. While foundational tasks like machine translation and information extraction rely heavily on dependency parsing, most African languages lack gold-standard treebanks required for developing and evaluating syntactic models. Existing large-scale efforts, such as Universal Dependencies (UD), cover fewer than 15 African languages out of 186 documented languages. This scarcity creates a barrier to building grammar-aware tools, conducting cross-lingual transfer, and evaluating state-of-the-art multilingual models in low-resource settings. Furthermore, many African languages exhibit complex morphological and phonological features—such as agglutination, extensive noun class systems, and tonal distinctions—that are poorly supported by standard NLP pipelines and existing annotated corpora.

2. Methodology

Dataset Construction: AfriSUD

The authors introduce AfriSUD, the first large-scale, community-driven collection of syntactically annotated treebanks for nine diverse African languages. The dataset spans West, East, and Southern Africa, covering three major language families:

• Niger-Congo: Swahili, Kinyarwanda, Runyankore, isiXhosa (Bantu subgroup); Wolof (Senegambian); Yorùbá and Efik (Volta–Niger and Cross River).
• Afro-Asiatic: Hausa (Chadic branch).
• Creole: Naija (Nigerian Pidgin, English-based).

The corpus includes approximately 8,000 to 1,500 sentences per language, sourced from news, educational primers, and translated texts. The languages vary typologically, ranging from agglutinative (e.g., Bantu languages) to isolating (e.g., Yorùbá, Naija) and fusional (e.g., Hausa).

Annotation Framework:
The treebanks are annotated using the Surface-Syntactic Universal Dependencies (SUD) framework. SUD was selected for its ability to represent morphological and syntactic relations with high fidelity while maintaining cross-linguistic consistency, and its compatibility with conversion to UD.

• Pipeline: The process includes lemmatization, Universal Part-of-Speech (UPOS) tagging, dependency head annotation, and dependency relation labeling.
• Tools: Annotation was performed using Arborator-Grew. Initial pre-annotations were generated using a parser interface based on BertForDeprel, followed by manual correction.
• Quality Control: Each language was annotated by three native-speaking linguists (one coordinator, two annotators). A consensus-based adjudication procedure resolved disagreements. Automatic validation checks ensured structural integrity (e.g., no cycles, single root).

Annotation Challenges:
The paper details specific challenges in applying SUD to these languages:

• Morphological Binding: Decisions were made between morphological decomposition (segmenting roots and affixes) and single-token preservation. For example, Efik and Yorùbá tokens were segmented, while isiXhosa and Hausa maintained lexical integrity to align with SUD's surface-oriented approach.
• Ambiguity: Resolving lexical and syntactic ambiguities (e.g., multifunctional morphemes in Yorùbá) relied on context-driven analysis and team conventions.
• Language-Specific Relations: Adapting SUD relations for phenomena like serial verb constructions, Tense-Aspect-Mood (TAM) auxiliaries, and possessive markers required careful adaptation, sometimes deviating slightly from standard guidelines to reflect linguistic reality.

Experimental Setup

The authors evaluated a diverse range of models on AfriSUD for Part-of-Speech (POS) tagging and dependency parsing:

1. Baselines: Stanza (neural dependency parser) initialized with fastText embeddings.
2. Multilingual Encoders: Fine-tuned end-to-end biaffine parsers using general models (mBERT, XLM-RoBERTa) and Africa-centric models (AfriBERTa, AfroXLMR, AfroXLMR-large-76L).
3. Large Language Models (LLMs): Zero-shot and few-shot prompting experiments with Gemini-3.1-Pro, GPT-5.2, GPT-4o, and Gemma-3 (12B/27B). A supervised fine-tuning (SFT) baseline was also established for Gemma-3-12B.
4. Cross-lingual Transfer: Evaluated zero-shot transfer from seven source languages (including English, French, and African languages like Wolof and Naija) to the nine target languages.

3. Key Results

Baseline Performance

• POS Tagging: Transformer-based encoders generally outperformed Stanza. The Africa-centric model AfroXLMR-large-76L achieved the highest macro-average accuracy (90.6%), compared to Stanza's 88.4%.
• Dependency Parsing: Stanza remained the strongest baseline for dependency parsing, achieving the highest average Unlabeled Attachment Score (UAS: 84.4) and Labeled Attachment Score (LAS: 77.5). Among transformers, Africa-centric encoders (AfroXLMR-large-76L) outperformed general multilingual models, reaching 83.1 UAS and 73.9 LAS.
• UAS vs. LAS Gap: Across all models, LAS was consistently lower than UAS, indicating that while models can identify dependency heads, assigning the correct syntactic relation labels remains significantly more difficult.

LLM Performance

• Prompting: In-context demonstrations (few-shot prompting) consistently improved performance across all LLMs and metrics, with the most significant gains observed in LAS. For instance, GPT-5.2's LAS improved from 16.5 (0-shot) to 52.5 (5-shot).
• Comparison: Despite improvements, LLMs (even with 5-shot prompting) generally underperformed compared to supervised parsers like Stanza. Gemini-3.1-Pro performed best among LLMs (5-shot LAS: 59.2), but still lagged behind Stanza (LAS: 77.5).
• SFT vs. Prompting: Supervised fine-tuning of Gemma-3-12B significantly outperformed few-shot prompting, reducing the gap to the best-performing prompting model (Gemini-3.1-Pro) to just 1.2 LAS points.

Cross-lingual Transfer

Zero-shot transfer results using AfroXLMR-large-76L showed that transfer performance is highly dependent on the source-target pair. Using Naija (pcm) augmented with existing treebanks (pcm+nsc) yielded the best overall transfer results. Among non-African sources, Romanian performed best. Target languages like Swahili were easier to parse, while Efik and isiXhosa remained challenging.

Error Analysis

Analysis of LLM errors (specifically Gemini-3.1-Pro) revealed a "syntax gap":

• Head vs. Label Mismatch: Models were better at predicting dependency heads than relation labels. For example, in serial verb constructions (comp:svc), the model achieved 0% LAS but 46% UAS.
• Specific Failures: Errors concentrated in:
  • Serial Verb Constructions: Often flattened into generic mod or comp:obj relations.
  • TAM Auxiliaries: Multi-token chains (e.g., in Naija) were often flattened rather than modeled as cascading dependencies.
  • Possessives: Frequently mislabeled as mod or comp:obj.
  • Underspecified Relations (udep): Models struggled to distinguish between arguments and modifiers in ambiguous contexts.

4. Contributions and Significance

The paper makes the following primary contributions:

1. Resource Creation: The release of AfriSUD, the first large-scale, community-verified collection of SUD treebanks for nine typologically diverse African languages.
2. Methodological Documentation: A detailed account of annotation challenges and solutions specific to African languages, including strategies for handling agglutination, clitics, and tone within the SUD framework.
3. Comprehensive Benchmarking: A systematic evaluation of supervised parsers, multilingual encoders, and LLMs, establishing new baselines for African language syntactic NLP.
4. Identification of Gaps: The work highlights a persistent "syntax gap" where current architectures, including LLMs, struggle to capture the structural diversity of African languages, particularly in relation labeling and complex constructions like serial verbs and TAM chains.

Significance:
The authors position AfriSUD as a foundational resource for more linguistically grounded evaluation and syntactic modeling of African languages. By demonstrating that existing architectures (both transformer-based and LLMs) show clear limitations in capturing African syntactic structures, the paper argues for the necessity of continued resource development and architectural adaptation. The dataset and benchmarks are intended to facilitate reproducible research and support the development of grammar-aware tools for millions of speakers across Sub-Saharan Africa.

Limitations:
The authors note that the dataset covers a limited number of examples for certain rare dependency relations. Additionally, the LLM experiments are restricted to specific models and settings, and the reproducibility of closed-model results may be affected by future model updates. The error analysis focuses on specific SUD relations and the best-performing prompting model.