CommonMorph: Participatory Morphological Documentation Platform

1. Problem Statement

The documentation of morphological data (word structure and inflection patterns) is a critical bottleneck in preserving endangered and low-resource languages. Current approaches face several significant challenges:

• High Labor Intensity: Traditional methods, such as Finite-State Transducers (FSTs), require extensive linguistic engineering expertise and manual effort (estimated at ~40 person-hours per language), making them unscalable for the 1,500+ languages at risk of extinction.
• Data Scarcity: Modern data-driven approaches (neural models) require large volumes of annotated data, which rarely exist for low-resource languages. Existing resources like UniMorph cover less than 2.5% of the world's languages.
• Lack of Community Integration: Existing tools often rely on isolated expert contributions, lacking mechanisms for broad community validation or the systematic reuse of data from related language varieties.
• Generalization Issues: Rule-based systems struggle with out-of-vocabulary (OOV) forms and irregularities, while pure data-driven models fail in "zero-shot" scenarios without prior data.

2. Methodology: The CommonMorph Platform

CommonMorph is a web-based, open-source platform designed to streamline morphological data collection through a three-tiered collaborative workflow involving linguists, expert speakers, and non-expert community members.

A. System Architecture & Workflow

The platform operates on a hybrid Active Learning framework that transitions from rule-based heuristics to data-driven models as data accumulates:

1. Linguist Initialization: Linguists define the initial scope by inputting lemmas, inflection classes, paradigm structures, and morphophonological rules. They can import and adapt materials from related languages to reduce redundancy.
2. Elicitation & Validation:
   • Expert Speakers: Can view full paradigm tables and validate bulk data.
   • Non-Expert Speakers: Interact with simplified, context-aware prompts (e.g., "How do you tell more than one person to [verb] right now?") generated by the system. They verify or correct system suggestions.
3. Iterative Model Improvement: As validated data accumulates, the system retrains a neural inflection model to improve its prediction accuracy, reducing the manual correction burden over time.

B. Technical Components

• Multi-Source Prediction Pipeline: The system aggregates suggestions from three sources to maximize accuracy:
  1. Linguist Rules: Pre-defined morphophonological rules and patterns (crucial for zero-resource start).
  2. Neural Model: A lightweight, 2-layer LSTM (128 hidden dimensions) trained on validated data. It is designed to be trained on minimal server specs (CPU, ~1 minute per epoch).
  3. Large Language Models (LLMs): Used for few-shot prompting (1–3 shots) via API (e.g., Gemini-2.5-Flash, gpt-5-mini) to bridge the gap during low-data phases.
• Disagreement Handling: If multiple speakers propose conflicting forms for the same slot, the system flags the entry for expert consensus or linguist adjudication.
• Interoperability: Outputs are exportable in UniMorph format (TSV), ensuring compatibility with existing NLP tools and downstream tasks.
• Offline Capability: To accommodate users with unstable internet, linguists can print empty tables for offline completion, which are later re-integrated.

C. Design Principles

• Role-Specific Interfaces: Tailored UIs for linguists (complex configuration) and speakers (simplified, gamified interaction).
• Cross-Linguistic Reuse: Allows importing paradigms from related languages (e.g., using Kurdish data to assist Hawrami documentation), leveraging shared morphological patterns.
• Adaptive Complexity: Supports diverse morphological typologies, including fusional (Spanish, English), agglutinative (Turkish), and root-and-pattern (Arabic) systems.

3. Key Contributions

1. Unified Participatory Platform: The first system to integrate structured linguist workflows with community-driven validation in a single digital environment, specifically targeting low-resource languages.
2. Hybrid Active Learning Framework: A novel pipeline that dynamically combines expert linguistic rules, lightweight neural networks, and LLMs to optimize data collection efficiency from "cold start" to mature datasets.
3. Scalable Documentation Model: Demonstrates a replicable model for preserving linguistic diversity by lowering the barrier to entry for non-expert speakers while maintaining high data quality through expert oversight.
4. Open-Source Ecosystem: The platform is freely available, with code hosted on GitHub, promoting transparency and further development.

4. Results & Evaluation

The platform was evaluated across diverse language families (Indo-European, Semitic, Bantu, Turkic, Iranic) and through user surveys.

• Annotation Accuracy (Character Error Rate - CER):
  • Zero-Resource Performance: Linguist-defined patterns significantly reduced CER for zero-resource languages (e.g., Hawrami CER dropped from ~12% with patterns alone to ~5.7% with added morphophonological rules).
  • Neural Learning: As training data increased (100 to 2000 samples), the neural model's CER consistently decreased, validating the active learning approach.
  • LLM Performance: Few-shot prompting with LLMs achieved competitive results for well-resourced languages (e.g., Spanish, Turkish) but struggled with fully diacritized Arabic due to a mismatch in training data distribution.
• User Experience (Survey Results):
  • Ease of Use: Rated 4.7/5 by linguists and 5.0/5 by speakers.
  • Efficiency: Speakers reported a short learning curve (avg. 8.7 minutes) compared to linguists (17.3 minutes) but contributed significantly more time (4.1 hours vs. 1.0 hour).
  • Pedagogical Value: High scores indicate the platform effectively aids language learning and awareness among community members.
• Case Study Success: Successfully generated the first comprehensive digital verbal morphology data for the endangered Hawrami language (Tewêle dialect).

5. Significance

CommonMorph addresses a critical gap in computational linguistics and language preservation by:

• Democratizing Documentation: It shifts the paradigm from "expert-only" fieldwork to a collaborative model where community members are active co-creators of linguistic resources.
• Bridging the Data Gap: By effectively utilizing active learning and cross-linguistic transfer, it makes high-quality morphological datasets feasible for languages that previously lacked them.
• Future-Proofing: The platform's ability to generate interlinear glossed text (IGT) and its compatibility with UniMorph positions it as a foundational tool for future NLP applications in low-resource settings.
• Ethical Framework: The system prioritizes community ownership and data sovereignty, ensuring that linguistic resources are built with speakers rather than on them.

In conclusion, CommonMorph offers a scalable, efficient, and inclusive solution for the urgent task of documenting the world's endangered languages, combining the rigor of linguistic expertise with the power of modern machine learning.