Creating ConLangs to Probe the Metalinguistic Grammatical Knowledge of LLMs

1. Problem Statement

The paper addresses two primary challenges in the intersection of linguistics and Artificial Intelligence:

1. ConLang Creation: While tools exist for creating Constructed Languages (ConLangs), there is a lack of automated, agentic systems that can systematically generate all linguistic components (phonology, morphology, syntax, lexicon, orthography) based on high-level user specifications.
2. Metalinguistic Knowledge Evaluation: Current evaluations of Large Language Models (LLMs) often focus on encyclopedic knowledge or performance in natural languages. There is a need to probe whether LLMs possess deep metalinguistic reasoning capabilities—specifically, the ability to understand abstract grammatical concepts (e.g., ergativity, word order permutations, case marking) and apply them to novel, unseen linguistic structures without relying on memorized patterns from training data.

2. Methodology: IASC System

The authors introduce IASC (Interactive Agentic System for ConLangs), a modular, agentic framework designed to generate ConLangs. The system operates through a pipeline of distinct modules, each utilizing specific prompts and iterative refinement.

System Architecture

The system consists of five main modules:

• Phonotactics: Generates a Python program to create morphemes based on a target language's phoneme inventory and syllable structure (e.g., mimicking Welsh or Japanese).
• Morphosyntax (Core Focus): Transforms source English sentences into the target ConLang's grammatical structure. This is the primary benchmark for evaluating LLM capabilities.
• Lexicon: Generates lexical entries (stems and affixes) based on the phonotactic rules.
• Orthography: Creates a writing system (e.g., Latin, Cyrillic, Arabic scripts) and a conversion program from IPA to the target script.
• Handbook Generation: Synthesizes a grammatical description of the generated language.

Agentic Workflow

The system employs an agentic approach where the LLM iteratively refines its output:

1. Generation: The LLM generates code or linguistic rules based on a prompt.
2. Execution & Feedback: The system executes the generated code (e.g., generating morphemes or glosses).
3. Refinement: The output is analyzed for errors (e.g., missing phonemes, incorrect spacing, logic errors). The LLM is prompted again with the previous output and error feedback to correct and improve the module.

Experimental Design (Morphosyntax)

To evaluate metalinguistic knowledge, the authors focused on the Morphosyntax module:

• Task: Transform English source sentences into target sentences adhering to specific grammatical constraints (e.g., changing word order from SVO to OSV, applying Ergative-Absolutive case marking).
• Approach: They used a Cumulative Morphosyntax approach. Instead of a single-shot transformation, the model iteratively applies grammatical features (e.g., first applying word order, then case marking, then tense) to avoid prompt complexity and ensure adherence to specific rules.
• Dataset: An evaluation dataset of 45 source sentences transformed across 9 distinct feature sets:
  • Typologically Common: Arabic (VSO), French (SVO), Welsh (VSO).
  • Typologically Rare: Fijian (VOS), Mizo (OSV), Hixkaryana (OVS).
  • Synthetic "Hard" Language: A combination of rare features (OSV, Ergative-Absolutive, prefixation, complex morphology).
• Models Tested: Six LLMs (Claude 3.5 Sonnet, Gemini 2.5 Flash/Pro, GPT-4o-mini, GPT-5-mini, GPT-5).
• Metrics:
  • TER (Translation Edit Rate): Measures word order and structural edits.
  • SER (Stem Edit Rate): Measures syntactic correctness by ignoring morphological tags.
  • MFER (Morphological Feature Error Rate): Measures the accuracy of affix placement and form.
  • MSER (Morphosyntactic Error Rate): A weighted average of SER and MFER.
  • Lem F1: Measures the ability to correctly lemmatize (identify the base form of) words.

3. Key Contributions

1. IASC Framework: A fully functional, open-source system for the automated generation of ConLangs, demonstrating that LLMs can act as "AI Linguists" to synthesize complex linguistic systems.
2. Novel Benchmark: The creation of a rigorous evaluation dataset specifically designed to test metalinguistic reasoning rather than translation or encyclopedic recall.
3. Agentic Refinement: Demonstrating that iterative, feedback-driven prompting significantly improves the quality of generated linguistic rules and outputs compared to single-shot generation.
4. Insight into LLM Limitations: Providing empirical evidence that LLM performance correlates strongly with the typological frequency of linguistic features in their training data.

4. Results

The experiments revealed significant variations in performance based on model size and linguistic typology:

• Typological Bias: LLMs performed significantly better on typologically common patterns (e.g., SVO, VSO, suffixing) than on rare ones.
  • Example: Models achieved near-perfect Stem Edit Rates (SER) on Welsh and Arabic (VSO) but struggled significantly with Fijian (VOS) and Mizo (OSV).
  • The "Hard" language (combining rare features) yielded the highest error rates across all models, with average MSER scores reaching ~85% for some models.
• Model Size Correlation: Larger, more capable models (Gemini 2.5 Pro, GPT-5, Claude 3.5 Sonnet) outperformed smaller models (GPT-4o-mini, GPT-5-mini).
  • Gemini 2.5 Pro with few-shot In-Context Learning (ICL) achieved the best overall performance (lowest MSER of 14.68).
  • Smaller models often failed to follow complex instructions, ignoring morphology or failing to lemmatize.
• Analytic vs. Synthetic: Models struggled more with analytic languages (relying on word order and function words, e.g., Vietnamese, Mizo) regarding lemmatization, whereas they performed better on synthetic languages (rich inflection) where morphological cues were explicit.
• Impact of Few-Shot Learning: Providing 4 example sentences with grammatical glosses (ICL) significantly improved performance for most models, particularly for the larger models, reducing MSER scores by substantial margins (e.g., Gemini 2.5 Pro dropped from 54.18 to 14.68).

5. Significance

• Probing Linguistic Competence: The study suggests that while LLMs possess significant metalinguistic knowledge, it is heavily dependent on the frequency and documentation of linguistic features in their training data. They struggle with "impossible" or extremely rare configurations that deviate from the statistical norms of natural languages.
• The "AI Linguist" Concept: The paper validates the feasibility of using LLMs to assist in low-resource language documentation and translation by distilling grammatical rules into transformation pipelines.
• Future of ConLangs: IASC democratizes the creation of complex, consistent artificial languages, moving beyond simple word lists to fully grammatical systems.
• Limitations & Future Work: The authors note that current systems are biased toward English structures and agglutinative strategies. Future work aims to improve handling of fusional languages, non-affixal morphology, and low-resource language translation.

In conclusion, the paper establishes that LLMs are powerful tools for generating ConLangs and possess a degree of abstract grammatical reasoning, but their capabilities are bounded by the typological distribution of their training data, particularly struggling with rare word orders and complex morphological interactions.