Technical Summary: DaLA – Danish Linguistic Acceptability Evaluation Guided by Real World Errors

Problem Statement

Linguistic acceptability evaluation, the task of determining the well-formedness or naturalness of a sentence, is critical for assessing the linguistic capabilities of Large Language Models (LLMs). While significant resources exist for widely spoken languages, low-resource languages like Danish suffer from a scarcity of high-quality datasets. Existing Danish resources, such as ScaLA (Nielsen, 2023), rely on limited corruption strategies (e.g., word removal and swapping) that are not representative of real-world errors made by native speakers. Furthermore, these prior methods often depend on specific annotation formats (e.g., Universal Dependencies POS tags) and lack rigorous validation to ensure that corrupted sentences are genuinely ungrammatical. This gap hinders the training and evaluation of multilingual models, particularly given the subtle errors that arise when models process closely related languages (e.g., Danish and Norwegian).

Methodology

The authors introduce DaLA (Danish Linguistic Acceptability Evaluation), a dataset and evaluation framework designed to reflect real-world linguistic errors through a systematic, automated corruption process.

1. Error Analysis and Corruption Functions

The core of DaLA is a set of 14 new corruption functions derived from an analysis of common errors made by native Danish speakers. The selection process utilized the "Typiske Problemer" list from the Danish Language Council and Society of Danish Language and Literature. Errors were scored based on implementation complexity, error type (grammatical, semantic, mechanical), and the likelihood of producing a genuinely incorrect sentence.

• Error Types: The 14 new types include confusions between specific pronouns (nogle vs. nogen, som vs. der), verb pairs (ligge vs. lægge), suffixes (-ende vs. -ene), the "R problem" (suffix confusion), genitive corruption, and determiner errors.
• Implementation: Each error type is implemented as a dedicated function using the SpaCy library. The authors adopted a conservative approach, filtering input sentences to minimize cases where a corruption might accidentally result in a grammatically correct sentence.

2. Iterative Corruption Algorithm

To ensure a balanced distribution of error types despite the varying frequency of linguistic phenomena in the source data, the authors employ an iterative algorithm (Algorithm 1).

• Process: Corruption functions are sorted by their applicability frequency on the starting dataset. The algorithm applies the least frequent corruption first, marking sentences as "corrupted" once an error is injected. Subsequent, more frequent corruptions are applied only to sentences that remain uncorrupted.
• Source Data: The base sentences are drawn from the Universal Dependencies Danish (UDD) corpus, assumed to be linguistically acceptable.

3. Validation Framework

To ensure the quality of the generated dataset, a hybrid validation process was implemented:

• Automatic Validation: The writeassistant tool, a leading Danish grammatical error correction system, was used to classify corrupted sentences. The system achieved an average precision of 99.7% in identifying truly corrupted sentences.
• Manual Validation: An expert linguist manually reviewed sentences that the automatic system failed to flag or misclassified. For high-volume error types (pronoun and suffix determiner corruption), a subset of 304 examples was annotated to statistically adjust the precision scores.
• Result: The final average precision of the corruption functions is approximately 0.957, confirming that the vast majority of generated sentences are indeed ungrammatical.

4. Dataset Construction

The final DaLA dataset consists of sentence pairs (one correct, one corrupted).

• Standard Split: 1,664 pairs (512 training, 1,024 validation, 128 test) to maintain comparability with ScaLA.
• Extended Versions: "DaLA Medium" (4,592 training samples) and "DaLA Large" (6,124 training samples) are also provided, totaling up to 7,656 sentences.
• Labels: Each pair includes a binary acceptability label and a classification of the specific error type.

Key Contributions

1. Real-World Error Derivation: The derivation of 14 new corruption functions based on common native speaker errors, moving beyond the limited scope of previous Danish benchmarks.
2. Robust Dataset Creation: The creation of DaLA, a dataset of 3,328 sentences (with extended versions available) that does not rely on pre-existing acceptability judgments or specific annotation formats, making it applicable to arbitrary Danish sentence sources.
3. Rigorous Validation: The implementation of a dual-phase (automatic and manual) validation process to ensure high precision in error injection, addressing a gap in previous datasets where corruption accuracy was unverified.
4. Benchmarking: The provision of baseline results for state-of-the-art open-source and open-weight LLMs on this new benchmark.

Results

The authors evaluated several top-performing LLMs (including encoder-only, instruction-tuned, base generative, and reasoning models) on DaLA and compared their performance against the ScaLA benchmark using the Matthews Correlation Coefficient (MCC).

• Increased Difficulty: Nine out of ten models showed a performance decrease on DaLA compared to ScaLA. The average performance drop was 6.04%, with some models experiencing a decline of up to 14.55%.
• Discriminatory Power: The dataset demonstrated higher discriminatory power. Notably, reasoning models (e.g., Qwen3) suffered the sharpest decline (average 14.32%), suggesting that DaLA better captures nuances in linguistic acceptability that previous benchmarks missed.
• Model Specificity: The DFM Sentence Encoder Large was the only model to show improved performance, likely due to its specialized Danish-only training, suggesting DaLA favors models with deeper, language-specific knowledge.

Significance and Claims

The paper claims that DaLA represents a significant advancement in the evaluation of Danish linguistic acceptability by shifting from synthetic, rule-based errors to real-world, data-driven errors.

• Rigor: By incorporating a wider variety of corruption types and validating them rigorously, DaLA provides a more comprehensive and difficult test for LLMs.
• Representativeness: The benchmark is argued to be more representative of actual usage scenarios and common mistakes, offering a more authentic assessment of a model's linguistic capabilities.
• Differentiation: The results suggest that DaLA is better at distinguishing between high-performing and low-performing models, particularly in identifying models that rely on superficial patterns versus those with genuine grammatical understanding.

The authors conclude that while the method is currently rooted in Danish linguistic rules, the framework is extensible to other languages and error types (semantic, pragmatic), offering a pathway for more realistic linguistic evaluation across low-resource languages.