RoLegalGEC: Legal Domain Grammatical Error Detection and Correction Dataset for Romanian

1. Problem Statement

The paper addresses the scarcity of high-quality, annotated data for Grammatical Error Detection (GED) and Grammatical Error Correction (GEC) in low-resource languages, specifically Romanian, within the legal domain.

• Context: Legal texts require high precision; grammatical or semantic errors can lead to misinterpretation of laws.
• Gap: While English legal NLP is well-developed, Romanian lacks dedicated datasets for GED/GEC in legal contexts. Existing Romanian datasets (e.g., RoGEC) are general-purpose and often rely on synthetic data generated via simple noise injection, which fails to capture the nuanced, domain-specific errors found in legal documents.
• Challenge: Manually annotating legal texts is expensive and time-consuming. Synthetic generation requires a deep understanding of Romanian grammar to produce realistic errors without rendering text unintelligible.

2. Methodology

The authors propose a comprehensive pipeline involving the creation of a new dataset and the evaluation of various neural architectures.

A. The RoLegalGEC Dataset

• Source Corpora: The dataset is built upon two clean, grammatically correct legal corpora:
  1. MARCELL-RO: Romanian legislative documents (1990–2020).
  2. Europarl: European Parliament debates translated into Romanian.
• Size: 350,000 parallel sentence pairs (correct vs. erroneous).
• Error Taxonomy: A custom 20-type error taxonomy was developed for Romanian, balancing complexity with standardization (inspired by ERRANT). Types include:
  • Part-of-Speech specific: ADJ, NOUN, VERB, etc.
  • Morphological: NOUN:INFL (inflection), VERB:SVA (subject-verb agreement), VERB:TENSE.
  • Structural: Word Order (WO), Orthography (ORTH), Punctuation (PUNCT).
  • Functional: Conjunctions (CONJ), Prepositions (PREP), Determiners (DET).
• Synthetic Generation Methods: To populate the dataset, four distinct generation strategies were employed:
  1. Noise Injection: Character and word-level substitutions, deletions, and insertions based on statistical probabilities derived from Wikipedia edit histories (calibrated to Romanian).
  2. Confusion Lists: Swapping function words (prepositions, conjunctions) with similar but incorrect alternatives.
  3. Zero-Shot LLM Prompting: Using GPT-4o to generate specific error types (e.g., adjective degrees) based on direct instructions.
  4. Two-Shot LLM Prompting: Using GPT-4o with two examples of common Romanian errors to guide the generation of more complex, context-aware errors (e.g., subject-verb disagreement). Note: Two-shot prompting was found to be more accurate but 11.37% slower than zero-shot.

B. Model Architectures Evaluated

The authors trained and evaluated models for three tasks: GED (token classification), GEC (sequence-to-sequence correction), and GEC-D (correction augmented with error tags).

• For GED (Detection):
  • DistilBERT Variants: Multilingual (mDistilBERT) and Romanian-specific (RoDistilBERT).
  • Sequence Tagging: BERT-based encoders (RoBERT-base/large) with softmax decoders, adapted from the GECToR approach.
• For GEC (Correction):
  • BART Variants: Multilingual (mBART), Romanian Base (RoBART-base), and Romanian Large (RoBART-large).
  • T5 Variants: Multilingual (mT5), Romanian Base (RoT5-base), and Romanian Large (RoT5-large).
  • GEC-D Strategy: Input sequences were augmented with error tags (e.g., <SEP> error_tags) to guide the correction model.

C. Decoding Strategies

• Top-p (Nucleus) Search: Samples from the top probability mass.
• Beam Search: Explores multiple hypotheses (beam width = 5).

3. Key Contributions

1. RoLegalGEC Dataset: The first synthetic parallel dataset for Romanian legal GED/GEC, containing 350k examples with a detailed 20-class error taxonomy.
2. Synthetic Generation Framework: A novel hybrid methodology combining statistical noise injection, confusion lists, and LLM prompting (zero-shot and two-shot) to generate realistic, domain-specific errors.
3. Comprehensive Evaluation: An extensive benchmark of knowledge-distilled, sequence-tagging, and generative Transformer models on legal Romanian text.
4. GEC-D Task Formulation: Introduction and evaluation of a correction task that utilizes error detection tags as input, analyzing its impact on model performance.
5. Taxonomy and Analysis: A detailed error-wise analysis revealing how model architecture, size, and pre-training language affect performance across different error types.

4. Results

Grammatical Error Detection (GED)

• Best Model: mDistilBERT achieved the highest overall F0.5 score (0.81), outperforming Romanian-specific models and sequence taggers.
• Insight: Multilingual models excelled at language-agnostic errors (spelling, punctuation, word order), while Romanian-specific models showed more stability in part-of-speech specific errors.
• Sequence Tagging: Larger models (SeqTag-large) significantly improved performance on infrequent but critical errors (e.g., PRON, PREP) compared to base models, but did not surpass DistilBERT on general errors.

Grammatical Error Correction (GEC)

• Best Model: RoT5-base paired with Beam Search in a GEC-D (tag-augmented) setting achieved the best results, surpassing the current State-of-the-Art for Romanian GEC.
• Architecture Comparison: T5 architectures consistently outperformed BART architectures on this legal dataset. BART struggled with the specific nuances of legal Romanian, often introducing new errors.
• Impact of GEC-D:
  • For strong models (T5 + Beam), adding error tags provided a modest but consistent performance boost.
  • For weak models (BART + Top-p), adding tags often degraded performance, confusing the model.
• Model Size:
  • T5: Scaling from Base to Large improved performance.
  • BART: Scaling from Base to Large degraded performance, suggesting the larger model overfit or struggled with the specific legal domain constraints.

Language Choice Impact

• Multilingual vs. Romanian: Multilingual models generally showed higher Precision (fewer false corrections) but lower Recall (missed more errors) compared to Romanian-specific models. For the best-performing T5 models, the Romanian variant was deemed more secure and stable for this specific domain.

5. Significance and Future Work

• Resource Enrichment: RoLegalGEC fills a critical gap in Romanian NLP, providing a benchmark for legal text processing that was previously unavailable.
• Practical Application: The findings suggest that for legal domains, T5-based models with Beam Search and error-tag augmentation are the most effective approach, outperforming standard BART-based pipelines.
• Methodological Insight: The study highlights that "bigger is not always better" (BART-large underperformed) and that the choice of pre-training language (Multilingual vs. Monolingual) involves a trade-off between precision and recall.
• Future Directions: The authors plan to integrate RoLegalGEC into the LiRo benchmark and further refine the error generation pipeline to reduce computational costs and improve the realism of synthetic errors.

In conclusion, this paper establishes a new standard for Romanian legal NLP, demonstrating that synthetic data generation combined with carefully selected Transformer architectures can effectively overcome data scarcity in low-resource, high-stakes domains.