🤖 machine learning

Neural Grammatical Error Correction for Romanian

This paper introduces the first Grammatical Error Correction (GEC) corpus for Romanian and demonstrates that pretraining a large Transformer model on synthetically generated data significantly improves performance in this low-resource setting.

Original authors: Teodor-Mihai Cotet, Stefan Ruseti, Mihai Dascalu

Published 2026-04-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Teodor-Mihai Cotet, Stefan Ruseti, Mihai Dascalu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a child how to speak and write perfectly in a language that isn't very common in the world of computers, like Romanian.

Most "smart" tools (like autocorrect or spellcheckers) are like teachers who have spent their whole lives in English-speaking schools. They are experts in English, but when it comes to Romanian, they are like tourists with a basic phrasebook—they can catch a simple spelling mistake, but they don't understand the "soul" or the complex grammar of the language.

This paper describes how a team of researchers from Bucharest built a "super-tutor" for Romanian grammar. Here is how they did it, broken down into three simple steps:

1. Building the "Gold Standard" Textbook (The Corpus)

To teach someone, you first need a great textbook. The researchers created the RONACC corpus.

Think of this as a collection of "Before and After" photos. They took real sentences from Romanian TV and radio shows—where people often speak naturally or make mistakes—and had experts correct them. This created a "Gold Corpus": 10,000 pairs of sentences where one is the "messy" version and the other is the "perfect" version. It’s the ultimate answer key for the student to study.

2. The "Fake News" Training Method (Synthetic Data)

The problem is that 10,000 sentences isn't enough to make a computer truly "smart." It’s like trying to learn a whole language by reading only one book.

To fix this, the researchers used a clever trick: they played "Mad Libs" with Wikipedia. They took millions of perfect sentences from Wikipedia and intentionally "broke" them. They swapped words, deleted letters, and messed up the order to simulate human errors.

The Analogy: Imagine you want to teach a professional athlete how to react to being tripped. Instead of waiting for them to actually fall in a real game (which is rare), you set up a training course with padded obstacles and fake trips. By practicing on these "fake" mistakes, the athlete becomes incredibly fast at recognizing and recovering from real mistakes when the actual game starts.

3. The "Brain" (The Transformer Model)

The researchers used a piece of technology called a Transformer. Think of the Transformer as a highly attentive student. When it looks at a sentence, it doesn't just look at one word at a time; it looks at the entire sentence at once, noticing how every word relates to every other word.

They tested two types of "brains":

The Tiny Brain: A small, quick student who only studied the "Gold Textbook."
The Big Brain: A much larger, more powerful student who first studied the "Fake News" (the broken Wikipedia sentences) and then studied the "Gold Textbook."

The Result: Who won?

The Big Brain won by a landslide! By practicing on the "fake" mistakes first, it developed a "gut feeling" for how language should flow. When it finally sat down to study the real Romanian textbook, it was already an expert at spotting patterns.

Why does this matter?

Most AI progress happens in English because English has massive amounts of data. This paper provides a "blueprint" for other languages. It shows that even if you don't have a mountain of perfect data, you can use "fake" data and smart training tricks to build a powerful tool that helps people communicate more clearly in their own native tongue.

Technical Summary: Neural Grammatical Error Correction for Romanian

Problem Statement

Grammatical Error Correction (GEC) is a challenging task in Natural Language Processing (NLP) that involves correcting spelling, punctuation, word choice, and grammatical errors. While significant progress has been made for English, resources for non-English languages—particularly low-resource ones like Romanian—are scarce. Existing Romanian tools are largely limited to simple, rule-based spellcheckers, lacking the ability to handle complex linguistic nuances or contextual errors.

Key Contributions

The authors address the resource gap through three primary contributions:

RONACC Corpus: The introduction of the first Romanian GEC corpus, consisting of approximately 10,000 sentence pairs. It is categorized into three groups: NAC-phrases (spoken phrases without verbs), NAC-sentences (spoken sentences), and W-sentences (written sentences with common mistakes).
Adapted ERRANT Toolkit: The adaptation of the German version of the ERRANT (ERRor ANnotation Toolkit) to Romanian. This allows for the automated extraction and classification of edits using Universal Dependencies (UD) POS tags.
Synthetic Data Generation Pipeline: A methodology for augmenting low-resource training by generating 10 million synthetic error-prone sentences from Wikipedia, which can be applied to any language using only a POS tagger.

Methodology

The researchers focused on a neural approach using the Transformer architecture, experimenting with two main configurations:

Transformer-tiny: A small model (~1.5M parameters) trained exclusively on the gold (RONACC) corpus.
Transformer-base: A larger model (~94M parameters) that utilizes a two-stage training process:
- Pretraining: Training on 10 million artificially generated sentences. Errors were introduced via word substitution (using a spellchecker's confusion set), deletion, insertion, and swapping, as well as character-level perturbations.
- Finetuning/Retraining: The model was then adapted to the gold corpus using either finetuning (continuing training with preserved parameters) or retraining (resetting optimizer parameters).

Decoding and Evaluation:

Decoding: Used beam search (size 8) combined with a 5-gram Kneser-Ney smoothed language model for re-ranking.
Evaluation: Performance was measured using the $F_{0.5}$ score, which prioritizes precision over recall. This is critical for GEC, as users prefer a system that rarely suggests incorrect changes over one that misses many errors.

Results

The experimental results demonstrate the effectiveness of pretraining in low-resource settings:

Baseline Performance: The Transformer-tiny model achieved an $F_{0.5}$ of 44.38.
Best Performance: The Transformer-base model using the finetuning strategy achieved an $F_{0.5}$ of 53.76, representing a significant improvement over the baseline.
Group Analysis: The model performed best on W-sentences ( $F_{0.5} = 62.7$ ), likely due to the linguistic similarity between Wikipedia-based synthetic data and written mistakes. It performed least effectively on NAC-sentences, where errors are often non-idiomatic or involve informal spoken language.
Error Handling: The model showed strength in correcting spelling and specific Romanian morphological issues (like noun articulation), but struggled with highly informal spoken language and complex idiomatic substitutions.

Significance

This paper provides a foundational framework for Romanian NLP. By releasing the RONACC corpus, the adapted ERRANT toolkit, and the synthetic data generation method under an Apache 2 license, the authors have enabled future researchers to develop more sophisticated GEC systems for Romanian. The study proves that even with limited "gold" data, large-scale pretraining on synthetically corrupted data is a highly effective strategy for improving neural error correction in low-resource languages.