Improving Estonian Text Simplification through Pretrained Language Models and Custom Datasets

Technical Summary: Improving Estonian Text Simplification Through Pretrained Language Models and Custom Datasets

Problem Statement
Automatic Text Simplification (ATS) aims to convert complex text into simpler forms while preserving meaning, aiding language learners, individuals with cognitive impairments, and low-literacy readers. While ATS research is mature for English, utilizing rule-based systems, neural models, and large-scale datasets, low-resource languages like Estonian remain underserved. Existing Estonian ATS efforts have been limited to small-scale undergraduate theses focusing on lexical substitution or template-driven syntactic rules, lacking large-scale, publicly available datasets. Furthermore, reliance on proprietary Large Language Models (LLMs) like GPT-4.0 raises concerns regarding transparency, cost, and reproducibility, particularly for languages where commercial APIs may not be optimized.

Methodology
The authors propose a two-pronged approach to Estonian ATS: comparing a Neural Machine Translation (NMT) baseline against a fine-tuned Large Language Model (LLM).

1. Dataset Construction (Estonian Simplification Dataset):
   To address the scarcity of training data, the authors constructed a new dataset of 50,416 sentence pairs by combining three sources:

   • Translated Corpora: Manually translated portions of the TurkCorpus (2,359 English sentences with 8 simplifications each) and the Wikipedia Data Set 2.0 (167k English pairs). Only pairs where the simplification relationship remained clear after translation were retained.
   • GPT-4.0 Generation: Leveraging GPT-4.0 to generate a large volume of simplifications from Estonian Wikipedia articles. The authors employed "agent-style" prompting, where the model was instructed to act as a specific "lexical simplification assistant" or "syntactic restructuring assistant." This sequential prompting strategy (lexical first, then syntactic) proved more effective than generic prompting.
   • Human Validation: Native Estonian annotators corrected a subset of the GPT-4.0 outputs to ensure linguistic consistency and semantic coherence.
     The final dataset includes specific guidelines adapted to Estonian morphosyntax, focusing on grammatical complexity (inflection, tense), syntactic complexity (clause splitting), and lexical simplification (synonym replacement).

2. Model Architectures:

   • Baseline (OpenNMT): A neural machine translation model based on the OpenNMT-py framework. It treats simplification as a monolingual translation task (complex Estonian to simple Estonian) using an encoder-decoder architecture with attention mechanisms. It was trained on the constructed dataset.
   • Proposed Model (Fine-tuned LLaMA): The authors fine-tuned LLaMA 3.1 (8 billion parameters) on the same dataset. They utilized the Unsloth library to optimize the process via Low-Rank Adaptation (LoRA) and QLoRA, reducing memory consumption and computational costs. The model was trained to generate simplified sentences while preserving meaning.

3. Evaluation Protocol:
   The study employed a dual evaluation strategy on a held-out test set of 100 complex sentences (manually simplified by native linguists to form a gold standard):

   • Automatic Metrics: BLEU (n-gram overlap), SARI (simplification-specific metric rewarding additions, deletions, and retention), and FKGL (Flesch-Kincaid Grade Level for readability).
   • Human Evaluation: Two native Estonian linguists rated 50 sentence pairs from each model on a 0–4 scale across four dimensions: Grammaticality, Readability, Preservation of Meaning, and Reduction Effort.

Key Contributions

• Dataset Release: The creation and public release of the first large-scale Estonian Simplification Dataset, combining translated, GPT-4.0-generated, and manually validated pairs.
• Architectural Comparison: A head-to-head benchmark of OpenNMT (NMT approach) versus fine-tuned LLaMA for Estonian ATS.
• Methodological Framework: A reproducible pipeline for low-resource language simplification using open-access LLMs, lightweight fine-tuning (Unsloth/LoRA), and persona-based prompting strategies.
• Empirical Evidence: Demonstration that fine-tuned LLMs outperform traditional NMT baselines in low-resource settings when evaluated by human raters.

Results

• Automatic Metrics: The results were mixed. OpenNMT achieved a higher BLEU score (30.05 vs. 27.04), indicating closer n-gram overlap with reference texts. However, LLaMA 3.1 achieved a higher SARI score (49.72 vs. 47.43), suggesting more effective simplification edits (appropriate deletions and additions). LLaMA also showed a slight edge in readability (lower FKGL score: 8.71 vs. 9.02).
• Human Evaluation: LLaMA 3.1 significantly outperformed OpenNMT across all human-rated dimensions. The most notable gap was in Preservation of Meaning (3.24 for LLaMA vs. 1.76 for OpenNMT). LLaMA also scored higher in Grammaticality (3.46 vs. 2.26) and Readability (3.26 vs. 2.04). OpenNMT struggled with semantic integrity and fluency, often failing to apply effective reduction strategies.

Significance and Claims
The paper claims that while standard automatic metrics (BLEU/SARI) provide useful benchmarks, they are insufficient for capturing critical qualitative aspects of simplification, such as semantic preservation and nuance. The study underscores that fine-tuned open-access LLMs (LLaMA) are highly effective for text simplification in low-resource languages, provided a custom dataset is available.

The authors emphasize that their approach is not limited to Estonian. By using openly available models, lightweight fine-tuning frameworks, and adaptable data collection strategies (including persona-based prompting), the methodology can be transferred to other low-resource languages with similar morphological or syntactic complexities. The work aims to support practical applications, such as educational tools and accessibility for Estonian speakers with cognitive disabilities, and provides a reproducible foundation for future document-level simplification research.