OasisSimp: An Open-source Asian-English Sentence Simplification Dataset

1. Problem Statement

Sentence simplification (SS) aims to reduce linguistic complexity while preserving original meaning, aiding language learners, individuals with cognitive disabilities, and younger audiences. Despite significant progress in high-resource languages (e.g., English, Spanish), there is a critical resource gap for mid-resource and low-resource languages.

• Data Scarcity: Existing SS datasets are heavily skewed toward English (e.g., Newsela, WikiLarge). Languages like Thai, Pashto, and Tamil lack dedicated SS datasets, while Sinhala has only limited data.
• Evaluation Bias: Current Large Language Models (LLMs) are often evaluated primarily on high-resource languages, masking their performance disparities in low-resource settings.
• Methodological Gaps: Traditional supervised learning is impractical for low-resource languages due to a lack of parallel training data. While LLMs offer zero-shot/few-shot potential, their effectiveness in multilingual SS remains under-explored.

2. Methodology

A. Dataset Creation (OasisSimp)

The authors introduced OasisSimp, a multilingual dataset covering five languages: English, Sinhala, Tamil, Thai, and Pashto.

• Language Selection: Chosen for typological diversity (Indo-European, Dravidian, Tai-Kadai, Indo-Aryan, Eastern Iranian) and varying resource levels.
• Data Sources:
  • English: NewsEdits (The Globe and Mail), filtered for syntactic complexity.
  • Sinhala & Tamil: SiTa trilingual parallel corpus (government documents), filtered for rare words and technical terms.
  • Thai: ThaiSum news corpus, filtered for length and topic diversity.
  • Pashto: Wikipedia articles, filtered for semantic categories and complexity.
• Annotation Process:
  • Human-in-the-Loop: 3–6 native speakers per language with Bachelor's degrees or higher.
  • Guidelines: Annotators followed strict protocols to simplify sentences via Rewording (replacing technical terms), Sentence Splitting, Deletion (removing redundancy), and Reordering.
  • Quality Control: Structured training, iterative verification, and manual review to ensure fluency, grammatical correctness, and meaning preservation.
• Statistics: The dataset contains between 520 (Tamil) and 2,500 (English, Sinhala, Pashto) complex sentences, with an average of 3–5 simplified references per complex sentence.

B. Evaluation Framework

The authors evaluated eight open-weight multilingual LLMs (Aya, Cmd-R, DeepSeek, EuroLLM, Gemma, LLaMA, Mistral, Qwen) using the OasisSimp dataset.

• Settings: Experiments were conducted in Zero-shot and Few-shot (1-shot and 5-shot) configurations.
• Prompts: Models were instructed to "Simplify the given sentence..." following the BLESS benchmark strategy.
• Metrics:
  • SARI: Measures the quality of words added, kept, and deleted.
  • BERTScore: Measures semantic similarity between the generated simplification and reference sentences.
  • BLEU: Included in the appendix for completeness.

3. Key Contributions

1. OasisSimp Dataset: The first publicly available, human-annotated sentence simplification dataset for Thai, Pashto, and Tamil, and a significant expansion for Sinhala. It addresses the lack of parallel complex-simple sentence pairs in these languages.
2. Comprehensive Benchmarking: A systematic evaluation of eight diverse LLMs across five typologically distinct languages, providing a baseline for future research in low-resource SS.
3. Analysis of LLM Capabilities: Detailed insights into how LLMs handle sentence simplification in low-resource settings, specifically analyzing the trade-offs between meaning retention (KEEP), deletion (DEL), and addition (ADD).

4. Results and Findings

Performance Disparities

• High-Resource vs. Low-Resource: Models performed consistently well on English (high-resource) but showed significant variance on low-resource languages.
• Zero-Shot Limitations: Zero-shot performance was generally poor for low-resource languages, with some models (e.g., DeepSeek on Tamil, LLaMA on Pashto) producing negative SARI scores or failing to simplify effectively.
• Few-Shot Improvements: Providing 1 or 5 examples significantly improved performance across all languages. For instance, Gemma's SARI score on Pashto improved from 33.3 (0-shot) to 37.9 (5-shot).

Model-Specific Insights

• Gemma: Emerged as the most robust model, performing consistently well across all languages, including those not explicitly confirmed in its training data (e.g., Pashto). It demonstrated strong cross-lingual transfer capabilities.
• Language Inclusion: Models like LLaMA (which includes Tamil and Thai in training) showed better performance on those specific languages compared to models without explicit inclusion, though the lack of transparency in training data limits definitive conclusions.
• Component Analysis (SARI):
  • DELETE (DEL): Models were most successful at removing unnecessary information (high DEL scores).
  • KEEP: Improved significantly with few-shot prompting, indicating better structural alignment when examples are provided.
  • ADD: The most challenging component. Models struggled to generate new simple words or helpful information, particularly in low-resource languages (ADD scores often < 1 for non-Gemma models).

5. Significance and Impact

• Resource Democratization: OasisSimp provides a critical infrastructure for developing and evaluating SS systems for Asian languages, moving beyond the English-centric paradigm.
• Inclusive AI: By highlighting the limitations of current LLMs in low-resource settings, the paper underscores the need for better training data and specialized architectures to ensure equitable access to simplified information for diverse global audiences.
• Future Research: The dataset serves as a challenging benchmark to drive research into few-shot learning, cross-lingual transfer, and morphologically complex language simplification.

6. Limitations

• Dataset Size: Limited by annotator availability and funding (e.g., Tamil has only 520 pairs).
• Training Data Transparency: The exact composition of LLM training data is often opaque, making it difficult to definitively correlate language inclusion with performance.
• Human Evaluation: Due to resource constraints, the LLM outputs were not evaluated by human judges, relying solely on automatic metrics.

Conclusion: OasisSimp represents a significant step toward inclusive NLP, providing both the data and the benchmark necessary to advance sentence simplification for under-resourced languages. The results confirm that while LLMs show promise, they require few-shot guidance and further optimization to handle the nuances of low-resource sentence simplification effectively.