RILEC: Detection and Generation of L1 Russian Interference Errors in English Learner Texts

This paper introduces RILEC, a large-scale dataset and a generative framework for detecting and creating L1 Russian interference errors in English learner texts, demonstrating that models fine-tuned on this augmented data significantly improve the identification of specific error types like transliteration and tense misuse.

The Gist

Imagine you are trying to learn to play the piano, but your brain is secretly trying to play the violin at the same time. Every time you press a piano key, your muscle memory from the violin makes your finger slip to the wrong note. In the world of language learning, this is called L1 Interference. It's when your native language (like Russian) "hacks" your second language (English), causing you to make specific, predictable mistakes.

For example, a Russian speaker might write "I will have enough time" instead of "If we have enough time," because in Russian, the grammar for "if" and "will" works differently. Or they might write "cassa" instead of "cashier" because they are literally spelling the Russian word with English letters.

This paper introduces a new tool called RILEC to help teachers and students catch these specific "violin slips" in English essays. Here is how they built it, explained simply:

1. The Problem: Not Enough "Mistake" Data

To teach a computer to spot these specific Russian-style errors, you need a massive library of examples. But real student essays are hard to find, and even harder to find where someone has carefully labeled why the mistake happened. It's like trying to teach a doctor to diagnose a rare disease when you only have five patient files.

2. The Solution: The "Mistake Factory" (RILEC)

The authors built RILEC (Russian L1 Interference Learner English Corpus). Think of this as a giant, high-tech "Mistake Factory."

They started with a real collection of essays from Russian students (about 6,000 sentences). But they knew that wasn't enough to train a super-smart AI. So, they invented three different ways to manufacture new, realistic mistakes to fill the gaps:

• The "Robot Coach" (PPO-Optimized Models): Imagine a robot that has read thousands of essays. They trained this robot using a special technique called PPO (think of it as a video game reward system). Every time the robot made a mistake that looked like a real Russian learner's error, it got a "gold star." If it made a normal mistake, it got nothing. Over time, the robot learned to generate thousands of new sentences that sound exactly like a Russian learner struggling with English.
• The "Rule Book" (Rule-Based): For some very specific errors (like mixing up tenses or spelling Russian words with English letters), they wrote a strict set of instructions. It's like a mad-libs game: "Take this sentence, find the year, and change the verb to the wrong tense." This ensures they get plenty of examples for the tricky, rule-heavy errors.
• The "Creative Writer" (LLM Prompting): They asked a very smart AI (like a creative writing assistant) to look at a real mistake and say, "Okay, make up a new story that uses this exact same mistake." This helped them generate errors that were more natural and varied.

By combining these three methods, they expanded their dataset from 6,000 sentences to 18,000+ sentences. It's like turning a small seed into a massive forest of examples.

3. The Result: A Super-Spotter

Once they had this massive library of "Mistake Factory" data, they trained a new AI model to be a Super-Spotter.

• Before: If you showed a computer a student essay, it might say, "This sentence is wrong," but it wouldn't know why. It's like a teacher saying, "You got this wrong," without explaining the rule.
• After: The new model, trained on RILEC, can say, "You used the wrong tense because you are thinking in Russian," or "You spelled 'cashier' as 'cassa' because of transliteration."

4. Why This Matters

The paper found that this "Mistake Factory" approach worked incredibly well.

• Accuracy: The model got very good at spotting specific errors like Transliteration (spelling Russian words in English) and Tense Semantics (using the wrong time tense), scoring over 90% accuracy on those.
• The "Human" Touch: Even though the data was made by machines, the mistakes felt real. The model learned the style and logic of a Russian learner, not just random typos.

The Big Picture

Think of this research as building a specialized translator for errors. Instead of just fixing the grammar, it translates the student's brain back to the teacher. It explains, "Ah, I see what happened! You were thinking in Russian, and your brain translated 'cassa' directly."

This helps teachers give better feedback and helps students understand why they are making mistakes, rather than just being told they are wrong. It turns the frustrating process of learning a new language into a clearer, more logical journey.

Technical Summary

Here is a detailed technical summary of the paper "RILEC: Detection and Generation of L1 Russian Interference Errors in English Learner Texts."

1. Problem Statement

The paper addresses the challenge of L1 Interference Error Detection in English texts written by Russian speakers. While general Grammatical Error Detection (GED) and Correction (GEC) systems exist, they often fail to identify the root causes of errors, which are frequently driven by the transfer of native language (Russian) rules to English.

• The Gap: Existing tools detect ungrammaticality but do not explain why an error occurred (e.g., distinguishing between a random typo and a systematic transliteration or tense confusion caused by Russian grammar).
• Data Scarcity: There is a lack of large-scale, annotated datasets specifically targeting L1-motivated errors with fine-grained tagging of interference types. Most existing learner corpora (like FCE or CoNLL-2014) are either mixed-L1 or lack specific interference annotations.

2. Methodology

The authors propose a comprehensive framework involving dataset construction, synthetic data augmentation, and model training.

A. The RILEC Dataset

The core contribution is RILEC (Russian L1 Interference Learner English Corpus), a dataset of 18,830 sentences. It combines:

1. Real Data: A subset of the REALEC corpus (Russian Error-Annotated Learner of English Corpus), containing 6,086 sentences manually annotated for interference.
2. Synthetic Data: Generated via three distinct augmentation pipelines to expand the dataset size and diversity.

Annotation Scheme (5 Tags):
Based on Weinreich's (1979) framework, errors are categorized into:

1. Copying Expression: Word-for-word translation of Russian collocations (e.g., "every of us" instead of "every one of us").
2. Synonyms: Confusion between English words that map to a single Russian polysemous word (e.g., using overcome instead of cover because both map to Russian preodolet').
3. Tense Semantics: Incorrect tense usage due to Russian grammatical flexibility (e.g., using Present Simple for past graphical data).
4. Transliteration: Writing Russian words using the English alphabet (e.g., cassa instead of cashier).
5. Word Form Transmission: Transferring Russian grammatical categories (e.g., pluralization) to English (e.g., "$5 billions").

B. Data Augmentation Strategies

To overcome the scarcity of real annotated errors, the authors employed three generation methods:

1. PPO-Optimized Generation (Model-Based):

   • Mechanism: Fine-tuned DistilGPT2 models were optimized using Proximal Policy Optimization (PPO).
   • Reward: Separate binary classifiers (based on RoBERTa) were trained for each error type to act as reward models. The generator received a positive reward if it produced the target error type.
   • Outcome: Successfully generated high-quality examples for Copying Expression, Synonyms, and Word Form Transmission. Failed to reliably generate Tense Semantics and Transliteration errors.

2. Rule-Based Generation:

   • Mechanism: Used to supplement the categories where PPO failed.
   • Tense Semantics: Replaced verbs in sentences containing years with Present Simple forms.
   • Transliteration: Replaced nouns with their transliterated versions using the Google Translate API.
   • GECToR Adaptation: Used a dictionary-based approach to inject replacement errors based on known error-correction pairs.

3. Prompt-Based LLM Generation:

   • Mechanism: Prompted large language models (Claude 2, GPT-3.5, Mistral) with examples of specific error types to generate new contexts.
   • Selection: Claude 2 was selected for generation (highest success rate in producing the target error), while Mistral was selected for annotating the generated data (highest accuracy in identifying error spans).

C. Model Training & Evaluation

• Task: Multi-span classification (identifying error spans and their specific L1 interference type).
• Model: RoBERTa-base fine-tuned on the various dataset splits.
• Evaluation: Tested on the REALEC-L1 test set and a manually curated "Test100" set. Metrics included F1-score, accuracy, and manual expert review.

3. Key Contributions

1. RILEC Corpus: The first large-scale dataset (18k+ sentences) specifically for L1-interference error detection in Russian-English learner texts, featuring a 5-tag annotation scheme.
2. Hybrid Augmentation Framework: A novel pipeline combining PPO-optimized small models, rule-based injection, and prompt-based LLM generation to create diverse, high-quality synthetic training data.
3. LLM Evaluation for Augmentation: A systematic analysis of LLM capabilities in generating erroneous text (finding that Claude 2 is best for generation, while Mistral is best for annotation).
4. Performance Benchmarking: Demonstrated that synthetic data significantly improves GED performance, particularly for specific interference types.

4. Results

• Overall Performance: The model trained on the full RILEC dataset achieved an average F1-score of 73.95%, significantly outperforming the baseline (trained only on real data) which scored 55.66%.
• Category-Specific Performance:
  • High Performance (>90% F1): Transliteration and Word Form Transmission. These are structurally distinct and easier for models to learn.
  • Moderate Performance: Tense Semantics (~80% F1).
  • Lower Performance: Copying Expression (33.33%) and Synonyms (69.39%). These require deep semantic understanding of L1-L2 mapping, which remains challenging.
• Augmentation Impact:
  • PPO-based data yielded the second-best results (71.81% avg F1), proving effective for semantic errors.
  • Rule-based data was larger but less effective for complex semantic errors, though it excelled at structural injections.
  • LLM-based data showed strong generalization despite smaller volume.
• Manual Evaluation: On real learner texts (REALEC70), the RILEC-trained model identified 19 true positives compared to the baseline's 6, confirming its practical utility for teachers and learners.

5. Significance and Future Work

• Educational Impact: The system moves beyond simple "error detection" to "error explanation." By identifying the cause (e.g., "This is a transliteration error"), teachers can provide targeted feedback, and learners can understand their specific L1 interference patterns.
• Methodological Insight: The study highlights that while LLMs are powerful, they struggle to generate specific types of ungrammatical errors (like tense shifts) without rule-based constraints or PPO optimization.
• Limitations:
  • Currently limited to Russian speakers; future work aims to expand to other L1s (e.g., Chinese, Spanish) using resources like ICNALE.
  • The fixed 5-tag scheme may miss nuanced linguistic phenomena.
  • Ethical considerations regarding the potential misuse of synthetic learner data are acknowledged.

In conclusion, RILEC establishes a new standard for L1-interference research, demonstrating that combining expert-annotated data with sophisticated synthetic augmentation (PPO + LLMs + Rules) creates robust models capable of diagnosing the specific linguistic roots of learner errors.