Transferring Natural Language Datasets Between Languages Using Large Language Models for Modern Decision Support and Sci-Tech Analytical Systems

Technical Summary: Transferring Natural Language Datasets Between Languages Using Large Language Models

Problem Statement
The paper addresses the scarcity of large-scale, annotated Russian corpora for Natural Language Processing (NLP) tasks, specifically regarding the extraction of terms and their definitions. While the English DEFT corpus (Definition Extraction from Texts) provides a rich, three-layer annotation structure for term-definition pair mining, equivalent resources are unavailable for Russian. Manually creating such datasets from scratch is labor-intensive and costly. The authors investigate whether Large Language Models (LLMs) can effectively transfer existing English datasets and their Named Entity Recognition (NER) annotations to Russian, thereby providing a "silver" quality baseline for training models in under-resourced languages.

Methodology
The authors propose a pipeline consisting of three main stages: preparing a gold set, translating text, and transferring NER annotations.

1. Data Preparation: The study utilizes the DEFT corpus (English texts from free e-books) and the Wikipedia portion of the WCL dataset. The original CoNLL-like format was converted to a Hugging Face Datasets structure. A "gold set" was manually created by translating a subset of the English data (dev set and a random sample of the train set) into Russian using Google Translate, followed by manual verification and NER annotation transfer using Label Studio. This resulted in 1,070 manually verified sentences (870 with NER, 200 without).
2. Annotation Transfer Strategy: The authors initially tested a prompt requiring LLMs to output index triplets (start, end, label) based on English indices. This approach failed due to the models' inability to accurately map indices across languages. They reformulated the task: the LLM is asked to identify the corresponding Russian substring for a given English NER span directly.
3. Model Benchmarking: A diverse set of LLMs was evaluated via the bothub.chat proxy, including OpenAI models (gpt-3.5-turbo, gpt-4o-mini, gpt-4.1-nano, gpt-4.1-mini), Llama3.1-8B, DeepSeek variants (chat-v3-0324, r1), and Qwen-2.5-72B.
4. Evaluation Metrics:
   • Annotation Transfer: Measured by exact matches, wider/narrower partial matches, and mismatches between the LLM output and the gold standard. Fuzzy matching was applied as a post-processing step to mitigate morphological mismatches.
   • Translation Quality: Evaluated using BLEU scores and embedding-based metrics (LaBSE) including "Parallel Comparison" (semantic distance between source and translation) and "BLEU-like" (semantic distance between gold and machine translation).
5. Downstream Training: The best-performing pipeline (using DeepSeek-chat-v3-0324) was applied to the full DEFT and WCL datasets to generate a translated Russian corpus (ruDEFT and WCL_Wiki_Ru). BERT-based models (multilingual BERT, RuBERT, and RoBERTa) were trained on these datasets to establish baselines for definition detection and term/definition recognition.

Key Results

• Annotation Transfer: Reformulating the task to extract substrings rather than indices significantly improved performance. The DeepSeek-chat-v3-0324 model achieved the highest performance with a 94.21% exact match rate and only 0.84% mismatched spans (with fuzzy search post-processing). GPT-4.1-mini and GPT-3.5-turbo also performed well, while Llama3.1-8B struggled more with morphological variations, requiring fuzzy matching to reduce mismatches.
• Translation Quality: Translation performance was generally high, with gpt-4o-mini and DeepSeek-chat-v3-0324 achieving the best BLEU scores (~0.52) and favorable embedding-based metrics.
• Downstream Model Performance:
  • Definition Detection (Task 1): Models trained on the transferred data achieved strong results (F1 ~0.85 on test sets), though a performance gap existed between the gold set and the transferred test set, suggesting LLM-induced biases in the text.
  • NER (Task 2): Performance was notably weaker (F1 ~0.58 for RuDEFT), indicating that the "silver" quality of the transferred annotations limits the model's ability to generalize perfectly. However, models performed better on the gold set than the transferred test set, and multilingual BERT showed slightly better robustness to annotation errors than RuBERT in this specific context.
  • WCL Dataset: Models achieved excellent results (F1 ~0.96) on the Wikipedia-based dataset, likely due to the structured nature of Wikipedia definitions and the simpler task of identifying only term entities.

Significance and Claims
The authors claim that LLMs can serve as effective "smart translators" for transferring complex NER annotations across languages, significantly reducing the effort required to create starting points for NLP in under-resourced languages. While the resulting datasets are of "silver" quality and require human verification for high-stakes applications, the study demonstrates that they are sufficient to train baseline models for scientific trend analysis and decision support systems (e.g., iFORA, SciApp).

The paper modestly concludes that while the transferred data enables the creation of functional tools, the NER models trained on this data are not yet perfect, necessitating careful verification of annotations. The work highlights that the cognitive complexity of the task (e.g., mapping indices vs. extracting substrings) significantly impacts LLM performance. The authors have open-sourced the generated Russian datasets (ruDEFT, WCL_Wiki_Ru) and the associated code to facilitate further research.