💻 computer science

A Semantic-Enhanced Multi-Task Framework with Structure-Aware Curriculum Learning for Low-Resource Neural Machine Translation

This paper proposes a semantic-enhanced multi-task framework with structure-aware curriculum learning, leveraging Qwen2-7B-Instruct and generative Semantic Role Labeling to significantly improve low-resource neural machine translation performance by resolving semantic misalignment and optimizing training dynamics.

Original authors: Jingxing Gao, Liqing Wang, Xingwei Chen, Yongyue Xu

Published 2026-09-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jingxing Gao, Liqing Wang, Xingwei Chen, Yongyue Xu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Language is a vast, shifting landscape, and for computers, some parts of it are vast deserts. In the field of machine translation, where software learns to convert text from one language to another, there is a persistent problem with "low-resource" languages. These are languages like Lao or Myanmar, which have very little digital text available for computers to study. When researchers try to teach powerful language models to translate these languages, the machines often stumble. They might invent words that don't exist or, more subtly, they might get the relationships between words wrong. A sentence might say that a government "strengthened" a regulation, but the computer, confused by the lack of examples, might drop the action entirely and just list the nouns, leaving the meaning hollow. This happens because the models tend to memorize simple word pairings rather than understanding the deeper logic of who did what to whom.

To solve this, researchers have tried two main approaches. One is to teach the computer about the structure of sentences, specifically how actions (predicates) connect to the people or things involved (arguments). Another approach is to mimic how humans learn: starting with simple examples and gradually moving to harder ones, a method known as curriculum learning. However, previous attempts to combine these ideas often failed because the computer's learning process became confused, trying to learn too many different things at once and getting stuck. A new study from Yunnan University proposes a way to fix this by teaching the computer to understand the "skeleton" of a sentence while guiding it through a carefully ordered path of learning.

The researchers built a system based on a large language model, a type of artificial intelligence that has already learned a great deal about human language. They did not try to retrain the entire massive model, which would be too expensive and slow. Instead, they added a small, flexible layer of instructions that could be adjusted easily. This layer was tasked with learning three things at the same time: translating Chinese into Lao, recognizing important names like people and places, and identifying the roles words play in a sentence, such as who is the doer and who is the receiver of an action. By forcing the computer to generate these structural details alongside the translation, the system was compelled to pay attention to the deep logic of the sentence rather than just surface-level word matches.

The key innovation, however, was how they fed the data to the computer. Instead of throwing all the training examples at the model at once, the researchers created a "difficulty score" for every sentence. This score was not based on how long the sentence was, which can be misleading, but on how complex its internal structure was. They looked at how many actions were happening, how many participants were involved, and how far apart the related words were. The computer then started training only on the simplest sentences. As it got better, the system gradually introduced more complex sentences, expanding the pool of examples step by step. This ensured the model mastered the basics of the language's structure before being asked to handle the most tangled and difficult phrases.

The results of this approach were striking. When tested on translating Chinese into Lao, the new method improved the quality of the translation significantly compared to standard training methods. The researchers measured this improvement using a standard scoring system, finding that their method boosted the score by more than five points, a substantial jump in this field. It also outperformed other large, pre-existing models that had been trained on massive amounts of data from many different languages. The system did not just learn to translate better; it learned to translate more accurately, preserving the specific relationships between words that were often lost before. For instance, in sentences where a government "strengthened" a rule, the new system correctly kept the action, whereas older methods often dropped it.

To ensure this was not just a lucky break with one specific language pair, the researchers tested the same system on translating Chinese into Myanmar. These two languages, Lao and Myanmar, share similar grammatical traits, relying heavily on word order rather than changing word endings to show meaning. The system performed just as well on Myanmar, improving the translation score by nearly two points over the baseline. This suggests that the method of teaching structural logic through a graduated curriculum is not a trick that works only for one language, but a robust way to help computers understand languages that lack large digital libraries.

The study also looked at how the computer learned over time. They found that when the system was first introduced to the complex structural tasks, its performance dipped slightly, a "cold start" where the model struggled to juggle the new requirements. However, once it passed this initial hurdle, its performance surged, eventually surpassing models that were trained without the structural guidance. The curriculum learning method smoothed out this rough start, helping the model avoid getting stuck in a plateau where it stops improving. By the end of training, the system had developed a stable ability to handle the intricate logic of low-resource languages.

This work demonstrates that for languages with little digital data, the answer may not be to simply gather more text, but to teach the computer how to think about the text it already has. By combining a deep understanding of sentence structure with a patient, step-by-step learning schedule, the researchers have shown a path to more reliable machine translation. The findings suggest that even in the absence of massive datasets, explicit guidance on how sentences are built can help artificial intelligence bridge the gap between languages that have long been left behind.

Technical Summary: A Semantic-Enhanced Multi-Task Framework with Structure-Aware Curriculum Learning for Low-Resource Neural Machine Translation

1. Problem Statement

Low-resource Neural Machine Translation (LR-NMT) faces persistent challenges when adapting Large Language Models (LLMs), primarily due to semantic misalignment and optimization interference. In data-scarce regimes, LLMs tend to overfit to shallow lexical co-occurrences rather than learning deep predicate-argument dependencies, leading to argument misalignment and the omission of critical semantic content. Furthermore, standard Multi-Task Learning (MTL) often induces negative transfer and convergence instability because competing objectives interfere with one another during fine-tuning.

Existing solutions are insufficient:

Semantic Role Labeling (SRL) is typically implemented via static graph-based encoders, which are architecturally incompatible with the autoregressive, decoder-only paradigm of modern LLMs.
Curriculum Learning (CL) often relies on surface-level heuristics (e.g., sequence length, token frequency) that fail to capture genuine structural complexity, particularly in isolating languages like Lao and Myanmar where grammatical relations depend on word order rather than morphological inflection.

2. Methodology

The authors propose a Semantic-Enhanced Multi-Task Framework built on the Qwen2-7B-Instruct model, utilizing Parameter-Efficient Fine-Tuning (LoRA). The framework integrates deep semantic modeling with dynamic curriculum scheduling within a unified adapter space.

2.1 Semantic-Enhanced Multi-Task Learning

The core innovation is reformulating Semantic Role Labeling (SRL) as a generative auxiliary task. Instead of using external graph encoders, the model is trained to generate linearized semantic sequences (predicate-argument structures) alongside the primary translation task. The framework employs four parallel generative tasks:

Primary Translation (Zh-Lo): Chinese-to-Lao translation.
Cross-Lingual Transfer (Zh-Th): Chinese-to-Thai translation, leveraging morphosyntactic overlap within the Kra-Dai language family to aid the shared adapter space.
Named Entity Recognition (NER): A generative sequence labeling task to ground entities (Person, Organization, Location) and mitigate out-of-vocabulary hallucinations.
Generative SRL: The model predicts predicate-argument structures (e.g., A0–A5, AM-TMP) extracted via the LTP parser, forcing the LLM to internalize deep logical dependencies.

The total loss function is defined as:
$L_{total} = L_{NMT} + L_{Thai} + L_{NER} + \lambda L_{SRL}$
Where $\lambda$ is a balancing coefficient (optimized at 0.5).

2.2 Semantic-Driven Dynamic Curriculum Learning

To mitigate optimization interference, the authors introduce a structural difficulty metric ( $D_s$ ) that replaces surface-level heuristics. This metric quantifies sentence complexity based on:

Predicate Complexity ( $C_{pred}$ ): Event density.
Argument Complexity ( $C_{arg}$ ): Number of participants.
Argument Span Complexity ( $C_{span}$ ): Length of argument spans (indicating nested structures).
Model Uncertainty ( $L_{init}$ ): The initial NMT loss of the sample.

The training corpus is sorted by $D_s$ . The curriculum scheduler dynamically expands the sampling pool ( $S_k$ ) from the easiest 25% of data to 100% across four training stages. This ensures the model progresses from syntactically simple to structurally complex instances, stabilizing early-stage optimization.

2.3 Implementation Details

Base Model: Qwen2-7B-Instruct.
Adaptation: LoRA (Low-Rank Adaptation) with rank $r=16$ , applied to query and value projections.
Efficiency: 4-bit quantization (QLoRA) allows training on a single 8GB consumer GPU (NVIDIA RTX 4060 Ti).
Training Strategy: Heterogeneous batch mixing (2:1:1:1 ratio for NMT:Thai:NER:SRL) with a unified forward pass.

3. Key Contributions

Generative SRL Paradigm: The study recasts SRL as an autoregressive auxiliary task, enabling decoder-only LLMs to encode predicate-argument structures within a shared adapter space without architectural incompatibility.
Structure-Aware Curriculum: The introduction of a difficulty metric derived from SRL complexity (rather than sequence length) to regulate multi-task training trajectories, specifically addressing the limitations of length-based heuristics in isolating languages.
Robust Low-Resource Framework: A unified approach that combines cross-lingual transfer, entity grounding, and structural priors to achieve significant gains in extremely low-resource settings.

4. Experimental Results

Experiments were conducted on Chinese-to-Lao (Zh-Lo) and Chinese-to-Myanmar (Zh-My) translation tasks.

4.1 Performance on Zh-Lo

The proposed framework (Qwen2-7B + SRL + CL) achieved a BLEU score of 13.82, representing a +5.58 BLEU improvement over the supervised fine-tuning (SFT) baseline (8.24).

Ablation Insights:
- Adding Thai transfer (+2.08 BLEU) established morphological alignments.
- Adding NER (+1.55 BLEU) reduced entity hallucinations.
- Adding SRL (+1.59 BLEU) corrected argument misalignment.
- Adding Curriculum Learning further optimized convergence.
Comparison: The method outperformed massive multilingual baselines (NLLB-200) and region-adapted models (SeaLLM-7B) under matched data conditions.

4.2 Cross-Lingual Generalization (Zh-My)

The framework demonstrated transferability to Chinese-to-Myanmar, yielding a +1.77 BLEU gain over the SFT baseline. This confirms the approach's effectiveness for typologically similar isolating languages.

4.3 Training Dynamics

Analysis of training trajectories revealed that while complex semantic tasks initially cause a "cold-start" delay, the semantic-driven curriculum significantly alleviates early gradient conflicts. The proposed scheduler achieved faster initial convergence and a higher final BLEU ceiling compared to uniform random sampling and length-based curriculum baselines.

4.4 Qualitative Improvements

Case studies showed the framework's ability to:

Restore dropped matrix verbs (e.g., "strengthened").
Correctly embed subordinate clauses (e.g., "decided to postpone").
Maintain long-distance dependencies and main clause heads in complex sentences where baselines omitted critical verbs.

5. Significance and Claims

The paper claims that under extremely low-resource conditions, explicit structural supervision (via generative SRL) combined with complexity-aware curriculum learning serves as an effective complement to, and in some cases an alternative to, massive data scaling or domain-specific pre-training.

The authors assert that their framework successfully addresses two critical bottlenecks in LR-NMT:

Semantic Fidelity: By internalizing predicate-argument dependencies, the model overcomes the tendency to overfit to shallow lexical patterns.
Optimization Stability: By regulating data sampling based on structural difficulty, the framework mitigates the negative transfer and convergence plateaus inherent in multi-task fine-tuning.

The study concludes that these mechanisms are particularly effective for isolating languages and offer a generalizable paradigm for adapting LLMs to diverse, low-resource linguistic scenarios.

6. Limitations and Future Work

The authors acknowledge that the current difficulty metric uses a uniform weighting strategy for its components, which could be refined via adaptive weighting in future work. Additionally, the evaluation is currently limited to isolating languages (Lao, Myanmar); extending the framework to morphologically rich, fusional languages is identified as a necessary step to assess broader cross-linguistic generalizability.