💬 NLP

Testing the Assumptions of Active Learning for Translation Tasks with Few Samples

This paper challenges the core assumptions of active learning for translation tasks with few samples by demonstrating that the informativeness and diversity of selected data do not correlate with performance, whereas sample ordering and interactions with pre-training data are the primary drivers of success.

Original authors: Lorenzo Jaime Yu Flores, Cesare Spinoso di-Piano, Ori Ernst, David Ifeoluwa Adelani, Jackie Chi Kit Cheung

Published 2026-04-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lorenzo Jaime Yu Flores, Cesare Spinoso di-Piano, Ori Ernst, David Ifeoluwa Adelani, Jackie Chi Kit Cheung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart but inexperienced chef (the AI model) how to cook a specific type of cuisine (Machine Translation) using a very limited budget. You can only afford to buy 100 to 500 ingredients (labeled data samples) to practice with.

The standard advice from "Active Learning" experts is: "Don't just buy random ingredients! Be smart. Buy the weird, rare, or confusing ones that the chef doesn't know how to handle yet. That's where the most learning happens."

This paper is a group of researchers who decided to test if that advice actually works when you only have a tiny budget. They ran a series of experiments and found that the standard advice is mostly wrong for this specific scenario.

Here is the breakdown of their findings using simple analogies:

1. The "Confusing Ingredient" Myth

The Assumption: Active Learning strategies try to pick the data points where the AI is most "confused" (uncertain) or the most "diverse" (unique). The idea is that if you teach the chef how to handle the weirdest, most confusing ingredients first, they will learn faster.
The Reality: The researchers found that picking these "confusing" or "rare" ingredients did not make the chef cook better. In fact, it didn't matter much which specific ingredients they picked; the final dish tasted about the same whether they picked the weird ones or just grabbed a handful of random ones.

The Takeaway: When you only have a tiny amount of data, trying to be "smart" about what you pick doesn't help. The "smart" strategies performed just as poorly as just grabbing random samples.

2. The "Easy Wins" Surprise

The Assumption: We should train the model on the things it is bad at, because that's where it needs the most help.
The Reality: The researchers discovered the opposite! The model actually learned better when it was trained on the things it was already pretty good at.

The Analogy: Imagine a student who is bad at math. The old advice says, "Give them the hardest calculus problems to solve!" The new finding says, "Actually, give them the easy algebra problems they can solve easily. It builds their confidence and helps them learn the underlying rules better."
The Takeaway: Active Learning strategies kept picking the "hard" examples (where the model failed), but the model actually improved more when it practiced on the "easy" examples (where it was already doing well).

3. The "Order of Operations" (The Real Secret)

If what you pick doesn't matter, what does? The researchers found that the order in which you feed the data matters way more than the data itself.

The Analogy: Imagine you are teaching someone to drive. It matters less whether you start with a red car or a blue car. What matters is when you teach them. If you teach them how to park before you teach them how to turn the steering wheel, they will crash. If you teach them in the right sequence, they succeed.
The Finding: The researchers found that simply shuffling the order of the training data changed the results by 50% to 90%. The "sequence" of learning was the biggest factor, not the "content" of the learning.

4. The "Ghost Knowledge" (Pre-training)

The model already knows a lot of things from its "childhood" (pre-training data).

The Analogy: The chef already knows how to chop onions because they watched thousands of cooking videos before you hired them.
The Finding: Sometimes, the model would learn a new word from your small dataset, but then immediately forget it or use it wrong because it was confused by its old memories. Other times, it would use a word you never taught it, pulling it from its old memory bank.
The Takeaway: In these tiny-data scenarios, the model is fighting a battle between what you are teaching it now and what it already knows. The "Active Learning" strategies completely ignored this battle, which is why they failed.

Summary: What Should We Do?

The paper concludes that if you are working with very little data (100–500 samples):

Stop trying to be too clever about picking "diverse" or "uncertain" data. It doesn't work.
Focus on the order. How you present the data to the model is more important than which data you pick.
Respect the model's past. The model's existing knowledge (pre-training) interacts with your new data in complex ways that current methods don't account for.

In short: When you have a tiny budget, don't worry about picking the "perfect" ingredients. Worry about the recipe order and how the chef's past experience mixes with your new instructions.

1. Problem Statement

Active Learning (AL) is a paradigm designed to optimize annotation budgets by iteratively selecting the most "informative" or "diverse" unlabeled samples for labeling and training. While AL has shown success in image classification and text classification, recent studies (e.g., Perlitz et al., 2023) indicate that AL strategies often fail to outperform random sampling in low-resource Natural Language Generation (NLG) tasks, specifically when the annotation budget is very small (100–500 samples).

The core problem addressed in this paper is: Why do standard AL strategies fail in low-data Machine Translation (MT) scenarios? The authors hypothesize that the fundamental assumptions driving AL (that selecting informative/diverse data leads to better performance) may not hold true in this specific regime.

2. Methodology

The authors conducted a comprehensive empirical study across three language pairs (English-Afrikaans, English-German, English-Filipino) and three model architectures (Flan-T5 Base, Llama 3.1-8B, Gemma-2-2B).

Experimental Setup

Budgets: Experiments were run with $b=100$ and $b=500$ samples.
AL Strategies: They tested various acquisition functions categorized into:
- Informativeness/Uncertainty: Mean Token Probability, Mean Token Entropy, BALD (Bayesian Active Learning by Disagreement), Lexical Similarity (via Dropout).
- Diversity: DelFy (word rarity), Core Set (L2 distance in embedding space).
Baselines: Random sampling was used as the primary baseline.
Evaluation: Performance was measured using ChrF+ (Character F-score) on the FLORES Plus test set.

Analysis Framework

The study was structured around two Research Questions (RQs):

RQ1: Do the core AL assumptions (training on informative/diverse data yields better performance) hold in low-data MT?
RQ2: What other factors (beyond data content) drive performance variance in low-data scenarios?

To answer these, the authors:

Calculated Spearman correlations between AL metrics (informativeness/diversity scores) and final test performance.
Analyzed the variance in performance attributable to sample ordering vs. sample content using variance decomposition.
Conducted a qualitative case study on an English-Filipino dataset to observe training dynamics, vocabulary acquisition, and interactions with pre-training knowledge.

3. Key Contributions & Findings

Finding 1: AL Strategies Fail to Outperform Random Sampling

Result: Across 54 configurations (3 models $\times$ 3 languages $\times$ 6 strategies), AL strategies outperformed random sampling in only 7 cases, and the margin was negligible (<1.99 ChrF points).
Implication: In the 100–500 sample regime, the specific selection strategy of AL provides no significant benefit over random selection.

Finding 2: Core Assumptions Do Not Hold (RQ1)

Assumption 1 (Informativeness/Diversity Correlation): The authors tested if higher informativeness or diversity scores correlate with better test performance.
- Result: There is no strong correlation. Informativeness metrics showed weak positive or even negative correlations. Diversity metrics showed slightly higher correlations but explained less than 29% of the variance (with only one significant correlation after Bonferroni correction).
- Conclusion: Optimizing for these metrics does not guarantee better model performance.
Assumption 2 (Training on "Hard" Samples): AL strategies typically select samples where the model performs poorly (high uncertainty).
- Result: The study found that models achieve better test performance when fine-tuned on samples where they already perform reasonably well. Conversely, training on samples where the model performs poorly (the standard AL selection) yields lower performance.
- Conclusion: The implicit AL assumption that "learning from mistakes" is optimal in low-data MT is false; models benefit more from reinforcing correct patterns.

Finding 3: Sample Ordering and Pre-training Interactions Dominate (RQ2)

Since data content (diversity/informativeness) failed to explain performance variance, the authors investigated other factors.

Sample Ordering: Using variance decomposition, the authors found that the order in which samples are presented during fine-tuning accounts for 53% to 94% of the performance variance (depending on model and budget). This dwarfs the impact of the actual content of the samples.
Pre-training Interactions (Case Study):
- Vocabulary Mismatch: Models often fail to learn vocabulary present in the training set or generate vocabulary not in the training set (relying on pre-training knowledge).
- Hallucination & Interference: In certain orderings, models incorrectly generate foreign words (e.g., Indonesian words in a Filipino translation task) or fail to use training vocabulary correctly.
- Conclusion: In low-data regimes, the interaction between the fine-tuning order and the model's pre-existing pre-training knowledge is a more critical determinant of success than the specific diversity or informativeness of the selected data.

4. Significance and Future Directions

Scientific Impact

This paper challenges the foundational dogma of Active Learning in the context of Large Language Models (LLMs) for low-resource generation. It demonstrates that data selection heuristics based on uncertainty or diversity are insufficient when data is extremely scarce. The "signal" provided by the selection strategy is overwhelmed by the "noise" introduced by training dynamics (ordering) and pre-training priors.

Practical Implications

Annotation Strategy: Practitioners should not rely solely on complex AL acquisition functions for MT when budgets are <500 samples; random sampling may be equally effective and cheaper to implement.
Training Dynamics: Future work must prioritize Curriculum Learning or Ordering Strategies over data selection. The order of presentation matters more than the specific subset chosen.

Future Work Recommendations

The authors suggest three directions:

Generalization: Verify if these findings hold for other generation tasks beyond MT.
Ordering Heuristics: Identify interpretable characteristics of sample ordering that correlate with performance to build new AL algorithms.
Learning Dynamics: Design AL strategies that explicitly account for how models "learn" (or fail to learn) vocabulary from pre-training, rather than just optimizing for static data metrics.

Summary Conclusion

The paper concludes that in low-data Machine Translation, the assumption that "more informative or diverse data leads to better performance" is invalid. Instead, performance is primarily driven by sample ordering and interactions with pre-training data. Consequently, future Active Learning methods for low-resource scenarios must move beyond traditional acquisition functions and incorporate heuristics related to training dynamics and model learning behavior.