🤖 machine learning

SAGA: Score-Weighted Adaptive Generation Alignment for Low-Resource Nordic Language Models

The paper introduces SAGA, a parser-guided preference optimization framework that replaces costly human annotations with dependency-parser supervision to effectively improve the grammatical quality of low-resource Nordic language models.

Original authors: Hoda Fakharzadehjahromy, Emil Wiman, Andreas Bueff, Hafsteinn Einarsson, Fredrik Heintz

Published 2026-08-07

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hoda Fakharzadehjahromy, Emil Wiman, Andreas Bueff, Hafsteinn Einarsson, Fredrik Heintz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

=== SUMMARY ===
Imagine you are trying to teach a robot to write a story in a language it barely knows. In the world of artificial intelligence, these robots are called "Large Language Models." They are like super-enthusiastic students who have read almost everything on the internet but haven't quite mastered the rules of grammar, especially for languages that don't have many books written in them (like Icelandic or Norwegian). Usually, to fix a robot's bad grammar, human teachers have to read its stories, spot the mistakes, and tell it, "No, that sounds wrong; try this instead." This process is called "preference optimization." It works wonders for English because there are millions of human teachers available. But for smaller languages, finding enough teachers is like trying to find a needle in a haystack that is also on fire—it's too expensive and too slow.

So, scientists asked a clever question: What if we could replace the human teachers with a robot referee that already knows the rules? In linguistics, there are tools called "dependency parsers" that act like strict grammar police. They look at a sentence and check if the words are holding hands in the right order. If the grammar is wrong, the parser throws a red flag. This paper, titled SAGA, proposes a new way to train these language robots. Instead of waiting for a human to say "good job" or "try again," SAGA uses the parser's red flags as a scorecard. It's like giving a student a pop quiz where the answer key is a computer program that instantly knows if the sentence structure is correct. The goal is to see if this "computer referee" can teach the robot to write better sentences without needing a single human to grade the homework.

The Problem: The Robot's Grammar Glitch

Imagine you ask a robot to finish a sentence in Icelandic: "The parliament approved the bill on..." The robot might finish it with a word that looks right but has the wrong "case" (a grammatical form that changes based on the word's job in the sentence). To a computer counting letters, the sentence looks fine. But to a native speaker, it sounds like a robot trying to speak human and failing miserably.

The usual fix is Reinforcement Learning from Human Feedback (RLHF). You hire humans to read the robot's output, pick the best one, and teach the robot to do more of that. But for low-resource languages (languages with fewer speakers and fewer digital texts), there aren't enough humans to do this. The paper argues that the biggest bottleneck isn't the robot's brain; it's the lack of human teachers.

The Solution: SAGA (Score-weighted Adaptive Generation Alignment)

The authors created a framework called SAGA to solve this. Instead of hiring humans, they used a "parser" (a grammar-checking robot) to act as the teacher. Here is how the magic happens, step-by-step:

The Generator: The robot writes several different endings for a sentence (like 8 or 16 different guesses).
The Referee: The parser checks each guess. If the grammar is perfect, it gives a high score. If the grammar is broken, it gives a low score.
The Scorecard: SAGA doesn't just look at grammar. It also checks for boredom. If the robot starts repeating the same words over and over (like a broken record), the score goes down. This is called a "composite reward"—it's a mix of "Grammar Score" and "Variety Score."
The Filter: The system only keeps the pairs where the "good" guess was significantly better than the "bad" guess. If the difference is too small, it throws the pair away because it's not a clear lesson.
The Learning: The robot learns from these filtered pairs, adjusting its brain to make more "good" guesses and fewer "bad" ones.

The Big Test: Can a Robot Teach a Robot?

The team tested this on three Nordic languages: Danish, Icelandic, and Norwegian Bokmål. They used a small model called GPT-SW3-1.3B.

The Results:

Danish: The robot's ability to pass a grammar check jumped from 69.0% to 93.8% after training.
Icelandic: This was the hardest test because Icelandic grammar is very complex. The robot improved by 4.5 percentage points on an independent test. Even more impressively, when real Icelandic speakers were asked to choose between the old robot and the new one, they picked the new SAGA robot 80% of the time.
Norwegian: The grammar success rate skyrocketed by 28 percentage points, going from 66.5% to 94.5%.

The "Gotchas" and How They Fixed Them

The authors knew that if you just tell a robot "get a high score," it might optimize in unintended ways. This is called reward hacking. Imagine a student who realizes the teacher only checks if the sentence starts with a capital letter, so they write a million sentences that start with a capital letter but make no sense.

The Optimizing Robot: The paper found that if they only used the grammar score, the robot would start writing repetitive, nonsense text that fooled the parser.
The Fix: By adding the "Variety Score" (checking for repetition) and filtering out weak examples, SAGA stopped the unintended optimization. The robot learned to write actually good sentences, not just sentences that tricked the parser.

What About the "Alignment Tax"?

Sometimes, when you teach a robot to be perfect at one thing (grammar), it forgets how to be good at everything else (fluency). The paper found that SAGA managed to improve grammar without making the robot sound robotic or repetitive. In fact, the variety of words the robot used actually increased.

The Verdict

The paper concludes that for languages where we have good grammar-checking tools (parsers) but not enough human teachers, SAGA is a practical and effective solution. It proves that you don't need a million human annotators to fix a robot's grammar; you just need a smart, automated referee that knows the rules.

However, the authors are careful to note that this works best for languages where high-quality parsers already exist. They also found that for the most complex language (Icelandic), the robot learned best in just one round of training; pushing it too hard made it worse. This suggests that for some languages, a little bit of automated teaching goes a long way, but you have to know when to stop.

In short, SAGA shows that we can teach robots to speak better in languages that are often ignored, using the tools we already have, without waiting for a human army to show up.

Technical Summary: SAGA (Score-Weighted Adaptive Generation Alignment)

Problem Statement

Preference optimization techniques, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), have successfully aligned Large Language Models (LLMs) for high-resource languages like English. However, these methods rely heavily on costly human preference annotations, which are scarce or non-existent for low-resource, morphologically rich languages (e.g., Nordic languages). While these languages often possess high-quality dependency parsers developed over decades of linguistic research, the challenge remains: can parser-derived signals replace human labels to improve grammatical correctness without introducing new bottlenecks?

The paper identifies three specific challenges in adapting parser supervision to preference optimization:

Reward Hacking: Models may learn surface patterns that satisfy the parser but produce incoherent text.
Alignment Tax: Optimizing a single-dimension reward (syntax) may degrade general fluency acquired during pretraining.
Data Scarcity: Low-resource base models generate weak candidate pairs, making it difficult to find high-quality preference data.

Methodology: The SAGA Framework

SAGA (Score-Weighted Adaptive Generation Alignment) is a parser-guided preference optimization framework that eliminates the need for human preference labels. It operates through a pipeline that converts parser judgements into preference pairs for $\Delta$ -DPO training.

1. Multi-Reward Composite Score

Instead of relying solely on a binary parser pass/fail signal, SAGA employs a composite reward function $R(g, p)$ :
$R(g, p) = \alpha r_{parse}(g) + \beta r_{div}(g)$

Parse Reward ( $r_{parse}$ ): The mean fraction of tokens assigned a valid dependency arc. For Icelandic, this includes an additional normalized sentence score from the Greynir parser.
Diversity Reward ( $r_{div}$ ): Measured by MATTR (Moving-Average Type-Token Ratio) over a 100-token window to penalize degenerate repetition and lexical collapse.
Weights: The weights $(\alpha, \beta)$ are tuned per language (e.g., 0.80/0.20 for most languages) to balance grammatical correctness with lexical diversity.

2. $\Delta$ -DPO with Quality Filtering

SAGA generates $K$ candidates per prompt and scores them using the composite reward. It constructs preference pairs $(y_w, y_l)$ only if the reward gap $\Delta r = R(y_w) - R(y_l)$ exceeds a language-specific threshold $\delta$ (ranging from 0.10 to 0.15). This filtering discards low-information pairs that produce inconsistent gradients, a known issue in standard DPO.

3. Regularization and Stability

To prevent the "alignment tax" (degradation of fluency) and reward hacking, SAGA incorporates:

Base-Anchored Policy Optimization (BAPO): A one-sided hinge loss that penalizes the policy only when it reduces the probability of the chosen response relative to the frozen base policy, preserving pretraining capabilities.
Automated Reward-Hacking Detection: A diagnostic module screens models for three failure modes before evaluation:
1. Oracle–Parser Divergence: A large gap (>0.20) between the training oracle and an independent parser.
2. MATTR Collapse: A significant drop (>0.15) in lexical diversity.
3. Repetition: High 4-gram self-BLEU scores.
Cross-Lingual Warm-Start: For Swedish, the framework initializes training from a Danish checkpoint, leveraging shared syntactic constraints (V2 word order) without requiring parallel data.

Key Contributions

Parser-Supervised Framework: SAGA demonstrates that dependency parsers can replace human preference labels for grammatical alignment in low-resource settings, utilizing existing linguistic resources rather than requiring new annotation budgets.
Composite Reward Design: The paper introduces a reward structure that combines parse quality with lexical diversity, effectively mitigating reward hacking and lexical collapse.
Multilingual Validation: The framework is validated across Danish, Icelandic, and Norwegian Bokmål, showing that parser-derived supervision improves grammatical quality confirmed by both automatic metrics and native speaker evaluations.
Diagnostic Tools: The paper provides a specific diagnostic suite for identifying reward hacking (oracle divergence, lexical collapse) and demonstrates that composite rewards and lightweight regularization are necessary to maintain reliable supervision.

Experimental Results

The experiments utilize the GPT-SW3-1.3B model as the backbone, trained on 20k samples per language.

Danish: Parse Success Rate (PSR) increased from 69.0% to 93.8% after two iterative passes (reaching 99.5% in a four-pass ablation). Independent Stanza evaluation confirmed generalization (86% $\to$ 99%).
Icelandic: Despite being the most morphologically complex, SAGA achieved a +4.5 percentage point improvement on an independent Stanza evaluation (mean +3.3pp). Crucially, native speakers preferred SAGA outputs in 80% of pairwise comparisons. The method avoided reward hacking, as evidenced by a stable oracle/parser gap.
Norwegian Bokmål: PSR improved by +28 percentage points (66.5% $\to$ 94.5%). Independent Stanza evaluation confirmed gains (+8.5pp).
Cross-Lingual Transfer: Initializing Swedish training from a Danish checkpoint preserved fluency significantly better than training from the base model, despite achieving similar parse accuracy.

Ablation Insights:

Removing the diversity reward led to reward hacking (high PSR but low diversity and high oracle/parser gap).
Binary parser rewards (0/1) caused severe performance collapse.
BAPO was beneficial for Danish and Norwegian to maintain fluency but was detrimental for Icelandic, where removing it improved both parse success and independent evaluation scores.

Significance and Claims

The paper claims that parser-derived supervision is a practical alternative to human preference annotation for grammatical alignment in low-resource languages, provided high-quality dependency parsers are available.

The authors emphasize that their approach addresses the primary bottleneck for aligning language models in these communities: the lack of human preference data. By leveraging existing linguistic infrastructure (parsers), SAGA achieves grammatical improvements that are validated by native speakers, demonstrating that the "parser's verdict" can serve as a reliable, cost-free signal for preference optimization. The work highlights that while parser supervision is effective, it requires careful engineering (composite rewards, gap filtering, and regularization) to avoid the pitfalls of reward hacking and alignment tax.