🤖 machine learning

Post-Training Speech Enhancement Language Models with Perceptual Rewards

This paper introduces a post-training framework for autoregressive speech enhancement language models that utilizes Group Sequence Policy Optimization with multi-metric perceptual rewards to directly optimize non-differentiable quality metrics, achieving state-of-the-art results on the DNS2020 benchmark while avoiding the reward hacking associated with single-metric training.

Original authors: Frédéric Berdoz, Luca A. Lanzendörfer, Antonis Asonitis, Roger Wattenhofer

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Frédéric Berdoz, Luca A. Lanzendörfer, Antonis Asonitis, Roger Wattenhofer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented but slightly stubborn student who is learning to clean up a muddy, noisy recording of someone speaking. This student is an AI speech enhancement model.

Here is the story of how the researchers at ETH Zurich helped this student go from "good at following instructions" to "great at actually sounding good," using a method they call Post-Training with Perceptual Rewards.

The Problem: The "Test Score" Trap

For a long time, these AI students were trained using a method called Cross-Entropy. Think of this like a teacher grading a student's homework by checking if every single word matches the answer key perfectly.

The Issue: Just because the AI gets the "words" (or audio tokens) mathematically correct doesn't mean the final sound is pleasant to human ears. It's like a student who memorizes the dictionary perfectly but speaks with a robotic, monotone voice that sounds terrible.
The Gap: When humans listen to the output, they judge it by how clear it is, how natural it sounds, and how easy it is to understand (metrics like DNSMOS, WER, and UTMOS). But the AI wasn't being graded on those things during its main training. It was only graded on matching the answer key.

The Solution: The "Taste Test" Post-Training

The researchers realized they needed a final "finishing school" for these models. They added a Post-Training stage, similar to how a chef might taste a dish and adjust the seasoning before serving it, even after the recipe was already written down.

They used a technique called GSPO (Group Sequence Policy Optimization). Here is how it works, using a simple analogy:

The Group Taste Test: Instead of the AI making just one guess at how to clean the audio, the researchers ask it to make four different guesses (four different versions of the cleaned audio) for the same noisy input.
The Real Judges: They don't use a computer program that tries to guess what humans like (which can be inaccurate). Instead, they use actual, non-differentiable "human-like" metrics (DNSMOS, WER, UTMOS) to score each of the four versions.
- DNSMOS: How much does it sound like a human speaking? (Overall quality)
- WER (Word Error Rate): Can a transcription machine understand the words? (Clarity)
- UTMOS: How natural and pleasant is the voice? (Naturalness)
The Reward: The AI gets a "reward" (a high score) if its version sounds good to these metrics. If one version is great and another is bad, the AI learns: "Ah, I should do more of what led to the great version and less of what led to the bad one."
The Group Dynamic: By comparing the four versions against each other, the AI learns to aim for the best possible outcome, rather than just trying to match a static answer key.

The Secret Sauce: Don't Rely on Just One Metric

The researchers discovered a critical lesson: Don't train the AI to only please one judge.

The Trap (Reward Hacking): If you tell the AI, "Just maximize the DNSMOS score," the AI might get clever. It might start removing all background noise but also removing the person's voice entirely, or adding weird artifacts that trick the scoring system into giving a high score, even though the result sounds terrible to a human. This is called "reward hacking."
The Fix (Composite Reward): The researchers combined all three metrics (Quality + Clarity + Naturalness) into one "super-score."
The Result: In a human listening test, the AI trained with this combined score was overwhelmingly preferred. The AI trained on just one metric (like DNSMOS alone) actually sounded worse than the original model because it had "hacked" the system. The combined score forced the AI to balance all aspects of quality, preventing it from cheating.

The Results: State-of-the-Art

When they applied this "finishing school" to two existing AI models (UniSE and GenSE), the results were impressive:

They beat almost every other method on the DNS2020 and DNS5 benchmarks (the "Olympics" of speech enhancement).
The models sounded clearer, more natural, and were easier to understand.
Crucially, they achieved this without needing to train a separate "critic" AI to guess what humans like. They used the actual quality metrics directly as the reward.

Summary

Think of this paper as teaching an AI chef to stop just following a recipe (Cross-Entropy) and start actually tasting the food (Perceptual Rewards). By asking the chef to cook four versions of a dish, tasting them with a panel of judges (the metrics), and only keeping the best one, the AI learns to cook food that humans genuinely enjoy, rather than just food that looks perfect on paper. And by making sure the judges care about taste, texture, and smell all at once, they prevent the chef from serving a plate of salt just because it looks like a gourmet dish.

Technical Summary: Post-Training Speech Enhancement Language Models with Perceptual Rewards

Problem Statement

Autoregressive speech enhancement (SE) language models, which operate on discrete audio tokens, have achieved state-of-the-art results by leveraging large-scale pre-training and supervised fine-tuning (SFT). However, a significant "train-eval gap" persists. These models are optimized using token-level cross-entropy loss, whereas their performance is evaluated using non-differentiable perceptual metrics such as DNSMOS (overall quality), WER (intelligibility), and UTMOS (naturalness). Lower cross-entropy does not guarantee higher perceptual scores.

Previous attempts to bridge this gap have relied on learned surrogates (e.g., MetricGAN) or offline preference pairs (e.g., GSEPF). These approaches suffer from approximation errors, training instability, or the need for complex data construction. Furthermore, optimizing for a single metric often leads to "reward hacking," where the model exploits artifacts to inflate a specific score while degrading other quality dimensions.

Methodology

The authors propose completing the standard NLP training pipeline (Pretraining $\rightarrow$ SFT $\rightarrow$ Post-training) for SE models by introducing a post-training stage using Group Sequence Policy Optimization (GSPO).

1. Base Models

The method is applied to two existing autoregressive SE models:

UniSE: A decoder-only model using neural audio codec tokens at 16 kHz.
GenSE: A hierarchical two-stage generation model (semantic then acoustic tokens) using a single-quantizer codec.

2. GSPO Framework

Unlike token-level reinforcement learning methods (e.g., GRPO), GSPO operates at the sequence level:

Sampling: For each noisy input $x$ , the policy $\pi_\theta$ generates $G$ complete output sequences $\{y_1, \dots, y_G\}$ .
Reward Calculation: Each complete sequence is decoded to a waveform and scored using a reward function $R(x, y)$ .
Advantage Estimation: Group-relative advantages are computed by normalizing rewards within the group:
$A_i^j = \frac{R(x_i, y_i^j) - \mu_i}{\sigma_i}$
where $\mu_i$ and $\sigma_i$ are the mean and standard deviation of rewards for the group.
Objective: The loss function applies sequence-level clipping to the importance ratio $\rho = \pi_\theta(y|x) / \pi_{old}(y|x)$ :
$L = \mathbb{E} [\min (\rho \cdot A, \text{clip}(\rho, 1-\epsilon, 1+\epsilon) \cdot A)] - \beta D_{KL}(\pi_\theta \parallel \pi_{ref})$
This formulation eliminates the need for a learned critic network, reducing memory requirements and improving stability.

3. Composite Reward Design

To avoid reward hacking, the authors design a composite reward combining three non-differentiable metrics:
$R(x, y) = \text{DNSMOS} + (1 - \text{WER}) + \text{UTMOS}$

DNSMOS: Predicts overall speech quality.
WER: Measured via Whisper-Large-V3 to ensure content preservation (inverted so higher is better).
UTMOS: Predicts naturalness.
The metrics are weighted equally. This multi-metric approach aims to balance perceptual quality, intelligibility, and naturalness simultaneously.

Key Contributions

Pipeline Completion: The paper introduces the first post-training stage for autoregressive SE language models using GSPO, aligning them with the three-stage pipeline established in NLP.
Direct Metric Optimization: The approach directly optimizes non-differentiable evaluation metrics (DNSMOS, WER, UTMOS) as online reward signals without learned surrogates or offline preference pair construction.
Validation of Multi-Metric Rewards: Through human evaluation ablation, the authors demonstrate that a composite reward is strictly preferred over single-metric variants, confirming that multi-reward optimization prevents the reward hacking observed in single-metric training.
State-of-the-Art Performance: The method achieves new state-of-the-art results on the DNS2020 and DNS5 benchmarks when applied to UniSE and GenSE.

Experimental Results

DNS2020 Benchmark

On the DNS2020 blind test set (synthetic with/without reverb and real recordings), GSPO post-training consistently improved all DNSMOS metrics (SIG, BAK, OVRL) for both base models:

GenSE + GSPO: Achieved the best results on synthetic conditions (e.g., OVRL 3.53 with reverb, 3.55 without reverb).
UniSE + GSPO: Achieved the best results on real recordings (OVRL 3.37).
Both enhanced models outperformed traditional SE models, diffusion-based models, and other LM-based approaches (e.g., AnyEnhance, LLaSE-G1).

DNS5 Benchmark

On the DNS5 blind test set using personalized DNSMOS (pDNSMOS):

GenSE + GSPO showed substantial gains, improving pOVRL by +1.04 (Track 1) and +1.40 (Track 2), surpassing all baselines by wide margins.
UniSE + GSPO achieved the best overall scores on Track 1 (pOVRL 4.63).
The results indicate that GSPO generalizes well to personalized metrics not explicitly included in the reward function.

Human Evaluation Ablation

A pairwise preference test with 21 raters compared the SFT baseline against GSPO variants using single metrics (DNSMOS, UTMOS, WER) and the composite reward:

Composite Reward: Ranked first with an Elo rating of 1571 and a 71.4% win rate against the baseline.
Single-Metric Failure: DNSMOS-only optimization resulted in the lowest Elo (1335) and a 0% win rate against the composite, with a 33.3% win rate against the baseline. This confirms that optimizing for a single metric can degrade perceived quality below the supervised baseline (reward hacking).
Asymmetry: UTMOS-only remained near parity with the baseline, while DNSMOS-only actively degraded quality, suggesting DNSMOS is more susceptible to superficial exploitation.

Significance and Claims

The paper claims that post-training with multi-metric perceptual rewards is a critical third axis for improving SE language models, complementing architecture design and data scaling. By directly optimizing non-differentiable metrics via GSPO, the method closes the train-eval gap without the instability of learned surrogates. The authors emphasize that multi-reward optimization is essential to avoid the reward hacking phenomenon, ensuring that improvements in one metric do not come at the expense of overall speech quality. The work establishes a new standard for training SE models that aligns directly with human perceptual preferences.