⚡ electrical engineering

OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL

OLIVE is a self-supervised speech representation learning framework that jointly optimizes view-augmented masked latent prediction and waveform reconstruction to produce robust, signal-informative representations that excel in generation and speaker tasks while maintaining competitive performance in recognition and semantic applications.

Original authors: Karl El Hajal, Mathew Magimai. -Doss

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Karl El Hajal, Mathew Magimai. -Doss

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand human speech. Most modern robots learn by playing a game of "fill in the blank." You show them a sentence with some words missing, and they have to guess what the missing words were based on the context. This is great for understanding what is being said (like reading a book), but it often forces the robot to throw away the tiny, messy details of how the voice sounds (the breath, the pitch, the texture), because those details aren't needed just to guess the missing words.

The paper introduces a new method called OLIVE (Online Latent prediction with Invariant Views and rEconstruction). Think of OLIVE as a robot that learns two skills at the same time, rather than just one.

The Two-Skill Training Camp

OLIVE splits its training into two distinct "branches," like a student studying for two different exams simultaneously:

The "Analysis" Branch (The Detective):
- The Goal: To understand the meaning and identify the speaker, regardless of background noise or volume changes.
- The Method: This is like the standard "fill in the blank" game. The robot listens to a sentence, hides parts of it, and tries to predict the missing context. To make the robot smarter, the researchers give it two slightly different versions of the same audio (one might be slightly louder, one might have a bit of background noise added). The robot learns to ignore these small differences and focus on the core message.
- The Result: This creates a very strong "brain" for understanding language and identifying who is speaking.
The "Synthesis" Branch (The Artist):
- The Goal: To be able to recreate the original sound wave perfectly, like a high-quality voice synthesizer.
- The Method: While the "Detective" is looking at the big picture, the "Artist" is looking at the raw, early details of the sound. It tries to take the robot's internal notes and rebuild the actual sound wave from scratch.
- The Result: This forces the robot to keep all the tiny, fine-grained details (the specific timbre of the voice, the breathiness) that the "Detective" might have otherwise thrown away.

The Magic Trick: Keeping the Best of Both Worlds

The clever part of OLIVE is how it manages these two jobs without them fighting each other.

The "Early" Layers: The robot's early processing layers are tasked with keeping the raw sound details intact so the "Artist" can rebuild the wave.
The "Later" Layers: The deeper, more abstract layers are free to focus entirely on understanding the meaning and context, just like the "Detective."

It's like a factory assembly line. The early workers make sure the raw materials (the sound details) are preserved perfectly. The later workers take those materials and assemble them into a complex product (the meaning of the sentence). Because the early workers are required to keep the raw materials, the later workers never accidentally throw away the "good stuff" just to save space.

What Did They Find?

The researchers tested this new robot against other famous speech-learning models (like wav2vec 2.0 and HuBERT). Here is what happened:

Better Voice Generation: Because OLIVE kept the fine details, it was much better at tasks that require recreating or changing voices (like voice conversion or speech enhancement). It could "hear" and "speak" with more natural texture.
Better Speaker Identification: It got better at telling who was speaking, likely because it remembered the unique "fingerprint" of the voice better than models that only focused on meaning.
Still Great at Understanding: Crucially, it didn't lose its ability to understand speech. It performed just as well as the other top models on tasks like recognizing words or translating languages.

The Bottom Line

OLIVE is a speech-learning framework that refuses to choose between "understanding" and "recreating." By training the model to do both at once—using a "Detective" to find the meaning and an "Artist" to rebuild the sound—it creates a speech AI that is both a brilliant listener and a high-fidelity voice synthesizer, all within a single model.

The paper concludes that this approach allows a single pre-trained model to retain the acoustic details necessary for high-quality sound generation while maintaining the strong ability to discriminate and understand speech.

Technical Summary: OLIVE - View-Augmented Latent Prediction with Waveform Reconstruction

Problem Statement

Recent self-supervised learning (SSL) advancements in speech have been driven by models like wav2vec 2.0, HuBERT, and data2vec, which excel at extracting contextual representations for discriminative downstream tasks (e.g., automatic speech recognition, speaker identification). However, these methods predominantly focus on analysis, often discarding signal-level information critical for speech generation and synthesis. While speech communication fundamentally involves both perception (hearing) and production (speaking), modern SSL systems treat synthesis as a peripheral downstream process rather than an integral part of representation learning. Existing attempts to couple analysis and synthesis often discard the decoder after pre-training or operate on intermediate acoustic features rather than raw waveforms, failing to preserve the discriminative strength of SSL while enabling high-fidelity reconstruction.

Methodology

The paper proposes OLIVE (Online Latent prediction with Invariant Views and rEconstruction), a framework that jointly optimizes analysis and synthesis objectives within a single pre-training stage. The architecture employs a functional separation between early and late encoder features, illustrated in Figure 1 of the paper.

1. Analysis Branch: View-Augmented Masked Distillation

This branch follows the data2vec 2.0 teacher-student paradigm but introduces view augmentation.

Process: Two independently augmented views of the input waveform ( $x'$ and $x''$ ) are generated. The student processes a masked version of one view, while the teacher processes the unmasked version of the other.
Objective: The student predicts the contextual latent targets provided by the teacher at masked positions.
Invariance: The augmentation family (e.g., mixup, gain perturbation) defines the variability to which the representation must become invariant. This shapes later contextual representations to be robust for downstream discriminative tasks.
Targets: Teacher targets are constructed by instance-normalizing and averaging the top $K$ layers of the teacher encoder.

2. Synthesis Branch: Waveform Reconstruction

This branch acts on earlier encoder features (local features) rather than the final contextual representations.

Process: A neural vocoder (based on HiFi-GAN) is conditioned on the student's local encoder features ( $r'_{1:T}$ ) to reconstruct the original waveform.
Objective: The synthesis branch uses a composite loss including mel-spectrogram reconstruction ( $L_{mel}$ ), adversarial training ( $L_{gen}, L_{disc}$ ), and feature matching ( $L_{fm}$ ).
Constraint: By forcing the early encoder features to retain sufficient information for waveform reconstruction, the model prevents the discarding of fine-grained acoustic details that purely discriminative models might lose.

3. Joint Optimization

The final objective function balances the two goals:
$L_{OLIVE} = L_{analysis} + \lambda_{synthesis} L_{G}^{synthesis}$
Where $\lambda_{synthesis}$ controls the trade-off. The student encoder, prediction head, and generator are updated jointly, while the teacher is updated via Exponential Moving Average (EMA). Crucially, the decoder (vocoder) is retained as part of the pre-trained model, unlike approaches that discard it post-training.

Key Contributions

Unified Framework: Introduction of OLIVE, which jointly optimizes analysis (invariant representation learning) and synthesis (waveform reconstruction) in a single pre-training stage.
Functional Separation Design: A novel architecture where waveform reconstruction constrains early encoder features to preserve acoustic detail, while later contextual representations are shaped primarily by view-augmented masked prediction to ensure invariance for robust downstream analysis.
Performance Validation: Demonstration that this design improves waveform reconstruction and downstream performance on generative and speaker-related tasks while maintaining competitive results on recognition and semantic tasks.

Experimental Results

The authors evaluated OLIVE on the SUPERB benchmark (covering content, speaker, paralinguistic, semantic, and generation tasks) and directly on waveform reconstruction quality using the LibriSpeech 960h dataset.

Downstream Tasks (SUPERB)

Generative & Speaker Tasks: OLIVE showed significant improvements. Specifically, OLIVE-J (the joint model) achieved the best scores in the Speaker and Generation categories (e.g., 83.1% accuracy on Speaker Identification, 12.3 SI-SDRi on Source Separation).
Recognition & Semantic Tasks: OLIVE maintained competitive performance with strong baselines like wav2vec 2.0, HuBERT, and data2vec 2.0 on tasks such as ASR, PR, and Semantic Textual Similarity.
Aggregate Scores: OLIVE-A (Mix+Gain) achieved the highest SUPERBs score, while OLIVE-J achieved the highest WavLMs score, indicating a strong balance between discriminative and generative capabilities.

Waveform Reconstruction

Direct Reconstruction: OLIVE-J's integrated vocoder outperformed standard SSL baselines on perceptual metrics (PESQ: 2.88, STOI: 0.921) and signal-level metrics (SI-SDR: 2.50).
Representation Quality: When using frozen features from OLIVE-J with a separately trained vocoder, the model achieved the best results across all metrics (PESQ: 3.06, SI-SDR: 2.94), suggesting the learned representation retains more acoustic detail than analysis-only models.
Comparison: OLIVE substantially improved reconstructability compared to analysis-only baselines, confirming that the synthesis objective preserves fine-grained acoustic information.

Significance and Claims

The paper claims that OLIVE successfully bridges the gap between discriminative and generative speech learning. By explicitly balancing the two objectives:

It prevents the loss of critical signal-level information required for high-fidelity generation, which often occurs in purely discriminative SSL.
It maintains the strong discriminative power necessary for tasks like ASR and speaker verification.
It produces a single pre-trained model that includes a functional decoder, enabling direct waveform reconstruction without needing to retrain a separate pathway.

The authors note limitations, specifically that the study is restricted to English speech and Base-sized models trained on LibriSpeech. They suggest future work should explore diverse pre-training data, different model sizes, and specific downstream generation tasks like text-to-speech.