Learning Multiple Utterance-Level Attribute Representations with a Unified Speech Encoder

This paper proposes a unified post-training framework that extends speech foundation models to generate multiple arbitrary utterance-level attribute representations, demonstrating its effectiveness through the joint learning of semantic and speaker embeddings for multilingual retrieval and speaker recognition tasks.

The Gist

Imagine you have a super-smart robot assistant named SpeechBot. Before this paper, SpeechBot was like a brilliant translator who could listen to a sentence in French and tell you exactly what it meant in English, but if you asked, "Who is speaking?" or "Are they happy or angry?", SpeechBot would get confused. It was so focused on the meaning of the words that it forgot to notice the voice itself.

This paper introduces a new way to train SpeechBot so it can do both at the same time without getting a headache.

Here is the breakdown of their invention using simple analogies:

1. The Problem: The "One-Track Mind"

Previously, researchers trained SpeechBot using a "Teacher-Student" method.

• The Teacher: A text expert who knows the meaning of every sentence.
• The Student (SpeechBot): Listens to audio and tries to copy the Teacher's understanding of the meaning.

The Catch: To learn the meaning perfectly, the Student had to ignore everything else. It was like a student studying for a history exam who is told, "Ignore the font, the handwriting, and the author's voice; just focus on the facts." As a result, the Student became great at history but terrible at recognizing the author's handwriting (the speaker's voice).

2. The Solution: The "Swiss Army Knife" Encoder

The authors asked: Can we teach SpeechBot to be a history expert AND a handwriting expert at the same time?

They built a Unified Framework. Imagine SpeechBot has a single, powerful brain (the Shared Encoder) that listens to the audio. But instead of having just one output, they attached two different "specialized tools" (branches) to it:

• Tool A (The Semantic Branch): This tool is designed to understand what is being said. It connects to the "Meaning Teacher" (a text model).
• Tool B (The Speaker Branch): This tool is designed to understand who is saying it. It connects to a "Voice Teacher" (a speaker recognition model).

3. How It Works: The "Smart Filter" Analogy

You might think, "If the brain is trying to do two things, won't the tools get in each other's way?"

The authors added a clever Smart Filter (called layer-interpolation weights). Think of the SpeechBot's brain as a multi-story building with 24 floors of information processing:

• The Meaning Tool looks at the middle floors (Floors 13 & 14) to find the "gist" of the story.
• The Voice Tool looks at the top floors (Floors 23 & 24) to catch the unique "fingerprint" of the voice.

The system automatically learns to say: "For this task, I'll use the middle floors. For that task, I'll use the top floors." They don't fight; they just use different parts of the same building.

4. The Results: Best of Both Worlds

The team tested this new SpeechBot on two challenges:

1. The Library Test (Retrieval): Can you find a specific speech clip just by typing a sentence in a different language?
   • Result: Yes! The new SpeechBot was almost as good as the old "Meaning-only" version. It didn't lose its ability to understand language.
2. The Party Test (Speaker Verification): Can you tell if two audio clips are from the same person?
   • Result: Yes! It was nearly as good as the "Voice-only" experts.

The Big Takeaway

Before this paper, you usually had to choose: Do you want a model that understands meaning or a model that recognizes people? You couldn't have both in one efficient package.

This paper proves you can have a single, unified model that acts like a Swiss Army Knife. It can listen to a sentence, tell you what it means in any language, and simultaneously tell you who is speaking, all without needing two separate, bulky computers.

In short: They taught the robot to listen to the words and the voice at the same time, using a smart system that knows exactly which part of its brain to use for each job.

Technical Summary

Here is a detailed technical summary of the paper "Learning Multiple Utterance-Level Attribute Representations with a Unified Speech Encoder."

1. Problem Statement

Speech foundation models (e.g., wav2vec 2.0, HuBERT) trained via self-supervised learning (SSL) excel at generating frame-level acoustic representations. However, many downstream applications (e.g., speech retrieval, speaker verification, cross-modal search) require utterance-level embeddings that capture high-level information.

Current post-training approaches, such as SENSE and SONAR, align speech encoders with text-based semantic spaces to create language-agnostic semantic embeddings. While effective for semantic tasks, these methods suffer from a critical limitation:

• Information Suppression: By optimizing solely for semantic alignment with text, the models tend to suppress paralinguistic information, such as speaker identity, emotion, or speaking style.
• The Core Question: Can a single speech encoder learn to generate multiple distinct utterance-level representations (e.g., both semantic and speaker) simultaneously without degrading the performance of either attribute?

2. Methodology

The authors propose a unified post-training framework that extends the teacher-student knowledge distillation paradigm to support multi-task learning.

Architecture Overview

The framework utilizes a shared SSL speech encoder (initialized with w2v-BERT 2.0) and attaches task-specific branches to it.

1. Shared Encoder: Processes the input audio and produces hidden representations ($H^{(\ell)}$) across multiple layers ($\ell$).
2. Task-Specific Branches: For each target attribute $\tau$ (e.g., semantics, speaker), a dedicated branch is created.
   • Linear Projection: Each layer's representation is projected into a task-specific feature space: $\tilde{H}^{(\ell)}_\tau = H^{(\ell)}W^{(\ell)\top}_\tau + b^{(\ell)}_\tau$.
   • Learnable Layer Interpolation: Unlike previous methods, this framework learns a scalar importance score $s_{\tau,\ell}$ for each layer. These are converted into normalized weights $\lambda_{\tau,\ell}$ via softmax. This allows the model to dynamically select which encoder layers are most relevant for a specific attribute.
   • Aggregation: The weighted sum of projected layers is aggregated using an attribute-specific attention pooling mechanism to produce a single utterance-level embedding.
3. Training Objective:
   • Teacher Models: Frozen pre-trained models provide the target embeddings.
     • Semantic Teacher: BGE-M3 (multilingual text embedding).
     • Speaker Teacher: ECAPA-TDNN (trained on VoxCeleb).
   • Loss Function: The model minimizes the cosine distance between the student's utterance-level embedding and the corresponding frozen teacher's embedding.
   • Optimization: The shared encoder and all task-specific branches are jointly optimized using multi-task learning.

3. Key Contributions

1. Unified Multi-Task Framework: Introduction of a general teacher-student framework capable of learning multiple utterance-level attributes (semantic and speaker) from a single shared encoder.
2. Joint Learning without Degradation: Demonstration that semantic and speaker representations can be learned simultaneously without significantly compromising the performance of either task compared to single-task baselines.
3. Layer Usage Analysis: Provision of empirical evidence showing that different attributes rely on different regions of the shared encoder. The model automatically learns to select complementary layers for different tasks.

4. Experimental Results

The model was trained on the Common Voice 19 dataset (83 languages, 8,250 hours) and evaluated on two main tasks.

A. Semantic Retrieval (Multilingual & Multimodal)

• Tasks: Speech-to-Speech and Speech-to-Text translation retrieval.
• Datasets: VoxPopuli, MTEDx, and FLEURS (including low-resource languages).
• Findings:
  • The multi-task model (Att(sem+spk)) performed nearly identically to the single-task semantic baseline (Att(sem)).
  • It consistently outperformed the Meta SONAR model across various language pairs.
  • In low-resource scenarios (FLEURS), the multi-task model even showed slight improvements over the single-task baseline for specific pairs (e.g., my-en), suggesting that speaker supervision does not hinder semantic generalization.

B. Speaker Verification

• Task: Speaker verification on VoxCeleb1-O.
• Metrics: Equal Error Rate (EER) and MinDCF.
• Findings:
  • The multi-task model achieved an EER of 0.91%, which is nearly identical to the frozen ECAPA-TDNN teacher (0.90%) and slightly better than the single-task speaker baseline (Att(spk) at 0.93%).
  • This confirms that the shared encoder successfully preserved speaker-discriminative information despite being optimized for semantic alignment.

C. Analysis of Layer Interpolation

• Semantic Branch: Concentrated weights on middle layers (peaking at layers 13–14), indicating semantic content is localized in the mid-depth of the network.
• Speaker Branch: Distributed weights broadly across the network, with a gradual increase peaking at the highest layers (23–24).
• Conclusion: The model automatically learns to utilize complementary parts of the encoder for different attributes, minimizing interference.

5. Significance and Future Work

• Efficiency: This approach eliminates the need to train and store separate models for different attributes (e.g., one model for search, another for speaker ID), reducing computational overhead and storage requirements.
• Versatility: It enables the creation of rich, multi-faceted speech representations from a single unified encoder, facilitating complex multimodal applications.
• Future Directions: The authors plan to extend this framework to include additional attributes such as emotion, language identification, and accent, aiming to build even more versatile speech foundation models.

In summary, the paper successfully demonstrates that a unified speech encoder can be post-trained to generate multiple, distinct utterance-level representations (semantic and speaker) simultaneously, achieving state-of-the-art performance in both domains without mutual interference.