Speech-Omni-Lite: Portable Speech Interfaces for Vision-Language Models

Speech-Omni-Lite is a cost-efficient framework that extends frozen visual-language backbones with lightweight, trainable speech modules and a novel data construction strategy to achieve high-performance spoken QA comparable to massive omni-models, using only thousands of hours of training data.

The Gist

Imagine you have a brilliant, world-class Librarian (the Vision-Language Model). This Librarian is incredibly smart: they can read books, analyze paintings, solve complex riddles, and write stories. However, there's one big problem: they are mute and deaf. They can only communicate via text on a screen.

To make them talk and listen, most researchers try to build a giant, expensive new brain around the Librarian, or they try to retrain the Librarian from scratch to learn a new language (speech). This costs millions of dollars in computer power and requires massive libraries of recorded conversations.

Enter SPEECH-OMNI-LITE.

Think of SPEECH-OMNI-LITE not as rebuilding the Librarian, but as giving them a high-tech, portable headset and a translator.

Here is how it works, broken down into simple parts:

1. The "Plug-and-Play" Headset (The Frozen Backbone)

The genius of this paper is that they never touch the Librarian's brain. The original model stays exactly as it is, frozen in time. This means the Librarian doesn't forget how to read or look at pictures.

Instead, they attach two tiny, lightweight modules:

• The Ear (Speech Projector): A small adapter that takes sound waves, turns them into a code the Librarian understands, and feeds them in.
• The Mouth (Speech Token Generator): A small adapter that takes the Librarian's text thoughts and turns them back into sound waves.

Because these adapters are tiny and the Librarian is frozen, you don't need a supercomputer to train them. It's like buying a $50 headset for a $5,000 computer instead of buying a whole new $5,000 computer.

2. The "Magic Recipe" for Data (QTATS)

Usually, to teach a computer to have a conversation, you need thousands of hours of people recording themselves asking and answering questions out loud. This is expensive and hard to find.

The authors came up with a clever shortcut called QTATS (Question-Text, Answer-Text, Answer-Speech).

• The Problem: They didn't have enough recorded conversations.
• The Solution: They took existing recordings of people reading text (like audiobooks or news) and used a smart AI to write the questions that would lead to those answers.
• The Analogy: Imagine you have a library of recorded speeches. Instead of hiring actors to record new conversations, you use a robot to write the questions that would have prompted those speeches. Now, you have a full conversation (Question + Answer) without recording a single new second of audio.

3. The Results: Small Data, Big Smarts

The paper shows that even with only thousands of hours of data (instead of the millions others use), their system performs almost as well as the giant, expensive models.

• Efficiency: They achieved results comparable to models trained on 10x more data.
• Portability: Because the "headset" is so small and smart, you can take the same headset and put it on a small Librarian or a giant Librarian, and it works great on both. You don't need to retrain the whole system every time you change the brain size.

Why Does This Matter?

• Democratization: You don't need a billion-dollar budget to build a talking AI. Small research teams can now do it.
• Speed & Cost: It saves massive amounts of electricity and time.
• Accessibility: It makes it easier to give voice to existing AI tools, helping people who rely on voice commands interact with computers more naturally.

In short: SPEECH-OMNI-LITE is the ultimate "plug-and-play" upgrade. It takes a silent, visual AI and gives it a voice and ears, using a clever trick to learn from existing data, all without breaking the bank or the original model's brain.

Technical Summary

Here is a detailed technical summary of the paper "SPEECH-OMNI-LITE: Portable Speech Interfaces for Vision–Language Models".

1. Problem Statement

While large-scale "omni-models" (unifying text, vision, speech, and video) demonstrate impressive capabilities, they face two critical barriers:

• High Computational & Data Costs: Training these models requires massive multimodal datasets (millions of hours of speech) and substantial computational resources, making them inaccessible to many research groups.
• Catastrophic Forgetting & Lack of Portability: Existing methods often require fine-tuning the entire foundation model to align new modalities. This risks degrading the model's original vision-language (VL) performance (catastrophic forgetting) and creates speech modules that are tightly coupled to specific backbones, making them difficult to transfer to new models without retraining.
• Data Scarcity: High-quality spoken Question-Answering (QA) corpora are expensive to collect, hindering effective speech generation training.

Core Question: How can we extend a well-trained VL foundation model with speech understanding and generation capabilities in a cost-efficient manner, minimizing data acquisition and computational expense while preserving the backbone's native performance?

2. Methodology: SPEECH-OMNI-LITE

The authors propose SPEECH-OMNI-LITE, a framework that augments a frozen pre-trained VL backbone with lightweight, trainable speech modules. The architecture follows a "Thinker-Talker" paradigm but with a unique "frozen backbone" constraint.

A. Model Architecture

The system consists of five main components:

1. Pre-trained Discrete Speech Tokenizer: Converts incoming audio into discrete tokens (12.5 Hz) using a HuBERT-based encoder and Finite Scalar Quantization (FSQ). It operates in a streaming, causal manner.
2. Trainable Speech Projector: A lightweight module (MLP + LLaMA decoder layers) that maps discrete speech tokens into the VL backbone's input embedding space.
3. Frozen VL Backbone: The core model (e.g., Qwen3-VL) remains fully frozen during training. It processes the projected speech tokens alongside visual/text inputs to generate hidden states.
4. Trainable Speech Token Generator: An encoder-decoder module that converts the VL backbone's hidden states into discrete speech tokens. It utilizes Multi-Token Prediction (MTP) to accelerate decoding.
5. Pre-trained Speech De-tokenizer: Converts the generated discrete tokens back into waveforms (based on F5-TTS with a Cross-Attention DiT backbone).

B. Data Construction Strategy: QTATS

To solve the scarcity of spoken QA data, the authors introduce a novel Question–Text Answer–Text–Speech (QTATS) construction pipeline:

• Source: Utilizes abundant existing ASR (Automatic Speech Recognition) speech-text pairs.
• Process:
  1. Take an ASR pair $(x_{sph}, y_{txt})$.
  2. Treat $y_{txt}$ as the Answer Text.
  3. Use an LLM to generate a corresponding Question Text ($q_{txt}$) conditioned on the answer.
  4. Retain the original speech $x_{sph}$ as the Answer Speech.
• Result: A triplet $(q_{txt}, a_{txt}, a_{sph})$ that simulates a spoken QA scenario without requiring dedicated spoken recordings.
• Training Mechanism: An auxiliary Text Projector is used during training to map the generated question text ($q_{txt}$) into the VL backbone, allowing the model to learn speech generation conditioned on QA-style hidden states, even though the supervision comes from ASR data.

C. Training Procedure

The training is divided into two independent phases, keeping the VL backbone frozen throughout:

1. Speech Projector Training:
   • Stage 1: Align speech tokens with text transcripts (ASR task).
   • Stage 2: Adapt to speech QA using QTATS data.
2. Speech Token Generator Training:
   • Train an auxiliary text projector on QTATS to generate appropriate VL hidden states.
   • Train the speech token generator to map these hidden states to the target speech tokens (from the original ASR audio).
   • Note: The text projector is discarded after training.

3. Key Contributions

1. Speech Extension without Catastrophic Forgetting: The framework successfully adds speech capabilities to a pre-trained VL backbone while keeping all backbone parameters frozen, ensuring native vision-language performance is preserved.
2. Lightweight and Transferable Modules: The speech projector and generator are compact and trainable. Crucially, a projector trained on a smaller backbone (e.g., 8B) can be transferred to larger backbones (e.g., 32B) with minimal adjustment (only the final linear layer), demonstrating strong portability.
3. Low-Cost Data Strategy (QTATS): The paper is the first to convert ASR speech-text pairs into spoken QA data via reverse question generation. This eliminates the need for costly large-scale spoken QA datasets or TTS synthesis for training the speech generator.

4. Experimental Results

• Performance vs. Cost: SPEECH-OMNI-LITE achieves spoken QA performance comparable to omni-models trained on millions of hours of speech, despite using only ~4,000 hours of ASR data (approx. 1/10th the training cost).
• Speech Understanding (S→T): On ASR benchmarks (Chinese/English), it performs competitively, though slightly below massive omni-models due to limited ASR data. However, it excels in Speech-to-Text QA tasks (e.g., LLaMA Questions, AlpacaEval).
• Speech Generation (S→S): The model achieves competitive Speech-to-Speech QA accuracy and content consistency (low WER between generated speech and text) despite being trained on "manufactured" QA data rather than genuine spoken QA.
• Transferability: When transferring the speech projector from an 8B backbone to a 32B backbone, performance improved consistently, proving the method's scalability across model sizes.
• Latency: The system supports streaming interaction with a fixed latency of ~54.3 ms per 640ms audio chunk.

5. Significance and Impact

• Democratization of Omni-Models: By decoupling speech capabilities from the heavy backbone training, this approach lowers the barrier for smaller research groups to build multimodal systems.
• Resource Efficiency: Reduces the data requirement for speech generation training by an order of magnitude, leading to lower carbon footprints and energy consumption.
• Modular Design: The "plug-and-play" nature allows for easy integration of speech into various existing VL models without retraining the core intelligence, facilitating rapid deployment and iteration.
• Ethical Safety: Since the backbone is frozen, existing safety alignments and safety filters within the original VL model are preserved, mitigating risks associated with catastrophic forgetting of safety protocols.

In summary, SPEECH-OMNI-LITE demonstrates that high-quality, portable speech interfaces for vision-language models can be achieved through lightweight adapters and clever data construction, challenging the notion that massive scale is the only path to multimodal intelligence.