⚡ electrical engineering

Covo-Audio Technical Report

This paper introduces Covo-Audio, a 7B-parameter end-to-end Large Audio-Language Model that unifies continuous audio processing and generation to achieve state-of-the-art performance across diverse speech and audio tasks, while offering specialized variants for spoken dialogue and full-duplex interaction alongside a cost-effective strategy for decoupling dialogue intelligence from voice rendering.

Original authors: Wenfu Wang, Chenxing Li, Liqiang Zhang, Yiyang Zhao, Yuxiang Zou, Hanzhao Li, Mingyu Cui, Hao Zhang, Kun Wei, Le Xu, Zikang Huang, Jiajun Xu, Jiliang Hu, Xiang He, Zeyu Xie, Jiawen Kang, Youjun Chen

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenfu Wang, Chenxing Li, Liqiang Zhang, Yiyang Zhao, Yuxiang Zou, Hanzhao Li, Mingyu Cui, Hao Zhang, Kun Wei, Le Xu, Zikang Huang, Jiajun Xu, Jiliang Hu, Xiang He, Zeyu Xie, Jiawen Kang, Youjun Chen, Meng Yu, Dong Yu, Rilin Chen, Linlin Di, Shulin Feng, Na Hu, Yang Liu, Bang Wang, Shan Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to build the ultimate digital companion. You want someone who doesn't just hear your words but understands your tone, your emotions, and your intent. They should be able to interrupt you politely, pause when you're thinking, and respond with a voice that feels warm and human, all while solving complex math problems or telling a funny story.

For a long time, building this "perfect companion" was like trying to assemble a car by gluing together three separate vehicles: a microphone (to hear), a brain (to think), and a speaker (to talk). This "cascaded" approach often led to glitches. The microphone might mishear a word, the brain would get confused by the error, and the speaker would sound robotic.

Enter Covo-Audio.

Think of Covo-Audio not as three glued-together cars, but as a single, super-intelligent organism that was born to speak and listen. It's a 7-billion-parameter "Large Audio Language Model" (LALM) from Tencent that processes sound directly, without needing to convert everything to text first.

Here is how it works, broken down into simple concepts:

1. The "One-Brain" Architecture

Most AI models are like a translator who first writes down what you said, thinks about it, and then reads the translation back to you. This takes time and loses the "soul" of the voice.

Covo-Audio is different. It's like a polyglot who thinks in sound. It takes your raw voice (continuous audio) and processes it directly into its own internal "thoughts," then immediately generates a new voice response. It doesn't stop to write things down. This makes it incredibly fast and allows it to catch subtle emotional cues—like a sigh or a chuckle—that get lost in text.

2. The "Smart Voice" vs. "The Voice" (Decoupling)

One of the biggest headaches in AI is that if you want a specific voice (like a deep, soothing narrator), you usually need hours of that specific person's voice data and hours of them acting out complex dialogues. It's expensive and hard to scale.

Covo-Audio introduces a clever trick called Intelligence-Speaker Decoupling.

The Analogy: Imagine a brilliant actor (the Intelligence) who can play any role, and a recording studio (the Speaker). Usually, you have to hire a new actor for every new voice. Covo-Audio separates them. It trains the "actor" to be incredibly smart using general data, and then it can "wear" any voice mask (from a lightweight TTS system) without losing its smarts.
The Result: You can give the AI a new voice with just a tiny bit of data, and it will still sound natural and stay smart. It's like having a master chef who can cook a gourmet meal using ingredients from any local market, rather than needing a specific farm for every dish.

3. The "Full-Duplex" Dance (Talking While Listening)

Real human conversation isn't a game of "ping-pong" where you wait for the other person to finish before you speak. We interrupt, we say "uh-huh" while they talk, and we pause to think. This is called Full-Duplex.

Older AI models are like a walkie-talkie: "Over." (Wait for response). "Over."
Covo-Audio-Chat-FD is like a real conversation at a dinner party.

It can listen to you while it's still finishing its own sentence.
If you interrupt it ("Wait, I meant..."), it stops immediately, acknowledges you, and pivots.
It knows the difference between a pause where you are thinking and a pause where you are done speaking.
It does this by training on massive amounts of real conversation data, learning the rhythm of human speech rather than just the words.

4. The "Emotional Radar"

Covo-Audio isn't just about logic; it's about empathy. The team trained it on a "mood ring" of data.

If you sound sad, it doesn't just say "I am sorry." It adjusts its tone, speed, and word choice to sound comforting.
If you sound excited, it matches your energy.
It was tested on how well it handles emotions like anger, joy, and anxiety, and it scored higher than many other open-source models, proving it can be a supportive conversational partner.

Why Does This Matter?

The paper shows that you don't need a massive, expensive supercomputer (like a 30-billion parameter model) to get top-tier performance. Covo-Audio, with its compact 7-billion size, beats or matches much larger models in understanding, reasoning, and speaking naturally.

In a nutshell:
Covo-Audio is a step toward the "Jarvis" or "Siri" of our dreams. It's a model that listens to the music of your voice, not just the lyrics. It can interrupt you politely, comfort you when you're down, solve a riddle, and do it all in a single, fluid motion, all while letting you choose any voice you want without needing a massive data collection project.

It's not just an AI that talks; it's an AI that converses.

1. Problem Statement

Current speech interaction systems face a fundamental trade-off between intelligence (deep reasoning), naturalness (emotional/paralinguistic cues), and efficiency (low-latency, full-duplex interaction).

Cascaded Systems: Traditional approaches (ASR $\to$ LLM $\to$ TTS) suffer from information loss, error propagation, and high latency, hindering fluid conversation.
Existing End-to-End Models: While models like GPT-4o or Qwen-Omni have made strides, many rely on "Thinker-Talker" architectures (sequential text reasoning followed by speech generation), which sacrifice direct speech instruction following and make full-duplex interaction (listening while speaking) difficult.
Intelligence-Speaker Coupling: A major bottleneck in deploying end-to-end Large Audio Language Models (LALMs) is the "coupling" of dialogue intelligence with specific voice characteristics. Creating a high-intelligence bot with a specific voice traditionally requires massive amounts of high-quality, speaker-specific dialogue data, which is prohibitively expensive and limits voice customization.
Full-Duplex Limitations: Existing full-duplex models often require fine-grained text-speech alignment or struggle with complex conversational dynamics like interruptions (barge-in) and backchanneling.

2. Methodology

Covo-Audio is a 7B-parameter end-to-end Large Audio Language Model that processes continuous audio inputs and generates audio outputs within a single unified architecture.

Architecture

Backbone: Built upon Qwen2.5-7B-Base.
Encoder: Uses Whisper-large-v3 (50 Hz frame rate) for robust speech understanding, connected to the LLM via a 3-stage downsampling adapter (reducing frame rate to 6.25 Hz).
Tokenizer: A custom WavLM-large based tokenizer with a Vector Quantization (VQ) layer (16,384 codebook size) generating discrete audio tokens at 25 Hz.
Decoder: A two-stage hierarchical decoder using Flow-Matching (to convert discrete tokens to continuous latents) followed by BigVGAN (to reconstruct 24kHz waveforms).
Input/Output: The model natively processes interleaved sequences of continuous acoustic features, discrete audio tokens, and text tokens.

Training Strategy

The model undergoes a rigorous two-stage pre-training followed by targeted post-training:

Stage 1 (Modality Bridging): Freezes the LLM and Whisper encoder; optimizes the adapter using 200k hours of ASR data to align continuous audio features with the LLM latent space.
Stage 2 (Speech-Text Fusion): Unfreezes the entire model. Trains on a massive dataset (2T tokens) using a Hierarchical Tri-modal Interleaving strategy. This integrates continuous audio ( $a_c$ ), discrete tokens ( $a_d$ ), and text ( $t$ ) in both sequential ( $a_c \to t \to a_d$ ) and parallel ( $a_c \to t|a_d$ ) patterns to ensure semantic and acoustic synchronization.
Post-Training (SFT & RL):
- Intelligence Transfer: Uses high-quality text instruction data converted to speech to transfer reasoning capabilities to the audio modality.
- Empathy: Trains on emotion-aware datasets (joy, anger, sadness, etc.) to generate contextually appropriate empathetic responses.
- Full-Duplex (Covo-Audio-Chat-FD): Trained directly on large-scale full-duplex data. It uses a hybrid dual-stream scheme (continuous input, discrete output) and special tokens (THINK, SHIFT, BREAK) to manage listening, speaking, and interrupting states simultaneously.

Key Innovation: Intelligence-Speaker Decoupling

To solve the data scarcity and customization cost issue, the authors propose a technique to decouple dialogue intelligence from voice rendering:

Mechanism: They train a general "intelligence" model using diverse speakers. For voice customization, they take high-quality TTS data (which lacks dialogue context), construct pseudo-conversations, and train the model on this data.
Loss Masking: Crucially, during this training, the text response loss is masked. The model learns to generate the voice and prosody of the TTS speaker while retaining the reasoning capabilities learned from the general dialogue data. This allows flexible voice customization with minimal TTS data.

3. Key Contributions

Hierarchical Tri-modal Interleaving: A novel framework integrating continuous audio, discrete tokens, and text at both phrase and sentence levels, bridging the gap between high-fidelity prosody and robust semantics.
Intelligence-Speaker Decoupling: A technique that enables high-fidelity voice customization and personalization without retraining the core dialogue intelligence, significantly reducing data costs.
Native Full-Duplex Interaction: Evolution of the model into Covo-Audio-Chat-FD, which achieves low-latency, simultaneous listening and speaking with robust handling of turn-taking, pauses, interruptions, and backchanneling, trained via large-scale pre-training rather than sequential post-training.
Compact yet Powerful: Demonstrates that a 7B-parameter model can achieve state-of-the-art (SOTA) performance in audio tasks, rivaling much larger systems (e.g., 30B+ models) and closed-source giants.

4. Results

Covo-Audio was evaluated across speech-text modeling, spoken dialogue, speech/audio understanding, and full-duplex interaction.

Speech-Text Modeling: Outperforms comparable 7B models (e.g., GLM-4-Voice-Base, Moshi) on ASR, TTS, and story continuation tasks (A2A-tSC, sBLIMP, sWUGGY).
Spoken Dialogue (URO-Bench & VCB Bench):
- Achieves SOTA or competitive results in Chinese and English reasoning, instruction following, and robustness against noise/accents.
- Empathy (VStyle Benchmark): Achieves SOTA scores in Mandarin for anger, sadness, and anxiety recognition, outperforming many closed-source models.
Full-Duplex (Covo-Audio-Chat-FD):
- Substantially outperforms open-source full-duplex baselines (Moshi, Freeze-Omni) in turn-taking (99.7%), pause handling (97.6%), and interruption handling (96.81%).
- Maintains reasoning capabilities comparable to the half-duplex version.
Audio Understanding (MMAU & MMSU):
- MMAU: Achieves 75.30% average (2nd among open-source, best among 7B models), excelling in music understanding (76.05%).
- MMSU: Achieves 66.64% average accuracy, the highest score among all evaluated systems (both open and closed-source), significantly outperforming the next best (Audio Flamingo 3 at 61.40%).
Decoupling Efficiency: The Covo-Audio-Chat-TTS variant (using decoupled voices) achieves dialogue performance comparable to the standard model, validating the technique's effectiveness.

5. Significance

Paradigm Shift: Covo-Audio validates that end-to-end LALMs can achieve high-level semantic reasoning and natural interaction without the "Thinker-Talker" bottleneck, offering a more direct and efficient path for conversational AI.
Scalability & Efficiency: It proves that 7B-scale models, when carefully architected and trained, can match or exceed the performance of significantly larger models, making high-quality audio AI more deployable and cost-effective.
Practical Deployment: The Intelligence-Speaker Decoupling strategy solves a critical industry pain point, allowing developers to create personalized, high-intelligence voice assistants without the prohibitive cost of collecting massive speaker-specific dialogue datasets.
Open Source: By open-sourcing the Covo-Audio-Chat model and inference pipeline, the authors provide a strong baseline for the community to advance the field of unified audio-language intelligence.

Resources:

GitHub: https://github.com/Tencent/Covo-Audio
HuggingFace: https://huggingface.co/tencent/Covo-Audio-Chat