💬 NLP

Liberating LLM Capabilities in Full-Duplex Speech Models

This paper introduces Listen-Write-Speak (LWS), a novel text-first tri-channel paradigm that enables full-duplex speech models to simultaneously listen, write visible structured text, and speak in real-time using a shared autoregressive LLM without architectural changes, thereby unlocking text-native capabilities like code generation and structured reasoning while maintaining conversational responsiveness.

Original authors: Luoyuan Zhang, Bokai Xu, Junbo Cui, Weiyue Sun, Yingjing Xu, Hanyu Liu, Yuan Yao

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Luoyuan Zhang, Bokai Xu, Junbo Cui, Weiyue Sun, Yingjing Xu, Hanyu Liu, Yuan Yao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are in a meeting with a brilliant colleague who is also a master typist. Currently, most AI voice assistants are like colleagues who can only speak. If you ask them to write a complex computer program or draw a detailed chart, they have to describe it out loud: "Okay, start with a function, then define a variable..." This is slow, hard to follow, and you have to write it down yourself if you want to keep it.

The paper introduces a new way for AI to talk called Listen-Write-Speak (LWS). Think of this new AI as a colleague who does three things at once, perfectly synchronized:

Listen: They hear you talking in real-time, even while they are talking back to you.
Write: As they listen and think, they are simultaneously typing out their thoughts, code, or structured notes on a screen right in front of you.
Speak: At the same time, they are speaking a natural, conversational summary of what they are typing.

The "Three-Lane Highway" Analogy

The authors describe this as a tri-channel paradigm. Imagine a three-lane highway where traffic flows in one direction (the conversation timeline):

Lane 1 (Listening): This lane is always open. The AI never stops listening to you, even when it's speaking. This allows for "full-duplex" interaction, meaning you can interrupt the AI, and it won't get confused or stop processing.
Lane 2 (Visible Writing): This is the "thinking" lane. Instead of keeping its thoughts hidden inside a black box, the AI types them out on a screen. If you ask for a Python script, the code appears on the screen instantly. If you ask for a meeting summary, a neat table appears. This is the "first-class" output, meaning it's the main way the AI shows you its work.
Lane 3 (Speaking): This lane carries the voice. The AI reads a simplified, spoken version of what it's typing so you can hear the answer while you read the details.

How It Works (The Magic Trick)

The paper claims this doesn't require building a brand-new, complicated robot brain. Instead, they used a clever Token Schema.

Think of the AI's internal language as a set of Lego bricks. Usually, these bricks only build speech. The researchers invented special "instruction bricks" (like <unit>, <ls_cogn>, and <speak>) that tell the AI: "Right now, you are in a 1-second time block. In this block, you must listen to this audio, type this text, and say this sentence."

By organizing the conversation into these tiny, 1-second "time blocks" (Units), the AI can juggle all three tasks without getting tangled up. It's like a conductor telling an orchestra: "For the next second, the violin plays, the flute plays, and the percussionist taps the drum, all at the exact same time."

What They Found

The researchers tested this new AI against other voice models and found:

It's a better multitasker: On tests measuring how well the AI understands and reasons while talking, LWS scored higher than models that only speak or only think before speaking.
It's consistent: The AI didn't say one thing and write another. In 92.6% of cases, what it spoke matched perfectly with what it wrote.
It handles interruptions well: Because it keeps listening while it speaks, if you interrupt it, it can handle the change smoothly without crashing or waiting for you to finish.

The Bottom Line

The paper argues that by letting the AI write what it thinks while it speaks, we get the best of both worlds: the speed and natural feel of a voice conversation, combined with the precision and structure of written text (like code or data tables). It turns the AI from a "talking head" into a "collaborative partner" that shows its work as it goes.

Note on Limitations: The authors admit that because the AI has to do all this in real-time (every second), it might not be the best at extremely long, complex planning tasks that require hours of deep thinking before speaking. Also, currently, it only accepts voice as input, not images or files.

Technical Summary: Listen-Write-Speak (LWS)

Problem Statement

Speech-based Large Language Models (LLMs) currently face a fundamental bottleneck: their outputs are constrained to the spoken modality, which suppresses text-native capabilities such as code generation, structured analysis, and multi-step reasoning. Existing approaches to speech LLMs typically treat text as a hidden intermediate state or a subordinate modality used solely to improve spoken responses. This limits user-facing outputs to what can be verbalized, forcing users to manually transcribe complex structured data (e.g., code blocks, tables) or linearizing structured information into difficult-to-revise spoken lists. Furthermore, current models often lack genuine full-duplex capabilities (listening while speaking) or fail to maintain a persistent, user-visible reasoning channel during interaction.

Methodology: Listen-Write-Speak (LWS)

The authors propose Listen-Write-Speak (LWS), a text-first, tri-channel interaction paradigm. Unlike prior works that rely on complex architectural modifications or separate models, LWS implements this behavior entirely through a Token Schema within a standard autoregressive Transformer.

Core Architecture

The system operates on a shared causal attention context where three channels run concurrently within discrete temporal intervals called Units (set to 1 second in experiments):

Listening (Perception) Channel: Continuously ingests user audio streams (represented as discrete audio tokens) throughout the interaction, enabling full-duplex capability.
Visible Writing (Cognition) Channel: Always active. It externalizes the model's text-native cognition as free-form, user-visible text.
- During Listening: It builds incremental reasoning (e.g., partial understanding, disambiguation) based on the audio heard so far.
- During Speaking: It emits structured artifacts (code, tables, derivations) in parallel with natural speech.
Speaking (Expression) Channel: Generates natural spoken language as a real-time oral response. This channel is active only during the response phase.

Token Schema

LWS utilizes special tokens to delineate channel boundaries within each Unit, requiring no architectural changes to the underlying LLM:

Listening Units: <unit> $\rightarrow$ audio $\rightarrow$ <|lc|> (ls cogn start) $\rightarrow$ writing $\rightarrow$ <|/lc|> (ls cogn end) $\rightarrow$ </unit>.
Speaking Units: <unit> $\rightarrow$ audio $\rightarrow$ <|spk|> $\rightarrow$ speak $\rightarrow$ <|eos|> $\rightarrow$ <|rc|> (reply cogn start) $\rightarrow$ writing $\rightarrow$ <|/rc|> (reply cogn end) $\rightarrow$ </unit>.

The separation of listening-phase cognition (ls cogn) and reply-phase cognition (reply cogn) is explicit to prevent information leakage and simplify next-token prediction, as they condition on different information states.

Data Construction

Since public corpora lack per-second cognitive annotations aligned to audio timelines, the authors developed a two-stage data pipeline:

Offline Cognitive Synthesis: A strong teacher LLM synthesizes three text streams from standard QA pairs: a streaming reasoning chain (for ls cogn), a voice response (for speak), and the original structured response (for reply cogn). A strict causal constraint ensures reasoning at time $t$ only uses input heard up to $t$ .
Online Timeline Construction: The synthesized streams are aligned with real audio recordings and CTC-based character-level alignments to construct Unit sequences. This includes random silence intervals and interruption augmentation to simulate natural conversational pacing and overlap.

Key Contributions

Paradigm Shift: Formulation of LWS as a text-first tri-channel paradigm where a single autoregressive LLM simultaneously listens, writes visible text, and speaks, treating visible writing as a first-class output channel.
Token Schema Implementation: Realization of this complex behavior entirely through special tokens in a standard Transformer, avoiding the need for additional decoders or cross-channel fusion modules.
Data Pipeline: Development of a two-stage pipeline that synthesizes causally consistent, per-second cognitive annotations to train the model on streaming interaction.
Empirical Validation: Demonstration of strong full-duplex interaction and high-quality structured output without sacrificing real-time responsiveness.

Experimental Results

The model was evaluated on several benchmarks using a MiniCPM-V architecture with a Qwen3-8B backbone.

Spoken Understanding and Reasoning (URO-Bench): LWS achieved the best scores on Chinese Pro splits (Average: 84.6) and Chinese Basic understanding/reasoning (92.5/85.9). It consistently outperformed its internal ablations (models without writing while listening or speaking), validating the utility of the visible writing channel in both phases.
Reply Quality (VoiceBench AlpacaEval): LWS scored 4.72, outperforming all listed open-source baselines and trailing only GPT-4o-Audio (4.78). This score reflects the quality of the Visible Writing channel.
Channel Consistency: The model achieved a 92.6% consistency rate between spoken and written outputs, indicating that the dual-channel approach does not compromise coherence.
Full-Duplex Interaction (Full-Duplex-Bench): LWS demonstrated competitive turn-taking (0.97 Candor TOR at 0.48s latency) and strong interruption handling (4.02 GPT-4o quality at 0.65s latency), recovering from overlaps without significant delays.
Training Convergence: All three channels (listening writing, speaking, reply writing) converged smoothly with low variance, confirming the feasibility of joint tri-channel training in a single autoregressive model.

Significance and Claims

The paper claims that visible writing can serve as a first-class output channel for speech interaction without sacrificing real-time responsiveness. By unifying full-duplex listening, visible writing, and real-time speaking, LWS allows speech models to leverage their full text-native capabilities (code, structured analysis) in a multimodal setting. The authors emphasize that this approach requires no architectural modifications, making it scalable and reusable.

The study concludes that while LWS is optimized for responsiveness, it currently faces limitations in reasoning depth (due to real-time constraints) and input modality (speech-only input). However, it establishes a new baseline for how speech-based LLMs can interact with users, moving beyond linearized speech to include persistent, inspectable, and structured intermediate outputs.