Unit-Based Agent for Semi-Cascaded Full-Duplex Dialogue Systems

Technical Summary: Unit-Based Agent for Semi-Cascaded Full-Duplex Dialogue Systems

Problem Statement
Real-world voice interaction remains a significant challenge, particularly in achieving natural, full-duplex communication where users and machines can speak concurrently. Existing approaches generally fall into two paradigms: end-to-end models and cascaded pipelines. While end-to-end models preserve paralinguistic cues (such as emotion and prosody) by processing speech jointly, they demand heavy data and extensive training. Conversely, traditional cascaded pipelines (combining ASR, LLM, TTS, and speaker verification) are easier to deploy and integrate but suffer from latency and the loss of acoustic cues due to their multi-stage design. Furthermore, existing cascaded systems often struggle to effectively manage turn-taking signals and user utterance states within a unified decision space.

Methodology
The authors propose a train-free, plug-and-play framework that decomposes complex dialogue into minimal "conversational units." This system is instantiated as a semi-cascaded full-duplex dialogue system centered around a Multimodal Large Language Model (MLLM). The framework operates through four modular components:

1. Audio Acquisition Module: Continuously captures the user's speech stream. It employs Voice Activity Detection (VAD) to monitor activity and optionally uses Speaker Verification (SV) to filter non-target voices.
2. Context Module: Utilizes an asynchronous Automatic Speech Recognition (ASR) system to generate transcripts. These transcripts are cached and used as semantic augmentation in the next decision cycle, rather than being the primary input for the core logic.
3. Decision Module (MLLM): Powered by a multimodal LLM (specifically Qwen3-Omni in the experiments), this module takes the raw audio stream and state-specific prompts as input. It unifies turn-taking signals and user utterance states into a single decision space. The MLLM issues one of two actions to drive state transitions:
   • Continue: Keeps the system in the current state (listening or speaking).
   • Switch: Transitions the system to the other state.
   • Logic: In the listen state, the model determines if an utterance is semantically complete (switch to speak) or incomplete (continue listening). In the speak state, it distinguishes between user backchannels (continue speaking) and genuine interruptions (switch back to listen).
4. Speech Generation Module: Synthesizes the assistant's response using Text-to-Speech (TTS) with streaming support. If an interruption occurs during playback, the TTS is immediately terminated, and the system returns to the listen state to begin a new unit.

The system is designed to be "semi-cascaded" because it relies on the MLLM's built-in multimodal reasoning to process audio directly, using ASR transcripts only as contextual augmentation rather than as a mandatory intermediate step.

Key Contributions

• Unit-Based Decomposition: The framework breaks dialogue into discrete units, allowing the system to process each unit independently and predict transitions dynamically.
• Unified Decision Space: It unifies turn-taking control and utterance state classification (e.g., complete, incomplete, backchannel, interruption) into a single MLLM-driven decision process.
• Train-Free Operation: The system requires no fine-tuning of the underlying MLLM, operating in a plug-and-play manner with existing speech components (VAD, ASR, TTS).
• Semi-Cascaded Architecture: By leveraging the MLLM's ability to reason over raw audio, the system captures paralinguistic cues while maintaining the modularity of a cascaded pipeline.

Experimental Results
The framework was evaluated on the HumDial dataset as part of the Human-like Spoken Dialogue Systems Challenge (Track 2: Full-Duplex Interaction).

• Performance: The proposed system ranked second among all teams on the test set.
• Metrics: On the development set, the system achieved a First Response Delay of 1.528s (compared to a baseline of 2.753s), an Interruption Total Score of 50.0, and a Total Delay of 1.698s. On the test set, the Interruption Total Score improved to 57.8, with a Total Delay of 1.632s.
• Implementation: The system was built using open-source components (Silero VAD, CAM++, Paraformer ASR, Qwen3-Omni MLLM, and IndexTTS) and deployed via WebSocket and FastAPI to simulate real-world interactions.

Significance and Claims
The paper claims that this framework demonstrates the effectiveness of using an MLLM to directly control state transitions at the unit level, replacing the traditional ASR-to-LLM cascade. By reasoning directly over user audio, the system successfully captures paralinguistic cues, reduces response latency, and produces more informed responses. The authors emphasize that this approach achieves state-of-the-art semantic and interaction-state inference while remaining modular and compatible with existing speech technologies. The work suggests that MLLMs can serve as the central "brain" for full-duplex dialogue, simplifying the pipeline while enhancing naturalness and responsiveness.