Technical Summary: Lychee-FD – Hierarchical Acoustic-Semantic Modeling for Full-Duplex SLMs

1. Problem Statement

The development of seamless, high-performance, native intelligent full-duplex Spoken Language Models (SLMs) remains a critical challenge. While recent "Omni-modal" models have advanced voice interaction, they are fundamentally constrained by modality interference. This interference manifests as severe knowledge degradation and compromised semantic integrity, causing full-duplex SLMs to feel unnatural and unintelligent.

Existing paradigms face a dilemma (illustrated in Figure 1):

• Native End-to-End models sacrifice accuracy for efficiency, suffering from significant knowledge degradation when adapting half-duplex bases to full-duplex tasks (e.g., Moshi reports a -12.7% drop in speech intelligence).
• Thinker-Talker architectures preserve knowledge but introduce prohibitive inference costs and latency due to intricate multi-stage training and separate generation modules.

The core research question is: How can modality interference be resolved to simultaneously achieve high inference efficiency and robust knowledge retention in full-duplex SLMs?

2. Root Cause Analysis: Optimization Dynamics

Through an exhaustive fine-grained analysis of model optimization dynamics, the authors identify the root cause of modality interference as inherent gradient conflicts between acoustic and semantic modeling when forced to share a deep parameter space.

Two specific phenomena were quantified:

1. Optimization Divergence: By calculating the layer-wise gradient cosine similarity between text and speech objectives, the authors found that while shallow layers (0-9) exhibit synergistic optimization (positive similarity), deeper layers become increasingly orthogonal and negative. Forcing a unified parameter set to resolve these opposing gradient directions fractures the optimization trajectory.
2. Semantic Dilution: Full-duplex SLMs typically align sparse text tokens (3Hz) with dense audio frames (25Hz) using padding tokens. The authors observed that this "Aligned" setting consistently suppresses the magnitude of semantic gradients compared to "Dense" text supervision. Consequently, the optimization landscape becomes dominated by acoustic reconstruction, effectively suppressing semantic modeling and leading to knowledge degradation.

3. Methodology: Lychee-FD

To mitigate these issues, the authors introduce Lychee-FD, a native end-to-end full-duplex framework featuring two primary architectural innovations:

A. Hierarchical Parameter Separation

Addressing the optimization divergence in deep layers, Lychee-FD employs a hierarchical Transformer architecture:

• Shared Backbone: The shallow layers (0-9) remain a unified Transformer backbone to leverage shared low-level feature processing.
• Specialized Heads: In deeper layers, parameters are physically disentangled into three independent heads operating in parallel:
  • Semantic Head: Dedicated to text generation.
  • Acoustic Head: Dedicated to speech synthesis.
  • Control Head: Dedicated to interaction management (e.g., start/stop signals).
    This separation allows the model to articulate high-fidelity acoustic responses without corrupting underlying semantic modeling, while maintaining the original model depth to preserve inference efficiency.

B. Semantic Alignment Channel

To counteract semantic dilution caused by sparse alignment, Lychee-FD introduces a dedicated semantic alignment channel.

• During training, the model generates coherent internal monologues (text streams) alongside the acoustic stream.
• These monologues serve as a "semantic anchor," utilizing continuous textual supervision to maintain high-magnitude gradient flow for the language modeling objective.
• This ensures robust knowledge retention even when the acoustic stream is dense and the text stream is sparse.

4. Experimental Results

Extensive experiments on multiple benchmarks demonstrate that Lychee-FD significantly advances the state of the art.

Spoken Question Answering (Table 1)

• Lychee-FD achieved the highest average accuracy in both Speech-to-Text (S→T) and Speech-to-Speech (S→S) settings.
• Compared to the previous SOTA native full-duplex model (Fun-Audio-Chat), Lychee-FD improved S→S accuracy by 7.4% and S→T by 8.8%.
• It surpassed system-level pipelines (e.g., VITA-1.5) in reasoning capabilities while maintaining a perfect 100% Takeover Rate (TOR), indicating seamless real-time interaction.
• Ablation Studies: Removing the semantic alignment channel caused a ~5.5% drop in accuracy (confirming semantic dilution). Removing parameter separation caused a catastrophic drop in speech accuracy to 27.6% (confirming modality interference).

Full-Duplex Chatting (Table 2)

• Lychee-FD achieved state-of-the-art performance on 10 out of 11 interaction metrics across FDBench, FullDuplexBench 1.0, and 1.5.
• On FullDuplexBench 1.5, it delivered a 28.5% average improvement over system-level baselines.
• It demonstrated superior capability in managing interruptions, backchannels, and dynamic turn-taking without aggressive, indiscriminate speech.
• Latency: The model achieved the lowest first-response and interruption-response latencies (e.g., 570ms stop latency on FullDuplexBench 1.5), proving that the hierarchical separation introduces no computational overhead.

Speech Generation (Figure 4)

• Lychee-FD achieved the highest UTMOS score (4.50), surpassing its half-duplex backbone (StepAudio-2-mini) and other native models like Moshi.
• This indicates that the hierarchical separation preserves fine-grained prosodic details often lost in shared architectures.

5. Significance and Claims

The paper claims two key firsts:

1. Root Cause Elucidation: It is the first work to uncover and explain that modality interference in full-duplex SLMs stems from inherent gradient conflicts (divergence and dilution) within shared deep parameter spaces. This finding is validated across different architectures (StepAudio and Moshi).
2. Practical Solution: It presents the first fully native intelligent full-duplex framework that successfully reconciles the efficiency-intelligence trade-off. By decoupling conflicting modalities via hierarchical parameter separation and enforcing semantic coherence, Lychee-FD enables seamless, high-performance full-duplex interaction without the latency penalties of Thinker-Talker models or the knowledge loss of standard End-to-End models.

The authors conclude that this work substantially pushes forward the frontier of full-duplex SLM research, paving the way for the next generation of natural, fluid, and immersive human-machine interaction.