F-Actor: Controllable Conversational Behaviour in Full-Duplex Models

This paper introduces F-Actor, the first open, instruction-following full-duplex conversational speech model that efficiently adapts to dynamic contexts and controls various dialogue behaviors through a single-stage training protocol requiring only 2,000 hours of data.

The Gist

Imagine you are at a lively dinner party. The best conversations aren't just about one person talking while the other listens politely. They are full of life: people nodding, saying "uh-huh" while the other speaks, gently interrupting to add a funny story, or jumping in to start a new topic. This is called full-duplex conversation—it's the ability to listen and speak at the same time, just like humans do.

For a long time, computer voice assistants (like Siri or Alexa) have been terrible at this. They act like a strict teacher: "You speak, then I speak, then you speak." If you try to interrupt them, they usually just stop talking and wait for you to finish, which feels robotic and awkward.

This paper introduces F-Actor, a new kind of AI that finally learns to be a real dinner party guest. Here is the breakdown of how it works, using some simple analogies.

1. The Problem: The "Robot at the Party"

Current voice AIs are like actors who have memorized a script but don't know how to improvise.

• The Issue: They can't handle overlapping speech. If you say, "Wait, I think..." while the AI is talking, the AI usually ignores you or cuts itself off awkwardly.
• The Limitation: You also can't tell them how to behave. You can't say, "Hey AI, be really chatty today," or "Be quiet and just listen," or "Sound like a grumpy old man."

2. The Solution: F-Actor (The "Method Actor")

The researchers built F-Actor, which is like a method actor who can take direction instantly.

• The "Director's Note": Before the conversation starts, you can give the AI a set of instructions (a prompt). You can tell it:
  • Who it is: "Sound like a cheerful teenager."
  • What to talk about: "Let's discuss the weather."
  • How to behave: "Interrupt me twice," or "Give me three 'uh-huhs' while I talk."
  • Who starts: "You start the conversation."
• The Result: The AI doesn't just follow a script; it adapts its behavior in real-time based on your instructions, creating a conversation that feels surprisingly human.

3. How They Built It: The "Frozen Brain" Trick

Usually, training a robot to talk like a human requires a massive supercomputer and years of data (like teaching a child from birth). The researchers wanted to do this on a standard university budget.

• The Analogy: Imagine you have a brilliant, pre-trained brain (a Large Language Model) that already knows how to speak and think. Instead of trying to teach that brain how to hear and speak from scratch (which is hard and expensive), they froze the "ears" and "mouth" parts of the brain.
• The Magic: They only taught the "thinking" part (the middle layer) how to connect the ears and mouth to the instructions.
• The Efficiency: Because they didn't have to retrain the whole system, they only needed 2,000 hours of data (a tiny amount for AI) and a few days on standard graphics cards. It's like teaching a professional actor a new role in a week, rather than training a new actor from scratch for ten years.

4. The "Two-Stream" Dance

To handle the "listening while talking" part, the AI uses a clever trick.

• The Analogy: Imagine a dance floor with two lanes. One lane is for the User, and one lane is for the System.
• Usually, AI models try to put both dancers on one tiny lane, which causes them to trip over each other. F-Actor keeps them in separate lanes but lets them move in sync.
• It also uses a special "translator" (called a Codec) that turns sound into digital blocks (like LEGO bricks). This allows the AI to predict the next "brick" of sound instantly, even while it's still listening to the user's voice.

5. The Results: A Natural Conversation

When they tested F-Actor:

• It listened well: It could handle interruptions and back-and-forth chatter without getting confused.
• It followed orders: If you told it to interrupt 5 times, it did (mostly!). If you told it to sound like a specific person, it matched that voice.
• It felt real: The conversations had natural pauses, overlaps, and energy, much like a real human chat.

Why This Matters

This is a big step forward because it moves voice AI from being a tool (like a calculator) to being a partner.

• For the future: Imagine a virtual therapist who knows exactly when to interrupt to offer support, or a language tutor who knows when to jump in to correct your pronunciation without being rude.
• The Catch: The researchers admit the AI isn't perfect yet. It sometimes misses the exact number of interruptions you asked for, and it's currently limited to English. But they released their code to the public, so other scientists can build on this "actor" to make it even better.

In short: F-Actor is the first open-source AI that can be "directed" to act like a real human in a conversation, listening and speaking at the same time, all without needing a supercomputer to train it. It's the difference between a robot reading a menu and a friend joining you for dinner.

Technical Summary

Here is a detailed technical summary of the paper "F-Actor: Controllable Conversational Behaviour in Full-Duplex Models."

1. Problem Statement

Current spoken conversational systems (SCS) suffer from two primary limitations:

• Lack of Full-Duplex Capability: Most systems operate in half-duplex mode (turn-taking), failing to mimic the natural human ability to listen and speak simultaneously. This prevents natural behaviors like backchanneling (e.g., "uh-huh," "yes" while the other speaks) and interruptions, which are crucial for engaging, human-like interaction.
• Limited Controllability: Existing full-duplex models rarely allow users to explicitly control conversational dynamics. They cannot be easily instructed to adopt specific personas, control the frequency of interruptions/backchannels, or decide who initiates the conversation. Furthermore, many state-of-the-art (SOTA) models require massive computational resources, large-scale pretraining, or multi-stage optimization, making them inaccessible to academic researchers.

2. Methodology

The authors propose F-Actor, the first open, instruction-following, full-duplex conversational speech model designed to be trained efficiently under typical academic constraints.

Architecture

• Backbone: The model utilizes Llama3.2-1B-Instruct as the Large Language Model (LLM) backbone.
• Audio Codec: It employs Nemo-nano-codec (NanoCodec), which uses Finite Scalar Quantization (FSQ). Unlike Residual Vector Quantization (RVQ) used in models like Moshi, FSQ produces independent codebooks, allowing all audio tokens for a frame to be predicted in parallel without hierarchical depth-transformers.
• Dual-Stream Processing: The model processes two simultaneous audio streams: User and System.
  • Embeddings: Audio is converted into Discrete Acoustic Units (DAUs). The model uses separate embedding layers for user and system streams to distinguish speakers, rather than extending the LLM vocabulary.
  • Instruction Injection: Instructions are injected via:
    1. Textual Prompts: Encoding the conversation narrative, initiation rules, and desired counts for backchannels/interruptions.
    2. Speaker Embeddings: Extracted from a 5-second audio sample using ECAPA-TDNN and projected into the LLM token space to control voice characteristics.
• Output Heads: Eight linear heads are attached to the LLM to predict DAU logits for the four codebooks of both the user and system streams. During inference, only the system heads are active.
• Text Stream: The model can optionally generate a corresponding text stream alongside audio to improve alignment and coherence.

Training Strategy

• Single-Stage Training: The model is trained in a single stage without large-scale pretraining or multi-stage optimization.
• Frozen Encoder: The audio encoder (NanoCodec) is kept frozen. Only the LLM backbone and the linear DAU prediction heads are trained.
• Data Efficiency: The model is trained on 2,000 hours of data (Behavior-SD dataset), which is synthetic multi-turn dialogue annotated with narratives, backchannels, and interruptions.
• Resource Requirements: Training requires only 4 NVIDIA A100-40GB GPUs for approximately 48 hours.

Key Design Choices

• Audio Delay: The system introduces a delay (up to 2 tokens) between the text and audio streams to improve synchronization and Word Error Rate (WER).
• Word-Level Alignment: Text and audio tokens are aligned at the word level rather than the utterance level to improve coherence.
• Explicit Counting: Instructions specify exact counts for backchannels and interruptions (e.g., "backchannels: 2") rather than vague proportions, enabling controlled evaluation.

3. Key Contributions

1. First Open Controllable Full-Duplex Model: F-Actor is the first publicly released model that allows explicit instruction-following for full-duplex behaviors (voice, topic, backchanneling, interruptions, and initiation).
2. Efficient Training Protocol: Demonstrates that high-quality full-duplex models can be trained with limited resources (2k hours of data, 4 GPUs, 2 days) by freezing the encoder and using a single-stage training approach.
3. Systematic Analysis: The paper provides a comprehensive ablation study on design choices, including the impact of text streams, alignment strategies, audio delays, and quantization methods (FSQ vs. RVQ).
4. Reproducibility: Both the model weights and training code are released to the public.

4. Results

The model was evaluated on general capabilities and instruction-following performance.

• General Capabilities:

  • Speech Quality: Achieved a UTMOS score of 3.42 (out of 5), comparable to SOTA models like Moshi, despite the smaller training set.
  • Coherence: Using word-level alignment and a 2-token audio delay reduced the Word Error Rate (WER) between generated text and audio to 7.59%.
  • Turn-Taking: The model exhibited natural turn-taking statistics (IPUs, pauses, gaps, overlaps) similar to human conversation and baseline models.

• Instruction Following:

  • Initiation: Achieved >99% accuracy in correctly starting or waiting based on the prompt, provided a text stream was used.
  • Voice Consistency: Generated speech maintained high cosine similarity (0.54) with the target speaker embedding, with minimal speaker drift.
  • Narrative Adherence: An LLM judge rated the model's adherence to the conversation narrative highly (2.78/5 for the best configuration), with human evaluation confirming the LLM judge's rankings.
  • Behavioral Control:
    • Backchannels: Showed a significant positive correlation (r=0.54) between requested and generated counts.
    • Interruptions: Showed a lower correlation (r=0.25), likely due to the rarity of interruptions in the training data.
  • Sampling: A temperature of 0.9 provided the best trade-off between dynamic behavior (more interruptions/backchannels) and speech quality.

• Comparison with SOTA:

  • F-Actor performs comparably to Moshi and dGLSM in turn-taking metrics but offers superior controllability via instructions.
  • Unlike MinMo or Voila, F-Actor is open-source and explicitly evaluates instruction-following capabilities.

5. Significance

• Democratization of Research: By proving that controllable full-duplex systems can be trained on academic budgets, F-Actor lowers the barrier to entry for researchers, fostering reproducibility and further innovation in spoken dialogue systems.
• Human-Like Interaction: The ability to control backchanneling and interruptions moves SCS closer to the "full-duplex" ideal of human conversation, where systems can react proactively rather than just reactively.
• Safety and Ethics: The model is restricted to a fixed pool of speaker embeddings and trained on synthetic TTS data, mitigating risks of voice cloning and impersonation.
• Future Directions: The work highlights that while frequency control is possible, the model tends to under-generate rare events (like interruptions) if the training data is sparse, pointing to the need for denser behavioral datasets in future work.

In summary, F-Actor represents a significant step forward in making spoken AI more natural, adaptable, and accessible, bridging the gap between rigid turn-taking systems and fluid, human-like conversation.