💬 NLP

TiCo: Time-Controllable Training for Spoken Dialogue Models

The paper introduces TiCo, an efficient post-training method that equips spoken dialogue models with time awareness through Spoken Time Markers, enabling them to generate responses with controllable durations while maintaining high quality.

Original authors: Kai-Wei Chang, Wei-Chih Chen, En-Pei Hu, Hung-yi Lee, James Glass

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kai-Wei Chang, Wei-Chih Chen, En-Pei Hu, Hung-yi Lee, James Glass

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a very smart, but slightly chaotic, voice assistant. You ask it a question, and it starts answering. But here's the problem: you told it, "Please keep your answer under 15 seconds," and it just keeps rambling on for a minute. Or you ask for a quick 5-second tip, and it gives you a 30-second lecture.

This happens because most current AI voice models are like writers who have never learned to check a watch. They know what to say, but they have no idea how long it will take to say it. They don't understand that "15 seconds" is a hard limit, not just a suggestion.

Enter TiCo (Time-Controllable Training). Think of TiCo as a personal time-manager coach for these AI voice models. It teaches the AI how to pace itself, stop when the clock runs out, and stretch its answer if it has extra time.

Here is how TiCo works, broken down into simple concepts:

The Problem: The "Word Count" Trap

In the past, if you wanted an AI to write a short text, you might tell it, "Write 50 words." But in the spoken world, counting words doesn't work.

The Analogy: Imagine telling a runner, "Run 50 steps." If they are walking slowly, 50 steps might take 30 seconds. If they are sprinting, it might take 5 seconds.
The Reality: Speech is messy. A single word like "banana" takes longer to say than "cat." If you have a fast talker or a slow talker, the time changes. Existing AIs try to guess the time by counting words, and they almost always get it wrong.

The Solution: TiCo's Two-Step Training

TiCo teaches the AI a new superpower: Time Awareness. It does this in two stages, like training an athlete.

Stage 1: The "Time-Stamping" Practice

First, the AI is asked to have a normal conversation with itself. But this time, it has to wear a digital stopwatch on its wrist.

The Analogy: Imagine the AI is writing a script. As it writes every sentence, it has to pause and write a note like: "I just finished this sentence, and my stopwatch says 3.5 seconds have passed."
The Result: The AI learns to associate the words it is generating with the actual time they take to speak. It builds an internal clock. It learns that saying "The ocean is deep" takes about 2 seconds, while "The Mariana Trench is the deepest part of the ocean" takes about 4 seconds.

Stage 2: The "Coach's Whistle" (Reinforcement Learning)

Now that the AI knows how to read its watch, we give it a specific challenge. We say, "Answer this question, but you must stop exactly at 15 seconds."

The Analogy: This is like a coach standing next to the runner with a whistle. If the AI starts to run too fast (finish too early), the coach gives a small reward. If it runs too slow (goes over time), the coach gives a penalty.
The Magic: The AI tries many different ways to answer. When it gets the timing right, it gets a "treat" (a reward). Over thousands of tries, it learns exactly how much to elaborate or how much to cut out to hit that 15-second mark perfectly.

The Secret Ingredient: "Spoken Time Markers"

How does the AI actually keep track? TiCo introduces special invisible tags called Spoken Time Markers.

The Metaphor: Think of these markers as milestones on a road trip. As the AI generates its answer, it drops a marker every few seconds saying, "I am here, and 10 seconds have passed."
If the AI sees it has 10 seconds left but only 5 seconds of content to go, it knows to slow down and add more details. If it has 5 seconds left but 10 seconds of content, it knows to speed up or cut the fluff.

Why This Matters

This isn't just about being precise; it's about real-world usability.

Driving: If you are driving and ask for traffic updates, you don't want a 2-minute story. You need a 10-second warning.
Emergency: If a medical device needs to give instructions, it must be brief and clear, not a long lecture.
Battery Life: Shorter, controlled responses save battery on your smartwatch.

The Results

The researchers tested TiCo on a new "Time-Bench" (a test specifically for timing).

Before TiCo: The AI was like a driver who ignores the speed limit, often going way over or under the target time.
After TiCo: The AI became a precision pilot. It could hit a 15-second target almost perfectly, and it could even do this for longer times (like 40 seconds) or when you asked via text instead of voice.

In a Nutshell

TiCo takes a voice AI that talks too much or too little and teaches it to respect the clock. It gives the AI an internal stopwatch and a coach to help it pace its speech, ensuring that when you ask for a 15-second answer, you get exactly that—no more, no less.

1. Problem Statement

Spoken Dialogue Models (SDMs) are increasingly used in real-world applications like voice assistants, wearables, and emergency systems. In these scenarios, the duration of the response is a critical constraint (e.g., a driver needs a 10-second traffic update; a medical device needs a concise instruction).

While existing Large Language Models (LLMs) struggle with length constraints in text, the challenge is significantly harder for SDMs because:

Word Count $\neq$ Duration: A single word can have varying syllable counts, and speech duration depends on phonetic composition, prosody, and speaking rate.
Lack of Time Awareness: Current SDMs lack an internal mechanism to estimate elapsed speaking time during generation. They often fail to follow explicit instructions like "generate a response lasting exactly 15 seconds," frequently producing responses that are too long or too short.
Gap in Research: While benchmarks exist for text length control, there is no dedicated framework or benchmark for time-controllable spoken generation.

2. Methodology: TiCo Framework

The authors propose TiCo, a two-stage post-training framework that enables SDMs to estimate and regulate speech duration in real-time using Spoken Time Markers (STMs).

Core Mechanism: Spoken Time Markers (STMs)

STMs are special tokens (e.g., <10.6 seconds>) inserted into the model's intermediate representation (the semantic plan before speech synthesis). These markers allow the model to track temporal progress and adjust content generation to meet a target duration.

Stage 1: Time-Awareness Training (Self-Generation & SFT)

Goal: Teach the model to associate intermediate generation states with elapsed time.
Process:
1. The model performs self-generation: It takes an input query and generates a response without duration constraints.
2. ASR Alignment: The generated speech is transcribed using an ASR system with timestamps.
3. Marker Insertion: The model aligns the intermediate text tokens with the ASR timestamps to insert STMs (e.g., <3.2 seconds>) into the sequence.
4. Supervised Fine-Tuning (SFT): The model is trained to predict these augmented sequences (text + STMs) autoregressively. This allows the model to learn the mapping between semantic content and temporal duration without needing external labeled data.

Stage 2: Time-Controllable Training (Reinforcement Learning)

Goal: Optimize the model to strictly follow duration instructions.
Process:
1. Instruction Augmentation: The input prompt is modified to include a target duration (e.g., "Your response should be 15 seconds").
2. RL with Verifiable Rewards (RLVR): The authors use GRPO (Group Relative Policy Optimization).
3. Reward Design:
  - Main Reward: A Gaussian function measuring the difference between the target duration and the duration indicated by the final STM in the generated sequence.
  - Auxiliary Rewards: To prevent "reward hacking" (e.g., generating fake markers), the system includes rewards for marker presence, monotonicity (time must increase), and penalties for repetition or trivially copying the target number.
4. Regularization: The training incorporates CHORD, which blends the RL objective with the Stage 1 SFT data to prevent catastrophic forgetting of general conversational abilities.

3. Key Contributions

TiCo Framework: A simple, efficient post-training method requiring only a small amount of data and no new question-answer pairs. It relies on self-generation and RL to instill time awareness.
TiCo-Bench: The first benchmark specifically designed to evaluate time-controllable instruction following in SDMs. It covers diverse tasks (QA, reasoning, summarization) and modalities (speech and text queries), with 1,440 evaluation samples across short (10–30s) and long (30–60s) duration settings.
Generalization: Demonstrated that the learned time-control capability generalizes to:
- Longer durations (up to 60s) than seen during training (41s).
- Text queries, despite the model being trained primarily on speech queries.

4. Experimental Results

The authors evaluated TiCo on Qwen2.5-Omni-7B against strong baselines, including commercial models (GPT-audio, Kimi Audio), open-source SDMs, and cascaded systems (LLM + TTS).

Performance Metrics: Measured by Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE) of the actual duration vs. the target.
Key Findings:
- Superior Accuracy: TiCo achieved an overall MAE of 4.54s and MAPE of 14.9%, significantly outperforming the base model (13.01s / 42.3%) and the strongest cascaded baseline (10.41s / 29.0%).
- Robustness: TiCo maintained low error rates across both short and long duration settings, whereas other models struggled significantly with tight time budgets (short durations).
- Quality Preservation: Despite the strict duration constraints, the GPT-score for response quality (helpfulness, coherence) remained comparable to the base model, indicating that time control did not degrade semantic quality.
- Cross-Modal Transfer: TiCo trained on speech queries successfully generalized to text queries, achieving the best performance on text-based benchmarks as well.

5. Significance and Impact

Practical Deployment: TiCo addresses a critical gap in real-world voice assistant deployment, where response length directly impacts user experience, battery life, and safety (e.g., in driving or medical contexts).
Paradigm Shift: It moves beyond simple text-length constraints to speech-time constraints, acknowledging the complexity of the speech generation process (prosody, speaking rate).
Efficiency: The method is data-efficient, leveraging self-generated data and reinforcement learning, making it applicable to various SDM architectures without requiring massive new datasets.
Future Direction: The paper establishes a foundation for "time-aware" generative AI, suggesting that temporal control can be learned as a robust intermediate planning capability, paving the way for more controllable and human-like interactive agents.