Technical Summary: AgentSteerTTS

Problem Statement

Despite significant advances in the naturalness and fluency of Text-to-Speech (TTS) synthesis, existing models struggle with fine-grained control over composite instructions. Current systems often fail to faithfully render user-specified emotional or stylistic attributes when multiple, potentially conflicting constraints are present (e.g., "Happy but slightly Arrogant").

The paper identifies two primary bottlenecks causing this failure:

1. Semantic-Acoustic Misalignment: Discrete textual intents do not map cleanly to continuous acoustic realizations. The authors observe that under composite prompting, target emotion dimensions are systematically under-activated (suppressed), while non-target dimensions receive spurious activation (leakage). This results in "mode-averaged" outputs that dilute specific attributes rather than combining them.
2. Speaker-Emotion Entanglement: Speaker identity (timbre) and prosodic/emotional cues are coupled in natural speech. In latent generative models, the realization of a target emotion often shifts systematically across different speakers. This creates a trade-off where improving composite intent fidelity often leads to speaker drift (loss of identity), and vice versa.

Methodology: AgentSteerTTS

To address these challenges, the authors propose AgentSteerTTS, a multi-agent closed-loop framework designed for intent-faithful expressive control. The architecture operates through a "decouple–anchor–refine" mechanism involving three specialized agents:

1. Adversarial Disentanglement Agent (ADM)

This module addresses the speaker-emotion entanglement problem.

• Mechanism: It factorizes reference audio into two latent subspaces: $Z_{id}$ (speaker identity) and $Z_{emo}$ (emotion/prosody).
• Training: It employs a reconstruction loss ($L_{rec}$) to ensure acoustic fidelity, an orthogonality loss ($L_{orth}$) to minimize cross-covariance between the two factors, and bidirectional adversarial training. Two discriminators attempt to predict speaker labels from the emotion stream and emotion labels from the identity stream. Gradient Reversal Layers (GRL) are used to prevent the encoders from learning these cross-dependencies, effectively suppressing leakage.

2. Dual-Stream Anchoring Controller (DAC)

This module addresses the semantic-acoustic misalignment by grounding abstract text instructions in concrete acoustic prototypes.

• Retrieval Agent: Given a user instruction, this agent queries a large-scale acoustic prototype library (curated high-expressivity samples with MLLM annotations) to retrieve relevant emotional anchors. It performs text-based embedding search and optional temporal cropping to select the most consistent segment.
• Synthesis Agent: This agent fuses the retrieved acoustic prototype with the user's intent. It uses a gated attention mechanism to combine:
  • Acoustic Stream: Features from the retrieved prototype and the target speaker's neutral state.
  • Text Stream: Discrete intensity vectors derived from the instruction.
• Output: The fusion produces a continuous control vector ($\hat{z}_{emo}$) that anchors the generation in the emotion subspace while respecting the target speaker's identity.

3. Fast–Slow Feedback Agent

This module refines the output during inference to correct residual semantic drift.

• Fast Agent (Latent Gradient Correction): Before waveform synthesis, a lightweight, differentiable Latent Consistency Predictor (LCP) estimates the alignment between the intermediate Mel-spectrogram and the target intent. It performs online gradient descent to adjust a scaling factor ($\alpha$) for the emotion embedding, steering the generation without running the vocoder.
• Slow Agent (Perceptual Critique): After waveform generation, a Supervisor Agent (powered by an MLLM backend like Gemini or Qwen) evaluates the output. If it detects deviations (e.g., "emotion too weak" or "incorrect type"), it triggers either a re-calibration of the Fast Agent or a re-retrieval of prototypes by the DAC.

Key Contributions

The paper claims three primary contributions:

1. Identification of Systematic Bias: The authors empirically demonstrate that composite-instruction TTS suffers from systematic target suppression and non-target leakage, rather than random noise, and that this is exacerbated by speaker-emotion entanglement.
2. Novel Framework: The proposal of AgentSteerTTS, a multi-agent closed-loop system that integrates adversarial disentanglement, retrieval-augmented anchoring, and hierarchical feedback to achieve fine-grained control.
3. Empirical Validation: Extensive experiments showing that the method improves intent alignment while preserving speaker identity, outperforming state-of-the-art baselines on both public datasets and a new composite-instruction benchmark.

Experimental Results

The authors evaluated AgentSteerTTS on the ESD dataset and a custom Fine-grained Composite Dataset (involving instructions like "Happy but slightly Arrogant").

• Composite Control Performance: On the composite benchmark, AgentSteerTTS achieved an E-SIM (Emotion Similarity) of 0.955 (with a Gemini3 backend), significantly outperforming IndexTTS2 (0.864) and CosyVoice2 (0.748).
• Speaker Preservation: The system maintained high speaker similarity (S-SIM = 0.841), demonstrating that improved emotion control did not come at the cost of identity drift.
• Ablation Studies:
  • Removing the Adversarial Disentanglement module increased speaker drift ($\Delta$S-SIM) from 0.021 to 0.048.
  • Removing Prototype Retrieval (text-only conditioning) caused the Composite Success Rate (CSR) to drop from 0.78 to 0.49.
  • Removing the Feedback Loops reduced CSR, indicating their role in correcting intensity mismatches.
• Qualitative Analysis: Spectrogram and prosodic trajectory visualizations confirmed that the model produces localized, intent-aligned spectral edits rather than global distortions or simple averages of single emotions.

Significance and Limitations

Significance:
The paper argues that AgentSteerTTS provides a robust solution to the "structural mismatch" between discrete text and continuous audio. By explicitly decoupling speaker and emotion and using a closed-loop feedback mechanism, the framework enables reliable synthesis of complex, nuanced emotional states that current feed-forward models fail to capture.

Limitations:
The authors acknowledge several constraints:

• Dependency on Prototype Library: Performance is constrained by the coverage and retrieval reliability of the acoustic prototype library, particularly for rare or unusual composite styles.
• Evaluator Reliance: The Slow Loop relies on external MLLM evaluators, which may be imperfect or over-confident on out-of-distribution prompts.
• Partial Disentanglement: The ADM targets the dominant speaker-emotion coupling but does not achieve full factorization of all acoustic factors (e.g., recording conditions, accents, or age), meaning residual variations may persist.

The paper concludes that while the method significantly advances composite control, future work should focus on reducing dependence on external evaluators and expanding the robustness of the prototype library.