Technical Summary: TORUS – A Test of Rendering-Understanding Self-Coherence for Unified Audio Models

1. Problem Statement

Unified audio models, capable of audio generation, understanding, and increasingly editing, are proliferating. However, a fundamental question remains unaddressed: Do the two heads of a unified model agree about the same audio?

Current evaluation practices assess generation, editing, and understanding in isolation using specialized benchmarks. A model may perform well on separate leaderboards for fidelity, instruction-following, and question-answering, yet there is no mechanism to verify if the model's understanding head can correctly reason about the audio its own generation head produced. While vision research has begun to close this loop by testing unified models on their own generated images, the audio domain lacks an analogous evaluation. Directly transferring vision protocols is non-trivial due to audio's strong text priors (allowing answers without listening) and unique temporal/causal structures. Consequently, the field lacks a benchmark to measure the self-coherence of unified audio models.

2. Methodology: The TORUS Benchmark

The authors introduce TORUS (Test of Rendering-Understanding Self-Coherence), the first audio-native self-coherence benchmark.

2.1 Benchmark Structure

TORUS comprises 48 three-stage self-coherence tests containing 432 six-option multiple-choice questions (MCQs). The tests span three audio types (speech, sound, music) and five task families. Each test follows a closed-loop protocol involving a Generation Head (G) and an Understanding Head (U):

1. Stage 1 (Generation): G generates an audio scene ($a_{gen}$) from a prompt ($p_{gen}$). U answers 3 MCQs ($Q_{gen}$) based on $a_{gen}$.
2. Stage 2 (Editing): G edits $a_{gen}$ using an edit prompt ($p_{edit}$) to produce $a_{edit}$. U answers 3 MCQs ($Q_{edit}$) based on $a_{edit}$.
3. Stage 3 (Counterfactual Edit): G edits $a_{edit}$ using a counterfactual prompt ($p_{cf}$) that alters a specific attribute while preserving others, resulting in $a_{cf}$. U answers 3 MCQs ($Q_{cf}$) based on $a_{cf}$.

The Coherence Metric is defined as the percentage of questions answered correctly by the model across these stages.

2.2 Data Curation and Leak Prevention

To prevent models from answering correctly via text priors (without listening) or failing due to vagueness, the authors implemented a rigorous, human-in-the-loop pipeline:

• Human-Authored Seeds: Define the scene progression and attributes.
• LLM Expansion: A frontier LLM generates prompts and audio specifications.
• Question Authoring: An LLM creates 3 MCQs per stage based on specific "keys" (attributes) derived from the ideal audio.
• Coupled Gate (Leak Check):
  • Muted-Audio Solver: An audio model attempts to answer the question with 1 second of silence. If it succeeds, the question is flagged as a "text prior leak" and rewritten.
  • Ideal-Generation Solver: An LLM given the perfect audio description attempts to answer. If it fails, the question is flagged as too difficult/vague.
• Human Verification: Final review ensures the test evaluates reasoning and generation capabilities reasonably.

2.3 Experimental Setup

• Models Evaluated: Five open unified models (Audio-Omni, Audex-30B, Audex-2B, UniAudio-2, Unified-IO 2).
• Cascaded Baseline: A pipeline of state-of-the-art (SOTA) specialized models for generation, editing, and understanding (e.g., TangoFlux, ACE-Step 1.5, Step-Audio-EditX, gpt-audio) chained together to serve as a performance upper bound.
• Handling Non-Native Editing: For unified models lacking native audio-in editing, a "self-caption edit" chain was used: the model captions its own output, and this caption serves as the prompt for the next generation step.
• Metrics:
  • Self-Evaluation: Coherence (% accuracy).
  • Objective Evaluation: Word Error Rate (WER), distributional metrics (FAD, FD, KL), and modality correctness (CM).

3. Key Results

The evaluation reveals a significant gap between specialized pipelines and unified models in self-coherence.

• Overall Performance: The Cascaded Baseline achieved 63.2% accuracy. The best unified model (Audex-30B) achieved 50.5%, trailing the baseline by 12.7%. The worst unified model (Unified-IO 2) scored 21.8%.
• Stage Degradation: Most systems peaked at Stage 1 (Generation) and degraded significantly in Stages 2 and 3 (Editing and Counterfactual). This indicates that audio editing is a primary failure point for current systems.
• Scale Effects: Larger models generally performed better (Audex-30B > Audex-2B).
• Dissociation of Fidelity and Understanding:
  • Generative Fidelity: Human preference studies (A/B testing) showed that top unified models (Audio-Omni, Audex-30B) were often preferred over the Cascaded Baseline in terms of perceptual quality.
  • Self-Understanding: Despite high perceptual quality, these same models scored poorly on TORUS. For instance, Audio-Omni had the highest generative fidelity but trailed the baseline by 20 percentage points in coherence.
  • Independence: Low coherence was not solely due to poor generation quality (rendering). Some models with poor distributional metrics (e.g., Audex-2B) outperformed models with better render statistics in answering questions, suggesting that "rendering intelligible speech" and "understanding it" are currently independent capabilities.
• Modality Errors: Even the best systems generated the wrong modality (e.g., noise instead of speech) roughly 40% of the time, highlighting a need for more robust generation.

4. Key Contributions

1. First Audio-Native Self-Coherence Benchmark: TORUS introduces 48 three-stage tests with 432 leak-checked questions, covering speech, sound, and music across five task families.
2. Leak-Checked Test-Generation Pipeline: A novel pipeline utilizing coupled gates (muted-audio and ideal-generation solvers) to ensure questions cannot be answered via text priors alone and are answerable with perfect audio.
3. Evidence of Limited Self-Coherence: The study provides the first joint evaluation showing that current unified models trail specialized cascaded baselines by 12.7% to 41.4% in self-coherence. It highlights that while unified models can generate high-quality audio, they struggle to reason about their own generations, particularly during editing tasks.

5. Significance and Claims

The paper positions self-coherence as an essential, previously unmeasured property for future audio systems. The authors argue that for unified audio models to achieve general intelligence, they must not only generate and understand audio but also ensure their two heads "agree" on the same content.

The findings suggest that:

• Current unified models are not yet self-consistent; they cannot reliably make sense of their own outputs.
• The gap between generation quality and self-understanding is substantial; high-fidelity generation does not guarantee self-coherence.
• Audio editing remains a critical open problem, causing performance drops in both unified and specialized systems.
• The field requires benchmarks that move beyond isolated task evaluation to closed-loop, self-referential testing.

The authors modestly frame TORUS as a "prerequisite check" and a "sanity check" to initiate open dialogue, acknowledging that the current baseline represents the best of today's specialized pipelines and that unified models are expected to eventually surpass them. They call for further research into models that can truly integrate generation, editing, and understanding into a coherent whole.