141 papers
This paper introduces the first formal framework for Audio-Visual World Models (AVWM), presenting the AVW-4k dataset and the AV-CDiT model to enable high-fidelity, synchronized simulation of binaural audio and visual dynamics that significantly enhances agent performance in continuous navigation tasks.
LARA-Gen introduces a framework for continuous, fine-grained emotion control in music generation by aligning latent affective representations with an external emotion predictor and utilizing a valence-arousal control module, thereby overcoming the limitations of text-based prompting and significantly improving both emotional adherence and music quality.
This paper introduces EmoSURA, a novel evaluation framework that improves the assessment of long-form emotional speech captions by decomposing them into atomic perceptual units for audio-grounded verification, addressing the limitations of traditional metrics and LLM judges while providing the standardized SURABench resource.
This paper proposes a paralinguistic- and emotion-aware model that detects optimal timing for emotional validation in Japanese empathetic dialogue using only speech signals, achieving significant improvements over conventional baselines by integrating self-supervised paralinguistic and multi-task emotion classification encoders.
This study demonstrates that in noisy dyadic conversations, speakers adapt by increasing hand-gesture complexity and speech volume while listeners enhance backchanneling, with background noise also modulating head and trunk movements and slightly reducing hand-speech synchrony.
This paper investigates speech enhancement strategies within the latent space of neural audio codecs, demonstrating that predicting continuous latent representations with non-autoregressive models and fine-tuning the encoder yields the best overall performance, despite a trade-off in codec reconstruction quality.
This paper proposes a noise-conditioned mixture-of-experts framework for robust speaker verification that decomposes the feature space into specialized noise-aware subspaces using a routing mechanism, expert specialization strategy, and SNR-decaying curriculum learning to outperform conventional unified representation methods under diverse noisy conditions.
This paper presents the first systematic investigation into how accent information is encoded in Discrete Speech Representation Tokens (DSRTs), introducing a unified evaluation framework that reveals layer selection is the most critical factor for retaining accents, while ASR supervision significantly diminishes them and naive codebook reduction fails to disentangle accent from phonetic and speaker information.
This paper introduces Human-CLAP, a human-perception-based contrastive language-audio model trained on subjective evaluation scores to significantly improve the correlation between automated CLAP scores and human judgments, addressing the previously low alignment of standard CLAP metrics with human perception.
This paper proposes TI-DANSE+, an improved distributed signal estimation algorithm for topology-unconstrained wireless acoustic sensor networks that accelerates convergence by utilizing partial in-network sums and a tree-pruning strategy, while maintaining robustness to link failures and reducing communication bandwidth compared to existing methods.
This paper proposes S2S-ZEST, a textless and non-parallel speech-to-speech framework that achieves zero-shot emotion style transfer by extracting semantic, speaker, and emotion representations to synthesize speech that preserves the source's content and identity while adopting the reference's emotional style.
This paper systematically evaluates four Contrastive Decoding strategies across diverse Large Audio Language Models, identifying Audio-Aware and Audio Contrastive Decoding as most effective while introducing a Transition Matrix framework to demonstrate that these methods successfully rectify specific error patterns like false audio absence claims but fail to correct flawed reasoning or confident misassertions.
This paper demonstrates that neural audio codecs achieve optimal adversarial robustness in speech recognition at intermediate residual vector quantization depths, which effectively balance the suppression of adversarial perturbations with the preservation of speech content, outperforming traditional compression defenses.
The paper proposes Universal Speech Content Factorization (USCF), a simple and invertible linear method that extracts low-rank, speaker-independent speech representations to enable competitive zero-shot voice conversion and efficient training of timbre-prompted text-to-speech models using minimal target speaker data.
This paper addresses the reproducibility crisis in music source separation by attempting to replicate the Band-Split RNN model, ultimately releasing an optimized version with improved performance and publicly available code to advocate for more transparent and sustainable research practices.
VoiceBridge is a novel one-step latent bridge model that leverages an energy-preserving variational autoencoder and a joint neural prior to efficiently reconstruct high-quality 48 kHz fullband speech from diverse distortions across various in-domain and out-of-domain general speech restoration tasks without requiring distillation.
VSSFlow introduces a unified flow-matching framework that seamlessly integrates Video-to-Sound and Visual Text-to-Speech generation through a disentangled condition aggregation mechanism, demonstrating that joint learning can surpass specialized state-of-the-art baselines without performance degradation.
This paper introduces SCENEBench, a comprehensive benchmark suite designed to evaluate Large Audio Language Models on critical non-speech and cross-lingual audio understanding tasks relevant to assistive and industrial applications, revealing significant performance gaps and latency challenges in current state-of-the-art models.
This paper introduces MUGEN, a comprehensive benchmark revealing that Large Audio-Language Models struggle with multi-audio understanding as input scaling increases, and demonstrates that combining training-free strategies like Audio-Permutational Self-Consistency with Chain-of-Thought can significantly improve performance.
This paper introduces the Pulse-Train-Resonator (PTR), a differentiable synthesis model that improves engine sound generation by directly modeling sequential exhaust pressure pulses and physical resonances rather than approximating spectral characteristics, achieving superior reconstruction accuracy and interpretability across diverse engine types.