121 papers
This paper proposes Multi-View Tensor Fusion (MVTF), a novel framework that leverages synchronized multi-perspective lip videos during training to learn cross-view correlations, thereby significantly enhancing target speaker extraction performance and robustness for both single-view and multi-view inference scenarios.
The paper proposes HyWA, a novel Personalized Voice Activity Detection (PVAD) approach that utilizes a hypernetwork to generate personalized weights for selected layers of a standard VAD model, demonstrating consistent performance improvements and enhanced deployment flexibility compared to existing speaker-conditioning methods.
This paper proposes a robust Audio-Visual Target Speaker Extraction framework that leverages emotion-aware multiple enrollment fusion, demonstrating that training with high modality missing rates significantly enhances performance stability against real-world signal loss while achieving optimal results by fusing single-frame facial images with frame-level lip features.
This paper introduces \texttt{nlm}, an open-source set of C++ Max externals that enables efficient, real-time non-linear modal synthesis for strings, membranes, and plates, thereby making advanced physical modeling accessible to composers and sound designers through interactive parameter control and multichannel output.
This study demonstrates that the choice of musical feature encoding fundamentally reshapes network topology and uncertainty distributions, revealing a critical trade-off where compressed single-feature representations offer high descriptive accuracy with lower model error, while richer multi-feature encodings preserve finer distinctions at the cost of increased state space complexity and higher model error.
This paper reveals a systematic gender bias in speech quality assessment where male listeners consistently rate audio higher than female listeners, particularly for low-quality speech, and proposes a gender-aware model that learns distinct scoring patterns to improve prediction accuracy and equity.
This paper introduces Geo-ATBench, a new benchmark and the Geo-AT task that leverage geospatial semantic context to resolve acoustic ambiguities in multi-label audio tagging, demonstrating through the GeoFusion-AT framework that incorporating location-based priors significantly improves recognition performance and aligns with human judgment.
The paper proposes G-STAR, an end-to-end system that integrates a time-aware speaker-tracking module with a Speech-LLM backbone to achieve robust, timestamped speaker-attributed recognition for long-form, overlapping multi-party speech while maintaining global identity consistency.
The paper introduces FireRedASR2S, a state-of-the-art industrial-grade all-in-one automatic speech recognition system that unifies high-performance modules for speech transcription, voice activity detection, language identification, and punctuation prediction, achieving superior results across Mandarin, Chinese dialects, and English benchmarks compared to existing solutions.
This paper systematically analyzes widely used speech tokenizers and reveals that they primarily encode phonetic rather than lexical-semantic information, highlighting a critical mismatch with text-derived semantics that necessitates new design approaches for effective multimodal LLM integration.
This paper introduces a calibration-reasoning framework that fine-tunes foundational Audio Large Language Models through a calibration stage and Group Relative Policy Optimization-based reinforcement learning to achieve state-of-the-art performance in multidimensional speech quality assessment, artifact localization, and MOS prediction.
This paper introduces ParaS2S, a reinforcement learning framework and corresponding benchmark (ParaS2SBench) that utilizes a novel PolyTone-trained automatic judge to effectively align speech-to-speech models with paralinguistic cues, achieving superior performance in response content and speaking style compared to supervised fine-tuning while requiring fewer paired demonstrations.
This paper introduces the trajectoRIR database, a comprehensive collection of 8,648 stationary room impulse responses and dynamic audio recordings captured by various microphone arrays moving along a controlled L-shaped trajectory, designed to support diverse acoustic signal processing tasks such as source localization, spatial reconstruction, and system identification.
This paper presents a competitive open-source cascaded system that combines EEND-VC speaker diarization and fine-tuned Qwen3 ASR to achieve first place in the DISPLACE-M challenge by effectively extracting medical conditions from overlapping, code-switched Hinglish clinical dialogues.
This study analyzes a multilingual corpus of Finnish, French, and Slovak child speech to demonstrate that autistic speakers exhibit a distinct, cross-linguistic prosodic profile characterized by increased intensity variability, clearer voice quality, and reduced temporal dynamics, thereby challenging deficiency-based models in favor of a complex, language-independent acoustic signature.
This cross-linguistic study demonstrates that while certain acoustic-prosodic markers of autism in children's speech generalize across Finnish, French, and Slovak, robust classification performance requires language-specific modeling due to significant variations in feature importance and transferability across typologically distinct languages.
This paper introduces a training-free, post-hoc method called activation steering that neutralizes accents in zero-shot Text-to-Speech while preserving speaker timbre by applying offline-extracted steering vectors during inference.
ImKWS is a novel test-time adaptation method for keyword spotting that addresses severe class imbalance between rare keywords and background noise by employing a dual-branch entropy minimization strategy with separate update strengths and multi-transformation consistency, thereby preventing model overconfidence and bias without requiring labeled data.
This paper proposes a parameter-free activation steering method that identifies accent information within a specific band of middle encoder layers in speech foundation models and corrects accent-induced representation shifts during inference, thereby significantly reducing word error rates across diverse accents without requiring model fine-tuning.
The paper proposes CSP-FT, a characteristic-specific partial fine-tuning strategy that selectively updates only the most and least relevant layers of LLM-based TTS models to achieve superior emotion and speaker adaptation with significantly faster training and reduced catastrophic forgetting compared to full fine-tuning.