LISE : Listenable Interpretable Speaker Embeddings

Technical Summary: Listenable Interpretable Speaker Embeddings (LISE)

Problem Statement
Deep neural network-based Automatic Speaker Verification (ASV) systems achieve high performance but rely on opaque, high-dimensional embeddings that lack structured, perceptually verifiable explanations of the vocal characteristics they encode. Existing interpretability approaches face significant trade-offs:

1. Label-dependent methods: Approaches using probing classifiers or disentanglement require annotated speaker attributes (e.g., age, accent). These are costly to produce, raise privacy concerns, cover only a subset of speaker variation, and often degrade ASV performance (e.g., EER rising from 2.3% to 17.0% in prior work).
2. Label-free but unvalidated methods: Some works reformulate encoders for intrinsic interpretability using high-dimensional sparse binary representations. While they preserve verification performance, their interpretability is unvalidated by human listeners. Correlations with low-level acoustic cues do not guarantee that humans perceive the dimensions as meaningful.

There is a critical gap in achieving a framework that is simultaneously label-free, performance-preserving, and validated by human perception.

Methodology: LISE
The authors propose LISE (Listenable Interpretable Speaker Embeddings), a post-hoc, label-free framework that decomposes pretrained speaker embeddings into a compact set of orthogonal, non-negative components. Unlike Sparse Autoencoders (SAEs) used in NLP, which rely on thousands of discrete features, LISE is tailored to the continuous and limited nature of human-describable voice characteristics.

The core decomposition reconstructs a speaker embedding $e \in \mathbb{R}^d$ as:
$$\hat{e} = Wc$$
Where $W \in \mathbb{R}^{d \times K}$ is a component matrix and $c \in \mathbb{R}^K$ are weights. The method adheres to three design principles:

1. Low Dimensionality: $K \ll d$ (typically $K \in [5, 50]$). This captures continuous voice structures rather than dispersing them across thousands of weakly activated dimensions.
2. Non-negative Continuous Weights: $c \ge 0$. This prevents subtractive interactions (where one component cancels another), ensuring each component's contribution is additive and interpretable. Continuous weights allow for graded perceptual differences rather than binary decisions.
3. Orthogonality: The optimization minimizes reconstruction error while enforcing orthogonality ($W^\top W \approx I$) via a regularization term. This ensures components capture distinct, independent axes of variation.

The framework is trained using non-negative least squares with orthogonality regularization on pretrained embeddings (x-vector and ECAPA-TDNN) without retraining the original speaker encoders.

Key Contributions

1. Label-free, Performance-Preserving Decomposition: LISE decomposes pretrained embeddings into a compact set of orthogonal, non-negative components without attribute supervision. It maximally preserves speaker verification performance on VoxCeleb1-O.
2. Human Perceptual Validation: The authors design a listening-based evaluation protocol to assess the perceptual validity of the learned components. Unlike prior work that relies on acoustic correlations, this study demonstrates that human listeners can reliably distinguish speakers based on the component weights.

Experimental Results

• Verification Performance: LISE was evaluated on VoxCeleb1-O using x-vector (512-dim) and ECAPA-TDNN (192-dim) embeddings.
  • With $K=35$ components, LISE achieved an Equal Error Rate (EER) of 3.08% for x-vector (original: 2.30%) and 2.10% for ECAPA-TDNN (original: 1.80%).
  • This performance is competitive with other unsupervised methods (PCA: 3.02% for x-vector) and significantly outperforms supervised attribute-based methods (Luu et al.: 6.70% for x-vector).
  • The system demonstrated robustness to data scarcity, maintaining low EER even when trained on 50% of the dataset.
• Perceptual Validation: In a listening task where participants distinguished speakers based on component weights:
  • Overall Accuracy: LISE achieved 83.9% accuracy, significantly outperforming PCA (59.1%) and Iben et al. (49.0%).
  • Component Reliability: 94.2% of LISE components (32/35) exceeded 70% accuracy, compared to only 11.4% for PCA and 0% for Iben et al.
  • Consistency: All 25 participants achieved high accuracy (73.7%–91.6%) with LISE, whereas results for other methods were lower and more variable.
  • Qualitative Findings: Participants provided coherent natural language descriptions for high-accuracy components (e.g., "Slower-paced female," "Deep resonant voice," "Young female, rapid tempo"), confirming the components capture recognizable auditory characteristics.

Significance and Claims
The paper claims that LISE successfully bridges the gap between high-performance ASV and human-interpretable representations without requiring labeled attributes. The primary significance lies in the demonstration of human-validated interpretability; the authors argue that dimension meaning is established only when it corresponds to a recognizable auditory characteristic for human listeners, a criterion previous methods failed to meet simultaneously with performance preservation.

The authors position LISE as a foundation for future applications in controllable voice synthesis (by using components as conditions for TTS), bias diagnosis, and auditory speaker analysis. They note that while the current evaluation uses the VoxCeleb dataset (primarily English), a small number of components captured language patterns, suggesting future work should validate on strictly monolingual datasets or explore multilingual use cases like accent control. The paper maintains a modest tone, acknowledging that while the components are interpretable, the specific descriptors (e.g., "bright," "dark") remain subjective and that systematic evaluation of TTS-based validation remains future work.