⚡ electrical engineering

Explainable AI in Speaker Recognition -- Attention Map Visualisation and Evaluation

This paper proposes and validates a new evaluation algorithm, Modified RISE-eval, to systematically assess the attention maps generated by GradCAM and LayerCAM in speaker recognition tasks, revealing that each method offers distinct advantages under different experimental conditions.

Original authors: Yanze Xu, Mark D. Plumbley, Wenwu Wang

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yanze Xu, Mark D. Plumbley, Wenwu Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Why Do We Need to "See" AI Thinking?

Imagine you have a very smart but silent robot that can tell who is speaking just by listening to their voice. It's great at its job, but it never explains how it figured it out. Did it listen to the pitch? The accent? The background noise?

This paper is about Explainable AI (XAI). The goal is to make the robot's "brain" transparent so humans can understand its decision-making process. Specifically, the authors want to see where the AI is "looking" (or listening) when it makes a guess. They call this an Attention Map.

Think of an Attention Map like a heat map on a weather forecast. Just as a weather map shows you exactly where the storm is hitting hardest, an Attention Map shows the AI which parts of the sound wave are most important for identifying the speaker.

The Problem: Not All Maps Are Created Equal

The researchers looked at two popular tools used to draw these maps: GradCAM and LayerCAM.

GradCAM is like a senior manager who looks at the big picture and makes a broad summary.
LayerCAM is like a junior analyst who looks at the fine details and specific data points.

The problem is that no one had a reliable way to test which tool was actually drawing the correct map. It's like having two GPS apps giving you different routes, but no way to know which one is actually leading you to the destination.

The Solution: The "Mute Button" Test

To fix this, the authors improved an existing testing method called RISE-eval and created a new version called Modified RISE-eval.

Here is how their new test works, using a mystery box analogy:

The Setup: You have a box of clues (the audio recording) and a map (the Attention Map) that says, "The answer is hidden in these specific clues."
The Test: The researchers take the map and start covering up (masking) the clues it says are important.
- If the map is good, covering up those clues should make the AI completely confused and unable to guess the speaker. The AI's performance should crash.
- If the map is bad, covering up those clues shouldn't matter much because the AI was actually looking at different clues anyway. The AI's performance stays high.
The Improvement: The old test (RISE-eval) had two flaws:
- It tried to cover up everything until the box was empty, which sometimes made the test results look the same for good and bad maps.
- It kept covering up parts of the box that were already empty (irrelevant noise), which confused the results.

The Modified RISE-eval fixes this by only covering up the "loud" parts of the map (the important clues) and stopping before it starts covering up the empty space. It's like only muting the specific instruments the conductor pointed to, rather than silencing the whole orchestra.

What They Found: It Depends on the "Depth" of the Brain

The researchers tested these tools on a speaker recognition system (an AI trained to identify voices). They looked at the AI's "brain" at different depths, from the shallow layers (early processing) to the deep layers (final decision).

Here is the verdict:

At the Shallow Layers (The "Early" Brain):
- LayerCAM was the winner.
- Analogy: At the beginning of the process, the AI is looking at fine details (like the shape of a sound wave). LayerCAM is better at highlighting these tiny, specific details. When they muted the parts LayerCAM pointed to, the AI failed faster.
At the Deep Layers (The "Final" Decision):
- GradCAM was the winner.
- Analogy: By the time the AI reaches the final decision, it has combined all the details into a big picture. GradCAM is better at highlighting these broad, important regions. When they muted the parts GradCAM pointed to in the final layer, the AI failed much more dramatically than when they used LayerCAM.

The Conclusion

You can't just pick one tool and use it for everything.

If you want to understand how the AI processes fine details (early in the network), use LayerCAM.
If you want to understand the final decision (the deep layers), use GradCAM.

The paper concludes that by using their new "Mute Button" test (Modified RISE-eval), we can finally tell which tool is doing a better job at explaining the AI's attention, ensuring we aren't just looking at pretty pictures, but actually understanding how the machine thinks.

Technical Summary: Explainable AI in Speaker Recognition – Attention Map Visualisation and Evaluation

Problem Statement
Explainable AI (XAI) seeks to make the decision-making processes of neural networks transparent. In the context of speaker recognition, neural networks are assumed to possess an attention mechanism analogous to humans, selectively processing relevant information (e.g., specific regions of an audio spectrogram) to determine speaker identity. While Class Activation Map (CAM)-based methods like GradCAM and LayerCAM are widely used to visualise these attention mechanisms, the evaluation of the resulting attention maps remains underexplored. Existing evaluation algorithms, such as the Randomised Input Sampling for Explanation - Evaluation (RISE-eval), suffer from specific limitations: they can produce indistinguishable results under certain strategies (e.g., insertion strategy) and introduce irrelevant factors (e.g., overmasking with near-zero values) that confound the evaluation of map quality.

Methodology
The paper proposes a systematic framework for analysing and evaluating attention maps in speaker recognition, consisting of two main phases: visualisation and evaluation.

Attention Map Visualisation:
The study employs a ResNet34-based Convolutional Neural Network (CNN) trained on the VoxCeleb2 dataset for speaker recognition. Two representative CAM-based methods are applied to visualise the network's attention at four different ResNet layers:
- GradCAM: Computes a weighted sum of activation maps using global average pooling of gradients.
- LayerCAM: Computes a weighted sum by element-wise multiplying gradients with activation values, allowing for spatially varying weights within channels.
  These methods generate heatmaps (attention maps) highlighting input regions (spectrogram frames) deemed important for classifying a specific speaker.
Attention Map Evaluation (Modified RISE-eval):
The authors critique the original RISE-eval algorithm, which progressively masks input regions based on attention maps to measure performance degradation. They identify two key issues:
- Indistinguishability: The "insertion strategy" (progressively revealing masked regions) often yields overlapping performance curves, making it difficult to differentiate map quality.
- Irrelevant Factors: The original method masks up to 100% of the input, including pixels with near-zero attention values. This "overmasking" effectively introduces random noise, drifting the evaluation away from the true attention mechanism.
To address these, the authors propose Modified RISE-eval:
- Strategy Selection: The insertion strategy is removed entirely; only the deletion strategy (progressively masking high-value regions) is retained.
- Threshold Mechanism: A threshold ( $t$ ) is introduced to filter out low-value pixels. Only pixels with attention values above $t$ are sampled for masking. This prevents the introduction of random noise and ensures the evaluation focuses on regions the network actually attends to.
- Scoring: The evaluation score is calculated based on the area under the "masking performance curve" (classification accuracy vs. mask ratio), specifically weighting performance drops that occur at lower mask ratios more heavily.

Key Results
The study evaluates GradCAM and LayerCAM across four ResNet layers using the Modified RISE-eval algorithm on a speaker recognition task involving concatenated utterances from two speakers.

Visualisation Performance:
- Layer 1 & 2: LayerCAM produces attention maps that cause a greater degradation in model performance when masked, indicating it better identifies critical regions in shallow layers.
- Layer 3: Both methods perform comparably.
- Layer 4 (Final Layer): GradCAM outperforms LayerCAM. Its attention maps cause a steeper performance drop, suggesting it more accurately highlights the regions truly influencing the final decision.
- Class-Discriminative Ability: Visual inspection reveals that Layer 4 GradCAM maps exhibit strong class-discriminative ability, accurately localizing the target speaker's region in concatenated spectrograms, whereas Layer 1 maps are less distinct.
Quantitative Scores:
The Modified RISE-eval scores confirm the trend: LayerCAM achieves higher scores for Layers 1, 2, and 3, while GradCAM achieves the highest score for Layer 4.

Significance and Claims
The paper claims the following contributions and insights:

Systematic Review and Improvement: It provides a dedicated review of the RISE-eval algorithm, identifying specific flaws (indistinguishability and overmasking) and proposing the Modified RISE-eval to resolve them.
Layer-Dependent Performance: The study demonstrates that the efficacy of attention map visualisation methods is layer-dependent. There is no single "best" method; LayerCAM is superior for analysing attention in shallower layers, while GradCAM is superior for the final decision-making layer.
Evaluation Paradigm: It establishes a methodology where the network itself validates the attention map by measuring the impact of masking on its own performance, rather than relying solely on human annotation or class-discriminative visual checks.
Limitations and Future Work: The authors modestly note that while Modified RISE-eval improves upon RISE-eval, a bias exists where shallower-layer maps (which are finer-grained and cover non-silent regions like formants) may be favoured over deeper-layer maps (which are coarser) due to the nature of the masking process. They acknowledge that interpreting what the visualised attention truly represents for human understanding remains an open theoretical challenge.

The work concludes that for speaker recognition tasks, practitioners should select the visualisation method based on the specific network layer being analysed, using Modified RISE-eval as a robust tool for quality assessment.