⚡ electrical engineering

Explanations for Automatic Speech Recognition

This paper addresses the challenge of quality assessment in neural network-based Automatic Speech Recognition by adapting existing Explainable AI techniques, specifically Statistical Fault Localization and Causal methods, to generate minimal and sufficient audio frame explanations for transcriptions, thereby enhancing system understanding and trust.

Original authors: Xiaoliang Wu, Peter Bell, Ajitha Rajan

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaoliang Wu, Peter Bell, Ajitha Rajan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but mysterious, robot friend who listens to your voice and types out what you say. This is an Automatic Speech Recognition (ASR) system. Sometimes, the robot types exactly what you said. Other times, it makes mistakes.

The problem is that these robots are "black boxes." You can't peek inside their brains to see why they typed "apple" instead of "pear." This paper introduces a way to open that black box and say, "Hey, here is exactly which part of your voice made the robot make that specific choice."

Here is a simple breakdown of what the researchers did, using some everyday analogies.

The Big Challenge: Why is this hard?

Usually, when we try to explain a computer's decision, we look at simple choices like "Is this a cat or a dog?" But speech is messy.

Variable Length: A sentence can be short or long. It's like trying to explain a movie plot by pointing at specific frames, but the movie length changes every time.
Gray Areas: If you say "I'd like an apple" and the robot writes "I like apple," is that a mistake? A human might say, "Close enough!" but a computer usually says "Wrong." The researchers had to teach the computer how to decide what counts as "close enough" before they could explain the errors.

The Solution: Finding the "Smoking Gun"

The researchers built a tool called X-ASR. Think of this tool as a detective trying to find the "smoking gun" in a crime scene. The crime scene is the audio recording, and the "gun" is the specific tiny slice of sound that caused the robot to write what it wrote.

They didn't invent new detective methods from scratch; they borrowed three famous techniques used for image recognition (like identifying cats in photos) and adapted them for audio.

Here are the three methods they used, explained simply:

1. SFL (Statistical Fault Localisation) – "The 'What If' Game"

Imagine you have a long sentence written on a piece of paper. To find out which word is the most important, you play a game:

You take a piece of black tape and cover up random words.
You ask the robot: "Does the sentence still make sense?"
If you cover up the word "apple" and the robot suddenly changes its mind, that word was crucial.
SFL does this thousands of times, statistically figuring out which specific "frames" (tiny slices of sound) are the most likely culprits for the robot's decision.

2. Causal – "The 'Blame Game'"

This method is a bit more sophisticated. It asks: "If I remove this specific slice of sound, does the robot's answer change?"

It looks for the smallest possible change needed to flip the robot's decision.
If removing just one tiny sound fragment changes the whole sentence, that fragment gets a high "responsibility score." It's like saying, "You are the sole reason this happened."
The researchers found this method is very strict and precise, often finding the smallest possible explanation.

3. LIME – "The Local Map"

LIME is a popular method that creates a simple, easy-to-understand map around a complex decision.

Imagine you are lost in a complex city (the AI model). LIME builds a simple, flat map of just the block you are standing on to help you understand where you are.
In this paper, they used LIME to rank which sound frames were important, but they tweaked it to try and find the shortest list of important sounds, similar to the other two methods.

The Experiment: Putting it to the Test

The researchers tested these three "detectives" on three different speech robots (Google, Sphinx, and Deepspeech) using 100 different voice samples.

They measured the detectives on two things:

Size: How small is the explanation? (A smaller explanation is better because it means the tool found the exact cause, not a bunch of unnecessary noise).
Consistency: If you ask the tool to explain the same voice clip to three different robots, does it point to the same parts of the audio?

What They Found

The Winners: The SFL and Causal methods were the best. They found explanations that were much smaller and more consistent than the LIME method.
The "Causal" Edge: The Causal method was the most precise, often finding the tiniest possible slice of audio that mattered.
The Trade-off: There was a trade-off between how strict the computer was about "correctness." If the computer was very strict (only accepting perfect matches), the explanations got bigger. If it was a bit more forgiving, the explanations got smaller.
The Sweet Spot: The researchers concluded that using SFL with a "forgiving" similarity check (called Bert) offered the best balance. It gave a small, clear explanation that was consistent across different robots.

The Bottom Line

This paper doesn't claim to fix the robots or make them perfect. Instead, it gives us a flashlight. It allows us to see exactly which split-second of sound caused a speech recognition system to make a specific transcription. This helps humans understand the system better and, eventually, trust it more.

The researchers admit this is just the first step. They plan to look inside the "black box" even deeper in the future, but for now, they've successfully built a tool to point at the "smoking gun" in a voice recording.

Technical Summary: Explanations for Automatic Speech Recognition (X-ASR)

Problem Statement
The prevalence of Automatic Speech Recognition (ASR) in daily life has heightened concerns regarding system quality and accountability. Neural network-based ASR systems often operate as "black boxes" with inaccessible internal states, making it difficult to ensure quality or understand failure modes. While Explainable AI (XAI) techniques have advanced rapidly, particularly for image classification, they face significant challenges when applied to ASR transcriptions. Unlike simple classification labels, transcriptions are variable-length sequences where judging correctness involves semantic comparisons with allowances for minor deviations, rather than straightforward binary matching. Furthermore, existing interpretable models (e.g., linear regression used in LIME) struggle to handle variable-length output sequences. Current white-box approaches for speech are often model-specific and fail to explain the transcription itself, limiting their utility for quality assessment.

Methodology: The X-ASR Framework
The authors propose X-ASR, a framework designed to generate explanations for ASR transcriptions by identifying a minimal and sufficient subset of audio frames that cause a specific transcription. The framework operates in two primary steps:

Classification of Transcriptions: Since ASR outputs are not strictly binary (Correct/Incorrect), the framework first assigns labels based on similarity to a reference transcription using a user-defined threshold ( $T$ ). Two similarity metrics are supported:
- Word Error Rate (WER): Scores similarity based on insertions, deletions, and substitutions.
- BERT: Computes cosine similarity between semantic vectors of the original and perturbed transcriptions using Sentence BERT.
  If the similarity exceeds $T$ , the transcription is labeled "Correct"; otherwise, it is "Incorrect."
Adaptation of Image-Based XAI Techniques: The authors adapt three perturbation-based XAI techniques from image classification to the audio domain, treating audio frames as the equivalent of image pixels (or "raw data bins" in the time dimension):
- Statistical Fault Localisation (SFL): Adapted from software testing, this technique generates "mutant" audios by randomly masking frames (setting data points to zero). It ranks frames based on the likelihood that masking them causes a classification failure (Incorrect label). Frames are ranked by a fault-localization measure (e.g., Tarantula), and a minimal subset is selected until the explanation set reproduces the original classification.
- Causal: Based on causal theory and counterfactual reasoning, this method partitions audio into "superframes." It calculates the "responsibility" of a superframe as $1/(1+k)$ , where $k$ is the size of the smallest "witness" (a set of other superframes that, when masked, changes the outcome). This process iteratively refines partitions to identify minimal sufficient causes.
- LIME (Baseline): A Local Interpretable Model-Agnostic Explanation technique is adapted by creating perturbations of audio frames and fitting a local linear model. To align with the goal of minimal explanations, the authors employ a greedy approach, iteratively adding top-ranked frames until the classification is correct.

Key Contributions

Framework Development: Introduction of X-ASR, the first framework to generate explanations for ASR transcriptions using perturbation-based techniques adapted from image classification.
Technique Adaptation: Successful adaptation of SFL and Causal (previously dominant in image classification) to handle variable-length audio sequences and semantic transcription correctness.
Evaluation Protocol: A comprehensive evaluation across three distinct ASR systems (Google API, Sphinx baseline, and Deepspeech) using 100 audio samples from the Commonvoice dataset.

Results
The study evaluates explanations based on size (number of frames in the explanation) and consistency (agreement of explanations across different ASR systems).

Size: SFL and Causal significantly outperform LIME in generating smaller, more selective explanations, particularly when using the BERT similarity metric. Causal produces the smallest explanations due to its stricter causal conditions and iterative refinement.
Consistency: SFL demonstrates the highest consistency across different ASR systems when paired with the BERT metric. Causal and LIME show higher consistency when paired with WER, but this is attributed to their larger explanation sizes (averaging 92% of input frames for Causal), which makes them less selective.
Metric Impact: The choice of similarity metric significantly affects results. WER, being stricter (zero tolerance for differences), generates larger explanations than BERT, which allows for some semantic deviation.
Trade-off: The authors identify a trade-off between size and consistency. They conclude that SFL with BERT offers the best compromise, achieving an average explanation size of 56% with 74% consistency.

Significance and Claims
The paper claims to address the critical need for quality assessment in ASR by providing explanations that help users understand system behavior and build trust. By demonstrating that image-based XAI techniques (SFL and Causal) can be effectively adapted for speech, the authors show that it is possible to generate minimal and sufficient causes for transcriptions without requiring access to the internal architecture of the ASR model (post-hoc explanation). The study modestly limits its scope to perturbation-based, model-agnostic techniques and explicitly states that future work will investigate white-box ASR explanation methods to further improve effectiveness. The authors emphasize that these explanations facilitate accountability by helping to identify failure-causing faults.