Impact Analysis of Speech Representation Learning Models for Acoustic Side-Channel Attack

Technical Summary: Impact Analysis of Speech Representation Learning Models for Acoustic Side-Channel Attack

Problem Statement
Acoustic side-channel attacks (ASCA) on keyboards exploit the distinct acoustic signatures of keystrokes to infer typed characters, posing a significant threat to user privacy in public environments and during online meetings. While ASCA has been previously studied, existing research is constrained by several limitations: reliance on outdated hardware or small-scale datasets, evaluations restricted to basic keyboard generalization, and a dependence on handcrafted features or classical machine learning models. Crucially, there is a lack of public datasets enabling systematic evaluation under Voice-over-IP (VoIP) codec distortions, and the efficacy of modern speech pretrained models (PTMs) in providing robust representations for ASCA remains unexplored. Furthermore, conventional downstream adaptation layers (e.g., Fully Connected Networks or Convolutional Neural Networks) may inadequately model the complex nonlinear feature interactions required when acoustic representations are altered by codec compression.

Methodology
To address these gaps, the authors introduce KEYAC, a novel dataset designed to analyze keystroke acoustic representations under realistic recording and transmission conditions. KEYAC comprises 37,440 impulsive acoustic events (keystrokes) captured across 37 keyboards via three distinct channels: local laptop microphones, smartphones (near-field), and real-time VoIP streaming pipelines (e.g., Zoom, Teams). This design facilitates the study of cross-device variability and VoIP-induced representation drift.

The study evaluates six speech representation learning models:

1. Self-Supervised Models: Wav2Vec2, HuBERT, WavLM, and XLS-R.
2. Supervised Models: X-Vectors and OpenAI Whisper (encoder only).

The authors assess these models under two primary settings: zero-shot inference and partial fine-tuning (keeping the backbone frozen). They compare three downstream adaptation strategies:

• Fully Connected Network (FCN): A dense layer with 90 hidden neurons.
• Convolutional Neural Network (CNN): Two convolutional blocks with kernel size 3 and max-pooling.
• Kolmogorov–Arnold Network (KAN): A proposed strategy replacing fixed linear weights with learnable univariate spline functions to explicitly model nonlinear feature interactions.

Experiments are conducted under three generalization scenarios:

1. Keyboard Generalization (Standard): Cross-keyboard evaluation on standard recordings.
2. Keyboard Generalization (VoIP): Cross-keyboard evaluation on VoIP-streamed audio.
3. Codec Generalization: Training on standard recordings and testing on VoIP-streamed audio to isolate codec distortion effects.

Key Contributions

1. KEYAC Dataset: The introduction of a dataset specifically designed to study ASCA under cross-device variability and VoIP codec distortions, filling a critical gap in public resources.
2. Systematic Evaluation: The first comprehensive evaluation of multiple speech PTMs for ASCA, analyzing their generalization capabilities across different keyboards and transmission conditions.
3. KAN-Based Adaptation: The proposal of a KAN-based fine-tuning strategy. The authors hypothesize that conventional architectures struggle with the nonlinear interactions introduced by codec artifacts, and demonstrate that KAN adapters effectively address this limitation.

Results
Empirical results on KEYAC reveal several key findings:

• Baseline Performance: While partial fine-tuning improves performance, models struggle to generalize across VoIP codecs. Among the baselines, WavLM consistently demonstrates the most robust representations, outperforming other PTMs (e.g., HuBERT, XLS-R) across all scenarios. For instance, in the standard in-domain setting with a CNN downstream, WavLM achieves 58.34% accuracy, whereas HuBERT and XLS-R score significantly lower (36.27% and 33.45%, respectively).
• Impact of VoIP: Performance deteriorates significantly under VoIP conditions. In the keyboard holdout (OOD) setting under VoIP, WavLM's accuracy drops to 44.12%, while other models experience even sharper declines.
• Effectiveness of KAN: The proposed KAN-based downstream adaptation consistently outperforms both FCN and CNN baselines across all PTMs and evaluation scenarios.
  • In the standard in-domain setting, WavLM with KAN achieves 68.47% accuracy and 67.12% macro-F1, a substantial improvement over the CNN baseline (58.34% accuracy).
  • Under VoIP conditions, the KAN approach maintains strong performance, with WavLM reaching 57.82% accuracy for keyboard generalization, significantly outperforming the CNN baseline (44.12%).
  • For codec generalization, KAN-based fine-tuning yields the best results (58.36% accuracy for WavLM), demonstrating its ability to mitigate the degradation caused by spectral and temporal distortions.

Significance and Claims
The paper claims that the degradation in ASCA performance under VoIP conditions stems from the limited ability of conventional adaptation layers to capture complex nonlinear interactions within distributed acoustic representations. By employing Kolmogorov–Arnold Networks (KAN), the authors establish a new state-of-the-art on the KEYAC dataset. The findings suggest that combining robust speech representations (specifically WavLM) with expressive nonlinear adaptation layers (KAN) is critical for improving ASCA generalization under realistic recording and transmission conditions. The work highlights the necessity of moving beyond linear or shallow nonlinear adaptation heads when dealing with codec-distorted acoustic side-channel data.