DECKER: Domain-invariant Embedding for Cross-Keyboard Extraction and Recognition

Technical Summary: DECKER – Domain-invariant Embedding for Cross-Keyboard Extraction and Recognition

1. Problem Statement

Acoustic Side-Channel Attacks (ASCA) on keyboards pose a significant security risk, allowing adversaries to infer sensitive information (passwords, private messages) from typing acoustics. While prior research has validated ASCA feasibility, existing studies are limited by narrow evaluation conditions: small datasets, single-keyboard setups, controlled environments, and specific user demographics. Consequently, current ASCA models tend to overfit to device-specific spectral patterns (keyboard identity) rather than learning the intrinsic acoustic signatures of keys. This results in poor generalization when attackers face unseen keyboards, diverse users, or noisy real-world environments. Furthermore, existing pipelines often treat keystrokes independently, failing to leverage linguistic context to correct local prediction errors, a gap exacerbated by the emergence of Large Language Models (LLMs).

2. Methodology

The paper introduces HEAR, a novel dataset, and DECKER, a domain-invariant keystroke inference framework designed to address generalization across keyboards, users, and environments.

The HEAR Dataset:
To enable realistic evaluation, the authors curated HEAR, containing recordings from 53 participants typing on 37 different laptop keyboards. The dataset spans three realistic capture modalities:

1. External microphone capture.
2. On-device microphone capture (no network noise).
3. Capture via VoIP streaming (simulating network channel noise).
   The dataset includes demographic metadata and user-awareness responses, supporting analysis of user bias and cross-device generalization.

The DECKER Framework:
DECKER employs a four-stage generalization strategy to learn embeddings that are key-discriminative but keyboard-invariant:

1. Keyboard Signature Normalization (KSN): A learnable inverse-filter module (based on 1D CNNs) applied at the waveform level to suppress device-specific spectral coloration (impulse responses) while preserving transient keystroke structures.
2. Domain-Adversarial Disentanglement: Utilizing a Gradient Reversal Layer (GRL), the framework trains an encoder to minimize keystroke classification loss while maximizing the difficulty of a domain classifier in predicting the keyboard identity from the embeddings.
3. Supervised Cross-Keyboard Contrastive Alignment: A contrastive learning objective that pulls embeddings of the same key from different keyboards closer together while pushing different keys apart, enforcing consistency across domains.
4. Acoustic Style Randomization (ASR): A data augmentation technique that synthesizes unseen keyboard responses by applying random IIR filters, spectral envelope warping, and decay perturbations to simulate diverse acoustic conditions.

Sequence Reconstruction:
To address the accumulation of errors in long sequences, DECKER integrates an LLM-assisted decoding stage. It uses a constrained beam search with pre-trained LLMs (e.g., GPT-2, FLAN-T5) to refine noisy keystroke predictions into coherent text or passwords, leveraging linguistic priors to correct substitution, insertion, and deletion errors.

3. Key Contributions

• HEAR Dataset: A comprehensive, multi-device, multi-user, and multi-environment dataset that unifies factors such as gender diversity, VoIP streams, and NLoS (Non-Line-of-Sight) capture, enabling the first robust study of ASCA generalization.
• Comprehensive Benchmark: A new evaluation framework on HEAR that tests conventional features and pre-trained models (PTMs) in unimodal and multimodal settings, quantifying the performance drop of existing methods in cross-keyboard and cross-user scenarios.
• DECKER Framework: A novel training pipeline that combines KSN, adversarial disentanglement, contrastive alignment, and ASR to achieve domain-invariant representations, significantly outperforming unimodal and multimodal baselines in generalization tasks.
• LLM-Assisted Risk Quantification: An investigation into sentence-level attacks demonstrating that LLMs substantially boost sequence recovery, providing a realistic upper bound on keystroke leakage risks in open-world scenarios.

4. Results

Empirical evaluations on the HEAR dataset yield the following findings:

• Baseline Failure: Strong unimodal (e.g., wav2vec2, ECAPA-TDNN) and multimodal fusion baselines suffer significant performance drops (37–41%) when tested on unseen keyboards, confirming they encode device identity rather than key identity.
• DECKER Performance: DECKER narrows the generalization gap significantly. On unseen keyboards, it achieves 81.3% Top-1 accuracy compared to 58.1% for the best unimodal baseline (ECAPA-TDNN) and 63.2% for multimodal fusion baselines.
• Component Ablation: Removing KSN causes the largest performance drop (from 81.3% to 63.7% on unseen keyboards), highlighting the critical role of waveform normalization. Domain adversarial training alone (GRL) improves performance but is insufficient without KSN and contrastive alignment.
• LLM Impact: LLM-assisted decoding significantly improves sentence-level reconstruction. For human-like passwords, GPT-4 improved exact-match accuracy to 72.8% (from 42.6% raw), whereas improvements for high-entropy random strings were negligible, indicating the attack's reliance on linguistic structure.
• Feasibility: Inference on a commodity Apple M1 laptop demonstrates that DECKER can process keystrokes in real-time (~430 keystrokes/sec) and reconstruct sentences with interactive latency, requiring no GPU clusters.

5. Significance and Claims

The paper claims that ASCA remains a credible and underestimated real-world threat. The authors argue that previous assessments may have been overly optimistic about the limitations of ASCA due to reliance on controlled, single-device training regimes. By demonstrating that domain-invariant learning can effectively generalize across diverse hardware and environments, the study suggests that the attack surface is broader than previously recognized.

The integration of LLMs is highlighted as a critical factor that amplifies the practical risk of ASCA, transforming noisy acoustic predictions into coherent, sensitive text. The authors conclude that future security defenses must account for both physical acoustic obfuscation and algorithmic countermeasures against linguistic post-processing, as ASCA is no longer confined to laboratory conditions but is viable in shared, noisy, and public environments.