Multimodal User Authentication Method via Fusion of Keystroke Dynamics and Glove-Based Hand Kinematics

Technical Summary: Multimodal User Authentication via Fusion of Keystroke Dynamics and Glove-Based Hand Kinematics

Problem Statement
Keystroke Dynamics (KD) is a cost-effective behavioral biometric for continuous authentication, relying on temporal features such as dwell time and flight time. However, unimodal KD systems suffer from significant limitations in high-security applications. They are highly susceptible to environmental variations (e.g., hardware differences between desktop and laptop keyboards), user fatigue, and sophisticated presentation attacks, such as "robot adversarial attacks" that mimic human typing rhythms. Existing device-centric physical sensing approaches (e.g., instrumented keyboards) often require specialized hardware modifications that hinder cross-device applicability, while vision-based methods face privacy and occlusion issues. Furthermore, prior wearable solutions typically capture only macro-level arm movements or are limited to one-time authentication tasks, leaving a gap in continuous authentication systems that can capture fine-grained finger pressure and 3D micro-motions simultaneously.

Methodology
The authors propose a robust multimodal authentication framework that fuses traditional keystroke dynamics with 19-dimensional hand kinematics and pressure dynamics.

• Hardware Architecture: A bespoke data glove was developed as a Proof of Concept (PoC) platform. It features:

  • 10 Piezoresistive Pressure Sensors: Placed on the fingertips and first joints (DIP) of the right hand using Velostat material to capture contact force distribution.
  • 9-Axis IMU: Integrated into a custom ESP32-based control unit on the wrist to capture 3-axis acceleration, angular velocity, and geomagnetism during the "flight time" between keystrokes.
  • Circuitry: A non-inverting amplifier circuit was designed to convert minute resistance changes from the Velostat sensors into detectable voltage ranges for the microcontroller.

• Data Processing and Model:

  • Preprocessing: Raw sensor data were filtered, normalized (Z-score), and synchronized with keystroke events. To handle variable typing speeds, a sliding window of 600 ms (centered on key press) was extracted and linearly interpolated to a fixed length of 45 time steps.
  • Deep Learning Architecture: A hybrid CNN-LSTM model was employed to process the 19-dimensional time-series data.
    • CNN Blocks: Two 1D-convolutional layers (expanding channels from 19 to 64, then to 128) extract spatial features from the sensor signals.
    • LSTM Block: A 2-layer stacked LSTM (hidden size 64) models temporal dependencies, condensing the sequence into a static feature vector.
    • Classification: A fully connected layer maps the features to "Genuine" or "Impostor" classes, followed by a Softmax function.
  • Post-Processing: A moving-average smoothing technique aggregates probability scores over consecutive keystrokes to filter instantaneous noise and improve decision stability.

• Evaluation Protocol: The study utilized a rigorous "unseen" evaluation protocol to test real-world applicability:

  • Training: Data from 1 target user and 8 "known" impostors collected on a standard desktop keyboard.
  • Testing: Evaluated on a laptop keyboard (cross-domain) against the target user and a single "unknown" impostor excluded from training.
  • Metrics: False Acceptance Rate (FAR), False Rejection Rate (FRR), and Equal Error Rate (EER).

Key Contributions

1. Multimodal Wearable Integration: The design of a custom data glove integrating 10 pressure sensors and a 9-axis IMU to extract fine-grained physical typing dynamics directly from the user's hand, establishing a high-fidelity PoC for continuous authentication.
2. Redundancy Against Unimodal Vulnerabilities: The proposal of a framework fusing temporal KD with physical traits (spatial motion and contact pressure), demonstrating that these modalities inherently compensate for each other's vulnerabilities against environmental noise and spoofing.
3. Robust Data Fusion and Unseen Evaluation: The implementation of a hybrid CNN-LSTM architecture that achieves superior robustness against unseen impostors and cross-domain hardware variations.

Experimental Results

• Overall Performance: Averaged over five independent trials, the multimodal method achieved a mean EER of 2.12% for individual 600-ms authentication events (Window 1).
• Temporal Smoothing: By aggregating scores over a sequence of 10 events (approximately 6 seconds), the system achieved a perfect authentication rate with a mean EER of 0.00%.
• Ablation Studies:
  • Architecture: The hybrid CNN-LSTM model outperformed single-stream baselines (CNN-only, LSTM-only, GRU, Transformer, etc.).
  • Modalities: In a static laboratory setting, the IMU-only model achieved 0.00% EER, equivalent to the fused model. However, the pressure-only model had a significantly higher EER (6.05%).
  • Error Analysis: While pressure and IMU data showed strong physical correlation (Pearson's r = 0.881), they exhibited distinct failure modes. Pressure sensors were highly sensitive to micro-misalignments caused by donning/doffing the glove across different sessions, whereas IMU data remained stable. Conversely, IMU data is theoretically vulnerable to macroscopic environmental vibrations and "spatial spoofing" (mimicking motion without contact).

Significance and Claims
The paper claims that while the IMU alone performed well in controlled conditions, the fusion of modalities is critical for real-world deployment. The primary significance lies in the creation of a physical "fail-safe" mechanism. By combining modalities with distinct error-inducing factors, the system ensures reliability even when one sensor is compromised by environmental noise (e.g., vibrations affecting IMU) or behavioral factors (e.g., sensor shift affecting pressure).

The authors emphasize that their approach decouples the biometric signature from the input device's hardware characteristics, allowing for cross-domain authentication (desktop to laptop) without the need for specialized keyboards. While the current form factor (a data glove) is a PoC, the study validates the hypothesis that combined physical traits provide significantly stronger biometric discrimination than timing features alone, paving the way for future miniaturization into unobtrusive wearables (e.g., smartwatches with pressure-sensitive keyboards).