Technical Summary: BRAVESPY

Problem Statement

While recent security research has identified privacy leakages in Virtual Reality (VR) and Metaverse devices, these vulnerabilities have largely been confined to observable behaviors (e.g., virtual keystrokes, app launches, and user avatars). A critical gap remains in understanding the leakage of unobservable brain-level perceptions—specifically, what a user is perceiving or thinking about within the virtual scene.

The paper posits that the tight coupling of VR headsets to the user's forehead and eyes allows inbuilt motion sensors (accelerometers and gyroscopes) to capture subtle physiological signals. Specifically, the authors hypothesize that pupillary responses (dilation and contraction) triggered by visual stimuli induce minute facial vibrations. These vibrations are highly correlated with brain EEG signals (electroencephalography). Since mainstream VR SDKs (e.g., OpenXR, Oculus Mobile SDK) impose no restrictions on accessing motion sensor data, malicious applications can continuously log this data in the background without user permission, potentially reconstructing EEG-correlated representations to infer sensitive, unobservable user states.

Methodology: BRAVESPY

The authors design and implement BRAVESPY, an end-to-end attack system that reconstructs brain EEG signals from unrestricted motion sensor data to infer privacy at three distinct levels.

1. System Overview

BRAVESPY operates in three stages:

1. Motion Data Processing:
   • Coordinate Calibration: Transforms 3D acceleration data from the headset's local coordinate system to a common body reference system to mitigate posture variations.
   • Body Movement Mitigation: Uses a Short-Time Energy (STE) approach with an adaptive filter (optimized via Kullback-Leibler divergence) to remove noise caused by spontaneous head movements and body rotations.
   • Eye-Blink Detection: Identifies eye-blink events using moving variance windows and Linear Discriminant Analysis (LDA) to segment relevant signal portions.
2. Feature Extraction:
   • Applies Savitzky-Golay filters to remove high-frequency noise (e.g., speech frequencies).
   • Segments signals based on variance peaks associated with pupillary responses.
   • Converts time-series signals into time-frequency spectrograms using Short-Term Fourier Transform (STFT) to create feature representations.
3. User Privacy Inference:
   • Brainwave Reconstruction: A Conditional Generative Adversarial Network (cGAN), based on the pix2pix architecture, is trained to map acceleration spectrograms to EEG spectrograms. The generator creates EEG representations, while the discriminator distinguishes between real and reconstructed signals.
   • Activity Recognition:
     • Observable Levels: A Vision Transformer (ViT) model classifies reconstructed EEG spectrograms to infer UI-level activities (websites, apps, videos) and User-level attributes (identity, gaze direction).
     • Unobservable Level: The reconstructed EEG signals are used to fine-tune a pre-trained Stable Diffusion model (via DreamDiffusion) to generate images representing the user's brain-level visual perceptions.

2. Threat Model

• Adversary: An intentional VR app developer or third-party plugin.
• Capability: Access to unrestricted motion sensor data (accelerometer/gyroscope) via standard SDKs without requiring elevated privileges, external hardware, or eye-tracking sensors.
• Goal: To infer sensitive information ranging from UI interactions to unobservable brain perceptions.

Key Contributions

1. Novel Attack Vector: Identification of unrestricted motion sensors in VR headsets as a viable channel for reconstructing EEG-correlated perceptions, bridging the gap between Metaverse technology and Brain-Computer Interfaces (BCI).
2. Three-Level Privacy Model: A systematic investigation of privacy leakage across:
   • UI-Level: Inferring specific websites, apps, and streaming videos.
   • User-Level: De-anonymizing users and tracking gaze movements/keystrokes.
   • Brain-Level: Reconstructing unobservable visual perceptions (images) from the user's mind.
3. Implementation and Validation: The design of BRAVESPY and its evaluation on four commercial VR headsets (Meta Oculus Quest 2, Meta Oculus Quest, PICO 4, HTC Vive Pro).

Experimental Results

The system was evaluated on 25 participants across 40 image classes and various VR applications.

• UI-Level Privacy:
  • Website Fingerprinting: Achieved 89.3% accuracy across 50 websites. Performance varied based on visual stimulus intensity (e.g., 100% for video-heavy sites like Twitch, ~70% for static search engines).
  • App Fingerprinting: Achieved 85.8% accuracy across 50 VR apps.
  • Video Fingerprinting: Achieved 80.0% accuracy for Netflix streaming videos, with higher accuracy for dynamic content (action movies) compared to static content.
• User-Level Privacy:
  • De-anonymization: Achieved 98.4% accuracy in identifying users based on eye-blink induced EEG patterns.
  • Gaze Tracking: Achieved 97.6% accuracy in recognizing 8 directions of gaze movement.
  • Keystroke Inference: By mapping gaze zones to a virtual keyboard, the system achieved 10.4% top-1 accuracy and 92.0% top-50 accuracy in inferring virtual keystrokes.
• Brain-Level Privacy:
  • Image Perception: Successfully reconstructed brain-level visual perceptions. The system achieved a 67.2% class accuracy in identifying which of 40 ImageNet classes (e.g., "Cat," "Dog") the user was visualizing. Accuracy ranged from 49% to 83% depending on image class heterogeneity.

Robustness: The system maintained high performance across different VR headsets (though brain-level accuracy dropped to ~52% on PICO 4 and HTC Vive due to sensor sensitivity differences), varying sampling rates (optimal at 500 Hz), and different light intensities.

Significance and Claims

The paper claims that BRAVESPY is the first study to demonstrate the feasibility of inferring unobservable brain-level perceptions in the Metaverse using only unrestricted motion sensors.

• Beyond Observable Behaviors: Unlike prior works limited to coarse-grained inference of observable actions, BRAVESPY reveals a new attack surface where "what a user is seeing" (observable) is distinguished from "what a user is perceiving" (unobservable).
• Privacy Implications: The ability to reconstruct EEG-correlated representations poses severe threats to user privacy, potentially exposing sensitive cognitive states, biometric identities, and visual thoughts without the user's knowledge or consent.
• Countermeasures: The authors propose permission-based sensor management and signal obfuscation (e.g., adding Gaussian noise) as potential defenses, noting that these measures significantly reduce inference accuracy (e.g., reducing brain-level inference accuracy by ~38.6%) but may impact VR usability.

The authors conclude that while the current framework is a proof-of-concept with limitations in generalizability across different headsets and brain activities, it establishes a critical precedent for understanding the intersection of VR sensor data and neural privacy.