I Know What You Did Last Summer: Identifying VR User Activity Through VR Network Traffic

1. Problem Statement and Motivation

The paper addresses a critical privacy and security vulnerability in Virtual Reality (VR) ecosystems. As VR adoption grows, headsets generate significant network traffic to support multi-user interactions, cloud rendering, and synchronization. Even when this traffic is encrypted (end-to-end), the paper posits that the metadata and traffic patterns (packet sizes, inter-arrival times, transmission rates) contain sufficient information to infer user behavior.

The core problem is that malicious actors (e.g., on the same Wi-Fi network) can intercept this traffic and use Machine Learning (ML) to:

• Identify which specific VR application a user is running.
• Deduce specific user activities (e.g., walking, talking, throwing a virtual object, flying) within that application.
• Build detailed user profiles for social engineering or behavioral tracking.

2. Methodology

The authors employed a structured approach involving data collection, feature engineering, and model training to validate their hypotheses.

A. Data Collection

• Hardware: Meta Quest Pro headset.
• Dataset: Network traffic traces from 25 distinct VR applications across various genres (games, social, productivity, fitness).
• Activities: Users performed specific actions within each app (e.g., "Making a Fist," "Walking," "Talking," "Ball Throwing").
• Capture Tool: Wireshark was used to capture TCP segments and UDP datagrams.
• Constraints: The study assumed encrypted traffic (no payload inspection) and a single active application per headset at any time.

B. Preprocessing and Feature Engineering

Raw packet data was processed to extract temporal and spatial features relevant to ML models:

• Temporal Metrics: Delta time between consecutive packets, average packet length.
• Statistical Aggregates: Calculated over rolling 5-second windows:
  • Maximum bits per second.
  • Standard deviation of time intervals.
  • Total sum of bits/packets.
  • Variance and Cumulative Distribution Function (CDF) values.
  • Minimum/Maximum values for packet counts and lengths.
• Noise Reduction: The study noted that "No Activity" (idle) states often resemble noise, which can degrade model performance.

C. Machine Learning Models

Three off-the-shelf ML models were trained and compared:

1. Random Forest (RF): An ensemble method (300 trees) chosen for its robustness against overfitting.
2. Decision Tree (DT): A single tree classifier.
3. Deep Learning (DL): A Multi-Layer Perceptron (MLP) with ReLU activation and Softmax output.

The dataset was split 80/20 for training and testing, with hyperparameter tuning and cross-validation used to ensure generalization.

3. Key Contributions

1. Comprehensive Dataset: Creation of a labeled network traffic dataset from 25 diverse VR applications and numerous user activities on the Meta Quest Pro.
2. Feasibility Demonstration: Proof that encrypted network traffic alone is sufficient to identify both the application and the specific user activity with high accuracy.
3. Data Efficiency Analysis: Demonstration that attackers do not need massive datasets; less than 10 minutes of traffic per application is sufficient for high-accuracy identification.
4. Cross-Application Generalization: Evidence that models trained on known apps can identify activity types (e.g., "walking") in unknown applications, suggesting common underlying traffic patterns (likely due to shared game engines like Unity or FMOD).

4. Key Results

A. Application Identification (RQ1)

• Best Model: Random Forest achieved the highest accuracy.
• Accuracy: 92.4% (Precision: 93%, Recall: 92%).
• Comparison: Decision Tree (87.1%) and Deep Learning (80.2%) performed lower. Deep Learning struggled with smaller datasets and specific applications (e.g., 0% accuracy on Monkey Doo), while Random Forest maintained consistency.

B. User Activity Identification (RQ2)

• Best Model: Random Forest.
• Accuracy: 91.2% (Precision: 91%, Recall: 91%).
• Nuance: Removing "No Activity" (idle) states from the dataset significantly improved accuracy (up to 96.16% for RF), as idle states introduced noise that confused the models.
• Specific Activities: Dynamic activities like "Walking" in Gorilla Tag or "Ball Throwing" achieved 100% accuracy. Static or subtle actions were harder to distinguish but generally remained above 80%.

C. Cross-Application and Unknown App Identification (RQ3 & RQ4)

• Models could reliably identify activity types (e.g., "Talking") across different applications, indicating that traffic signatures are often engine-specific rather than app-specific.
• When tested on traffic from an unknown application, the models successfully identified the user activities taking place, proving the generalizability of the approach.

D. Data Requirements (RQ5)

• Small Data Sufficiency: Training on just 20% of the collected data (approx. <10 minutes of traffic per app) yielded:

  • 91% accuracy for application identification.

  • 88% accuracy for activity identification.

• Increasing the dataset size beyond this point yielded diminishing returns (<3% improvement).

E. System Performance (RQ6)

• Resource Efficiency: The models are computationally lightweight.
  • Training Time: ~96 seconds for Random Forest on the full dataset.
  • CPU Usage: ~62% (RF) and ~11% (DT).
  • Memory: ~1.38 GB.
• Inference Time: Ranged from 0.69s to 3.16s depending on the number of activities being classified, making real-time monitoring feasible for attackers.

5. Significance and Implications

• Privacy Risk: The study demonstrates that encryption alone is insufficient to protect VR user privacy. Traffic analysis (side-channel attacks) can reveal intimate details about a user's physical movements, social interactions, and software usage.
• Threat Model: An attacker on a local network (e.g., public Wi-Fi) can easily deploy these models to profile users without needing to break encryption or access the device directly.
• Future Directions: The findings suggest a need for new privacy-preserving techniques in VR, such as traffic padding, constant-rate transmission, or obfuscation of packet timing, to prevent activity fingerprinting.

In conclusion, the paper provides a stark warning: the network traffic generated by modern VR headsets acts as a "digital fingerprint" that allows adversaries to reconstruct a user's virtual reality experience with high precision using standard machine learning techniques.