Empirical Evaluation of Multi-Modal Touch Detection in Over-the-Shoulder Video Surveillance

Technical Summary: Empirical Evaluation of Multi-Modal Touch Detection in Over-the-Shoulder Video Surveillance

Problem Statement
The paper addresses the feasibility of "virtual keylogging"—decoding user keystrokes on mobile devices from over-the-shoulder video surveillance without direct screen access. While Video Intelligence Surveillance (VIDINT) aims to decode fine-grained human-computer interactions, this task is historically hindered by perspective distortions, motion blur, partial hand occlusions, and ambient noise. Existing approaches often rely on supervised learning requiring large annotated datasets or specific calibration, which limits their applicability in opportunistic, uncontrolled surveillance scenarios. This study evaluates whether a training-free, multi-modal filter pipeline can successfully reconstruct typing sequences from such footage.

Methodology
The authors evaluate the BEHINT (Behavioral Intelligence) framework, a training-free system designed to detect touch events and map them to a reference keypad layout. The pipeline operates in two stages:

1. Multi-Modal Detection: Four parallel detectors process video frames to identify candidate finger contacts:

   • MediaPipe Anatomical Landmarks: Configured with low confidence thresholds (0.3) to capture partial hand skeletons (21 landmarks) even under occlusion.
   • HSV Skin Filtering: Segments skin color in HSV space ($H \in [0, 20], S \in [20, 255], V \in [70, 255]$) with morphological cleaning.
   • Temporal Frame Differencing: Detects motion by computing absolute pixel differences between consecutive frames.
   • Shape-Guided Edge Detection: Uses Canny edge detection to find finger boundaries, filtering contours by area and aspect ratio.
   • Outputs are merged via spatial clustering (DBSCAN) and mapped to a standard 4×3 grid layout.

2. Evaluation Strategy:

   • Staged Evaluation: A 120-frame first-person video of a known 4-key passcode entry ('2'→'5'→'8'→'#') serves as the ground truth for calculating Precision, Recall, F1-score, and Sequence Similarity.
   • Ablation & Sensitivity: The system undergoes ablation studies, resolution decay analysis (downscaling to 25%), noise injection (Gaussian $\sigma \in [0, 30]$), and proximity threshold tuning.
   • Real-World Generalization: The pipeline is tested on five uncontrolled, third-party videos (e.g., dialers, blank screens, green screens) without ground truth labels. Instead, the authors measure "detection volume" (touch points per frame) to identify failure mechanisms.

Key Contributions

• Reproducible Benchmark: A 120-frame staged video with known ground truth and a four-modality detection pipeline for over-the-shoulder keystroke inference.
• Operational Envelope Characterization: An ablation and sensitivity analysis demonstrating that the system's performance is not limited by resolution but by inherent flaws in the detection logic.
• Generalization Audit: A label-free audit on real-world footage revealing that the system over-reports contacts by 1–3 orders of magnitude, failing to distinguish between hand presence and actual fingertip contact.

Results

• Staged Performance: On the controlled 120-frame clip, the combined multi-modal system achieved an F1-score of 16.7% and a sequence similarity of 3.0%.
  • Individual modalities performed poorly: MediaPipe and Skin Detection failed entirely (F1 = 0%) due to occlusion and noise. Motion-Only and Edge-Only achieved slightly higher F1-scores (18.5% and 18.2%, respectively).
  • Resolution changes had negligible impact on performance, indicating the failure is not due to image detail.
  • Noise sensitivity was significant, with F1 dropping to 10.7% at $\sigma=20$.
  • Proximity threshold tuning yielded no performance variation, suggesting the detector emits a dense, spatially diffuse cloud of false positives that cannot be filtered by simple merging.
• Real-World Failure: On five real-world videos, the detector emitted a median of 57 touch points per frame (peaking at 205), whereas a real typing session generates approximately 0.2 taps per frame.
  • The HSV Skin Filter was the primary source of error, accounting for up to 100% of detections in some clips. It responded to the entire visible hand rather than specific fingertip contact.
  • The system failed to generalize; without hand-tuned device regions and known layouts to discard the massive volume of spurious data, no keystroke sequence could be recovered.

Significance and Claims
The paper concludes that a training-free, off-the-shelf multi-modal filter pipeline does not achieve reliable keystroke reconstruction from over-the-shoulder video outside of a strictly calibrated, staged setting.

• The authors explicitly state that their work is not a new attack but an empirical evaluation of whether a system an opportunistic adversary could assemble from free components actually works.
• The findings suggest that the "skin filter" mechanism is fundamentally flawed for this task because it detects skin area rather than contact.
• The paper argues that reliable virtual keylogging via this approach would require learned fingertip-contact classifiers and automatic device-region detection, rather than the classical filters evaluated here.
• The authors identify the need for a large-scale, labeled dataset with frame-level touch annotations as the necessary next step for future research, rather than claiming current methods are viable.