🤖 machine learning

SurfaceXR: Fusing Smartwatch IMUs and Egocentric Hand Pose for Seamless Surface Interactions

SurfaceXR is a sensor fusion system that combines headset-based hand tracking with smartwatch IMU data to overcome the limitations of mid-air gestures and egocentric vision, enabling robust and accurate touch tracking and gesture recognition on everyday surfaces.

Original authors: Vasco Xu, Brian Chen, Eric J. Gonzalez, Andrea Colaço, Henry Hoffmann, Mar Gonzalez-Franco, Karan Ahuja

Published 2026-03-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vasco Xu, Brian Chen, Eric J. Gonzalez, Andrea Colaço, Henry Hoffmann, Mar Gonzalez-Franco, Karan Ahuja

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to write a letter on a piece of paper that is floating in mid-air. You'd have to hold your arm up the whole time, your muscles would get tired (the "gorilla arm" effect), and your hand would probably shake, making your handwriting messy. That's what interacting with Virtual Reality (VR) and Augmented Reality (AR) often feels like right now.

Now, imagine instead that you could just rest your hand on a real table, a wall, or even your knee, and use that surface to control the digital world. It would be comfortable, precise, and natural. But here's the catch: computers are terrible at knowing exactly when your finger is touching that table versus just hovering slightly above it. They get confused by shadows, weird angles, and the fact that your hand might block the camera's view.

Enter SurfaceXR. Think of it as a "super-sense" for your VR headset.

The Problem: The "Blind" Camera

Your VR headset has cameras that act like eyes. They can see your hand, but they aren't perfect.

The Glitch: Sometimes the camera thinks your finger is floating 2 inches above the table when it's actually touching it (a "False Negative").
The Ghost Touch: Other times, it thinks you're touching the table when you're actually just waving your hand in the air (a "False Positive").
The Blind Spot: If you move your hand to the side where the camera can't see it, the system goes blind.

The Solution: The "Secret Vibe" Detector

The researchers realized that while the camera's "eyes" are sometimes blurry, your wrist has a secret superpower: vibration.

When you tap a table, your finger sends a tiny, high-frequency "shiver" or vibration through your hand and up your wrist. A regular camera can't see this shiver, but a smartwatch (like the Google Pixel Watch) has a built-in motion sensor (IMU) that is incredibly sensitive to these tiny vibrations.

SurfaceXR is like a detective team:

The Eye (Headset Camera): It sees where your hand is in 3D space. It's good at the big picture but bad at the fine details.
The Ear (Smartwatch): It listens to the vibrations of your wrist. It can't tell exactly where you are, but it knows exactly when you hit a surface because of the "thump."

How They Work Together (The Magic Fusion)

The system fuses these two inputs using a smart AI brain.

When the camera is confused: If the camera thinks your hand is floating, but the smartwatch feels a "thump," the AI says, "Ah! The camera is wrong. You are actually touching the table!"
When the camera is blocked: If you move your hand behind your back (out of the camera's view), the camera goes silent. But the smartwatch is still there, feeling the vibrations. The system can still tell you are tapping or swiping on a surface, even though the camera can't see you.

It's like having a friend whispering in your ear ("I feel a tap!") while you are trying to look at something with your eyes. If your eyes are closed, your friend's whisper saves the day.

What Can You Do With It?

Because this system is so accurate and doesn't need special sensors on the table (just a normal watch and a normal headset), it opens up cool new ways to play and work:

Drawing: You can sketch on a virtual canvas that is anchored to your real kitchen table. The lines are smooth because the system knows exactly when your pen touches the "paper."
Typing: You can type on a virtual keyboard on your lap. The system knows when you hit the keys, so you don't have to hover your fingers in the air.
Navigation: You can swipe through photos on a virtual wall. If you swipe up, the system knows instantly.

The Best Part: It's Already in Your Pocket

You don't need to buy a $500 special glove or stick sensors on your furniture.

The Watch: You probably already have a smartwatch.
The Headset: It works with standard VR headsets like the Meta Quest.
The Magic: It's all in the software that combines the two.

In a Nutshell

SurfaceXR is like giving your VR headset a "sixth sense." It combines the sight of the camera with the feeling of the smartwatch to create a perfect, comfortable way to touch and interact with the digital world, right on the surfaces around you. It turns your coffee table, your wall, or your knee into a super-accurate computer mouse and keyboard, without making your arms tired.

1. Problem Statement

Extended Reality (XR) interaction faces significant challenges with current input methods:

Mid-air gestures: Often cause fatigue ("gorilla arm" effect), discomfort, and imprecision during extended use.
Vision-based surface tracking: While surface interactions offer better accuracy and comfort, relying solely on headset egocentric cameras (e.g., Meta Quest) struggles with:
- Occlusions: Hands or fingers blocking the camera view.
- Oblique angles: Difficulty estimating the surface plane from non-perpendicular views.
- Tracking errors: Virtual hand offsets leading to false positives (system thinks user is touching when they aren't) or false negatives (system misses a touch).
Existing solutions: Prior work often requires specialized hardware (instrumented surfaces, high-frequency ring sensors) or is limited to simple tap events, lacking scalability and practicality for everyday use.

2. Methodology

SurfaceXR introduces a multimodal sensor fusion approach that combines standard headset hand tracking with micro-vibration data from commodity smartwatches.

A. Data Modalities

Hand Tracking (Visual):
- Source: Meta Quest 3 headset (60 Hz).
- Input: 3D positions and accelerations of 8 joints (4 per finger) for the index finger and thumb.
- Normalization: Joint positions are normalized relative to a surface-aligned coordinate system (or head-centric if no surface is registered) to handle scale and orientation differences.
IMU Data (Inertial):
- Source: Google Pixel Watch 3 (200 Hz).
- Input: 3-axis accelerometer and 3-axis gyroscope.
- Signal Processing: Raw data is normalized and passed through a signal processing pipeline using Butterworth bandpass filters (0.22–8 Hz, 8–32 Hz, and >32 Hz) to isolate specific motion frequencies. This captures the distinctive micro-vibrations generated when a finger contacts a surface.

B. Neural Network Architecture

The system employs a Multimodal Multi-Task Deep Neural Network:

Hand Encoder: A Temporal Convolutional Network (TCN) processes the sequential hand tracking data to capture both immediate finger movements and longer-term gestural patterns.
IMU Encoder: A 2D Convolutional Neural Network (CNN) extracts temporal patterns from the multi-channel IMU data.
Fusion Mechanism: Instead of simple concatenation, the system uses a Gated Fusion mechanism. Features from both encoders are projected into a shared space and weighted by a learned gating network (sigmoid output), allowing the model to dynamically prioritize the most reliable modality based on context (e.g., relying more on IMU when the hand is occluded).
Output Heads:
1. Touch Contact State: Binary classification (Touch vs. No-Touch) for frame-level detection.
2. Gesture Recognition: 9-class classification (8 gestures + "No Gesture") for window-level prediction.

C. Operating Modes

SurfaceXR is designed for flexibility, functioning in three modes:

Multimodal (Optimal): Fuses both hand tracking and IMU data.
Hand-Only: Functions using only headset cameras (useful when the watch is unavailable).
IMU-Only: Functions using only the smartwatch (crucial when hands are outside the camera's field of view).

3. Key Contributions

Novel Sensor Fusion: First work to combine commodity smartwatch IMUs with egocentric hand tracking for continuous surface interaction, eliminating the need for specialized hardware or instrumented surfaces.
Robustness to Occlusion and Calibration: The system remains effective even without explicit surface plane registration and can detect interactions when hands are outside the camera's field of view (IMU-only mode).
Comprehensive Gesture Set: Supports a rich set of interactions beyond simple taps, including swipes (4 directions), pinches, and double-taps.
Real-Time Pipeline: Achieves low-latency inference suitable for interactive applications.

4. Experimental Results

The system was evaluated with 21 participants across diverse scenarios (horizontal tables, vertical walls) using a leave-one-participant-out cross-validation approach.

Gesture Recognition:
- Multimodal F1-Score: 95.0% (across 9 classes).
- Ablation: Outperformed Hand-Only (92.6%) and IMU-Only (82.7%) significantly ( $p < .001$ ).
- Robustness: Without surface plane registration, the multimodal model dropped only 2.5% (to 92.6%), whereas the Hand-Only model dropped 4.4%.
Touch Contact Detection:
- F1-Score: 91.2% (Window-level).
- Latency: Median onset latency of 47 ms and offset latency of 170 ms.
- Comparison: Significantly outperformed a baseline threshold-based method (74.0% F1) and hand-only models, demonstrating the value of IMU micro-vibration cues.
Spatial Accuracy:
- Cross-Hair Targeting: Mean error of 7.07 mm.
- Path Tracing: Mean error of 12.4 mm.
- Note: While slightly higher than specialized depth-camera systems (e.g., MRTouch at 5.4 mm), SurfaceXR achieves this without additional hardware or calibration.
System Latency: Total pipeline latency is 13.61 ms, enabling real-time inference at 60 Hz.

5. Significance and Applications

SurfaceXR addresses the "gorilla arm" fatigue and precision issues of mid-air gestures by enabling natural surface interactions on everyday objects (tables, walls) without modifying the environment.

Practicality: Uses off-the-shelf hardware (Quest 3 + Pixel Watch), making it highly scalable and deployable.
Versatility: Demonstrated in several applications:
- AR/VR Sketching: Natural drawing with color selection and stroke width control.
- Text Entry & Trackpad: A dual-input interface allowing typing and cursor navigation, with automatic fallback to IMU-only if hands are occluded.
- AR Navigation: Photo album browsing using only the smartwatch (ideal for AR glasses where hands are often out of view).
Future Impact: Paves the way for more intuitive, comfortable, and robust XR interfaces, particularly in scenarios where vision-based tracking is unreliable.

Limitations & Future Work

Currently limited to single-touch (index finger) and right-handed users.
Tested primarily on flat, rigid surfaces; performance on soft or non-planar surfaces needs evaluation.
Future work aims to optimize for power efficiency (on-device inference) and integrate additional modalities (audio, UWB) to handle extreme occlusion.