💻 computer science

Point & Grasp: Flexible Selection of Out-of-Reach Objects Through Probabilistic Cue Integration

This paper introduces "Point & Grasp," a mixed reality interaction technique that uses a probabilistic framework to flexibly integrate multiple user cues—specifically pointing direction and grasp gestures—to more accurately and robustly select out-of-reach objects.

Original authors: Xuejing Luo, Hee-Seung Moon, Christian Holz, Antti Oulasvirta

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuejing Luo, Hee-Seung Moon, Christian Holz, Antti Oulasvirta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a high-tech video game or working in a virtual office, and you see a tool you need—like a screwdriver—floating high up on a shelf or far across the room. You can’t physically reach it, so you have to "point and click" to grab it.

In current Virtual Reality (VR), this is actually surprisingly annoying. It’s like trying to pick a specific grape out of a crowded bowl using only a long, shaky laser pointer. If the grapes are too close together, your laser hits the wrong one. If you try to use a "grabbing" motion, but all the fruits look similar, the computer gets confused.

This paper introduces a new way to solve this called Point&Grasp.

The Problem: The "Shaky Laser" and the "Generic Grab"

The researchers identified two main ways people currently try to select things in VR, and both have "blind spots":

The Directional Cue (The Shaky Laser): You point your hand at the object. This works great if the object is alone, but if there’s a cluster of items, your "laser" is too imprecise. It’s like trying to point at a single hair on a cat's head from across the room.
The Gestural Cue (The Generic Grab): You make a shape with your hand that looks like you're grabbing something. This is great for telling the computer what you want (e.g., "I'm making a shape for a handle"), but if you have three different tools with similar handles, the computer doesn't know which one you mean.

The Solution: The "Smart Detective" (Bayesian Integration)

Instead of forcing the computer to choose between the "laser" or the "grab," the researchers created a "Smart Detective" (technically called a Probabilistic Framework).

Think of it like this: Imagine you are in a dark room and you want to find a specific coffee mug.

Cue 1 (The Laser): You shine a flashlight in a general direction. It hits a cluster of three mugs. You know it's one of those three, but you aren't sure which.
Cue 2 (The Grab): You hold your hand in a specific way, like you're gripping a thin handle.

The Smart Detective doesn't just look at the flashlight or just look at your hand. It combines the clues. It says: "Okay, the flashlight is pointing at these three mugs, but the hand shape only really fits the handle of the blue one. Therefore, there is a 95% chance they want the blue mug."

By mathematically "fusing" these two clues, the system becomes much more accurate and much faster.

How they built it: The "Training Camp"

To make this detective smart, they couldn't just use old data. Most VR data is about people grabbing things they can actually touch. But when you grab something out of reach, your hand behaves differently—you tend to stretch your fingers more or hold your hand differently because you're "imagining" the touch.

The researchers created a massive new dataset called ORG (Out-of-Reach Grasping). They had people in VR practice "imaginary grabbing" hundreds of times. They recorded every finger position and every object shape. They then used this "training camp" to teach a neural network how to recognize the difference between a "mug grab" and a "hammer grab" in mid-air.

The Result: A Smoother Virtual World

When they tested Point&Grasp against the old methods, the results were clear:

It’s faster: You don't spend time "fine-tuning" your pointer.
It’s more reliable: Even when objects are crowded together or look similar, the "Smart Detective" usually figures it out.
It feels natural: Users felt like they were interacting with the world the way they do in real life—using both direction and shape to communicate intent.

In short: Instead of making you be a perfect marksman with a laser pointer, Point&Grasp lets you be a human, combining your pointing and your gesturing to tell the computer exactly what you want.

Technical Summary: Point & Grasp

Title: Point & Grasp: Flexible Selection of Out-of-Reach Objects Through Probabilistic Cue Integration
Authors: Xuejing Luo, Hee-Seung Moon, Christian Holz, and Antti Oulasvirta (CHI ’26)

1. Problem Statement

In Mixed Reality (MR), users frequently need to select objects that are physically out of reach (e.g., virtual tools in a 3D workspace). Existing selection methods typically rely on a single type of user-generated cue, which leads to performance degradation when that cue becomes ambiguous:

Directional Cues (e.g., Raycasting): Suffer from spatial ambiguity when objects are small, densely clustered, or occluded.
Gestural Cues (e.g., Grasping): Suffer from semantic ambiguity when multiple objects share similar shapes or functional affordances (e.g., a mug vs. a wineglass).

Current multi-cue approaches are often deterministic (rule-based), meaning they assign fixed roles to cues (e.g., gaze for selection, gesture for command). This prevents the system from flexibly shifting reliance to a different cue when the primary one becomes unreliable.

2. Methodology

The authors propose Point&Grasp, a technique that treats out-of-reach selection as a probabilistic intent inference problem using a Bayesian framework.

A. Bayesian Cue Integration Framework
The system models the posterior probability of an object $o$ being the intended target given multiple observed cues $C$ (direction $c_D$ and gesture $c_G$ ):
$p(o|C) \propto p(o) \cdot p(c_D|o) \cdot p(c_G|o)$

Directional Likelihood $p(c_D|o)$ : Modeled as a Gaussian distribution centered at the object's position, where the ray endpoint (wrist position) serves as the observation.
Gestural Likelihood $p(c_G|o)$ : Estimated via a learned neural network that captures the compatibility between a hand configuration and an object's geometry.

B. Gesture-Object Likelihood Model
To implement the gestural likelihood, the authors developed a dual-branch encoder architecture:

Hand Encoder: Processes 3D joint positions (relative to the wrist) through an MLP to create a 256-dimensional gesture embedding.
Object Encoder: Converts object point clouds into a Basis Point Set (BPS) representation, then processes them through an MLP to create a 256-dimensional object embedding.
Fusion: The embeddings are concatenated and passed through a fusion MLP with a sigmoid activation to output the probability $p(c_G|o)$ .

C. Data Collection (The ORG Dataset)
To train the model, the authors introduced the Out-of-Reach Grasping (ORG) dataset. Unlike existing datasets that focus on physical, in-reach grasping, ORG captures mid-air gestures performed toward distant objects. It includes:

Positive samples: Gestures intended for a specific object.
Negative samples: Subjective evaluations of whether a gesture is compatible with "distractor" objects.

3. Key Contributions

Probabilistic Framework: A principled Bayesian approach for fusing heterogeneous, complementary cues (spatial vs. semantic) for intent inference.
Point&Grasp Technique: A novel interaction method that utilizes a single hand-tracking input to provide both directional and semantic information.
ORG Dataset: The first systematic dataset of mid-air, out-of-reach grasping gestures, providing the necessary ground truth for probabilistic modeling.
Pose-Aware Model: A deep learning model capable of generalizing across different users and partially transferring to unseen object shapes.

4. Results

The authors conducted two user studies to validate the system:

Study 1 (vs. Single-Cue Baselines):
- Performance: Point&Grasp significantly outperformed "Point-only" and "Grasp-only" methods in both selection time and completion rate.
- Robustness: It successfully mitigated the weaknesses of individual cues. When spatial ambiguity was high, the system leaned on the gesture; when semantic ambiguity was high, it leaned on the direction.
- Cue Agreement: The system was able to "rescue" selections even when neither cue alone was sufficient, by accumulating weak evidence from both.
Study 2 (vs. State-of-the-Art):
- Compared against BubbleRay (adaptive region) and Expand (progressive refinement).
- Findings: Point&Grasp was faster than Expand in low-spatial-ambiguity scenes (avoiding refinement overhead) and faster than BubbleRay in high-spatial-ambiguity scenes. It demonstrated superior robustness by using semantic information to reduce dependence on precise pointing.

5. Significance

This work shifts the paradigm of MR interaction from deterministic, single-cue triggers to probabilistic, multimodal inference. By recognizing that human gestures are inherently multi-dimensional (providing both where we are looking and what we want to do), the authors provide a scalable way to handle complex, cluttered 3D environments. The framework is lightweight enough for real-time use and can be extended to include other modalities like gaze or speech.