World Mouse: Exploring Interactions with a Cross-Reality Cursor

The paper introduces "World Mouse," a cross-reality cursor system that bridges the gap between physical and virtual environments by leveraging surface normals and interpolation to enable precise, natural interactions with both real-world and digital objects in Extended Reality.

The Gist

Imagine you are sitting at your desk, but instead of just a flat computer screen in front of you, the entire room around you is alive with digital objects. There's a floating 3D model of a car hovering over your coffee table, a virtual whiteboard on the wall, and a digital calendar floating near your lamp.

Now, imagine trying to grab that car or write on that whiteboard. If you just wave your hands in the air, your arm gets tired (like holding a "T-Rex" pose), and it's hard to be precise. If you try to touch the air, your hand stops in mid-air because there's nothing there to touch.

Enter the "World Mouse."

Think of the World Mouse as a magic invisible string attached to your regular computer mouse. When you move your mouse on your desk, this string doesn't just move a dot on a flat screen; it reaches out into the 3D world, finds the objects around you, and lets you "touch" them without lifting your hand off the desk.

Here is how it works, broken down into simple ideas:

1. The "Magnetic" Surface (Within-Object Navigation)

Imagine your mouse cursor is a tiny, magnetic ball. When you move it over a physical object (like your real wooden table) or a virtual one (like a floating hologram), the cursor snaps to the surface.

• The Analogy: It's like a roller coaster that is glued to the track. No matter how you twist the track, the coaster stays on it. If you move your mouse, the cursor glides smoothly over the curves of a 3D object, letting you pick it up, rotate it, or click a button on its side with perfect precision.

2. The Invisible Bridge (Between-Object Navigation)

What happens when you want to move your cursor from your real coffee mug to a virtual book floating in the air? There is empty space in between.

• The Analogy: The World Mouse builds an invisible bridge or a trampoline between objects. As you move your mouse, the cursor doesn't just fly through the empty air; it glides along this invisible path, smoothly transitioning from the real world to the virtual world. It feels like the cursor is "walking" across the room, stepping from real furniture to digital furniture without ever falling off.

3. The Shape-Shifter (2D to 3D Transition)

You might be editing a photo on a 2D window on your screen, and then decide to drag that photo onto a 3D wall in your room.

• The Analogy: Think of the cursor as a water droplet. On your screen, it's a flat drop. As it slides off the edge of the window and into the 3D room, it doesn't disappear; it stretches and becomes a 3D sphere that can land on walls, floors, or floating objects. It changes shape to fit the world it enters, making the transition feel natural, not jarring.

Why Do We Need This?

For years, tech companies have told us that the future is all about waving our hands in the air or staring at things with our eyes.

• The Problem: Waving your hand gets tiring (like doing bicep curls all day), and staring at things isn't very precise (you can't easily click a tiny button just by looking at it).
• The Solution: The World Mouse brings back the comfort of the old-school mouse. It lets you sit comfortably in your chair and do complex 3D work with the same precision you use to play a video game or write an email.

What Can You Do With It?

The paper shows some cool things you could do:

• The "Spatial Clipboard": You can "copy" a real-world object (like your coffee mug) by scanning it, and then "paste" a digital version of it onto your virtual desk.
• Smart Home Control: You can hover your cursor over a real light switch in your room, and a digital menu pops up to let you change the color of the lights without touching the physical switch.
• AI Helper: If you are talking to an AI assistant, you can point your cursor exactly at the object you want the AI to fix, so there is no confusion about what you mean.

The Big Picture

The World Mouse is like a universal translator between your old, familiar computer habits and the new, wild world of 3D reality. It proves that we don't have to throw away the tools we know and love (like the mouse) to enter the future. Instead, we can upgrade them to work in a world where the digital and physical blend together seamlessly.

It's not about replacing your hands; it's about giving your hands a superpower: the ability to reach anywhere, anytime, without getting tired.

Technical Summary

Here is a detailed technical summary of the paper "World Mouse: Exploring Interactions with a Cross-Reality Cursor."

1. Problem Statement

As Extended Reality (XR) systems advance in mapping and understanding the physical world, interacting with blended representations of real and virtual objects remains a significant challenge. Current interaction paradigms face specific trade-offs:

• Direct Inputs (Touch/Gestures): While intuitive, they suffer from "gorilla arm" fatigue during long sessions and lack the precision required for fine-grained tasks like 3D modeling or data annotation.
• Indirect Inputs (Raycasting/Gaze): These often lack the precision needed for detailed manipulation and can be ambiguous in complex 3D scenes.
• The Gap: There is a lack of input methods that combine the precision and low fatigue of a traditional desktop mouse with the ability to navigate complex, blended 3D environments (containing both physical and digital objects). Existing cursor-based solutions are largely confined to purely virtual (VR) or purely augmented (AR) contexts, failing to seamlessly bridge the two.

2. Methodology: The World Mouse System

The authors propose World Mouse, a cross-reality cursor system that reinterprets the 2D desktop mouse for 3D spatial computing. The system maps standard 2D mouse deltas to angular displacements of a cursor ray relative to the user's viewpoint, dynamically inferring depth in real-time.

The technical implementation relies on two core mechanisms driven by a blended scene graph (a unified geometric representation of the physical and virtual world):

A. Core Mechanisms

1. Within-Object Interaction (Surface Navigation):

   • When the cursor intersects an object (real or virtual), it adheres to the object's surface.
   • The system uses surface normals to align the cursor's orientation with the local topology of the object.
   • Depth is adjusted dynamically to match the specific point on the object's surface, allowing for precise selection and manipulation similar to a 2D desktop experience.

2. Between-Object Navigation (Void Traversal):

   • To move across empty space between disjoint objects, the system utilizes an "invisible mesh" generated via interpolation.
   • Leveraging semantic segmentation and mesh reconstruction (e.g., via Meta's Scene API or Android XR), the system creates a continuous interaction surface.
   • The cursor traverses this interpolated path based on the spatial relationships of visible targets, creating a fluid "bridge" that prevents the cursor from getting stuck in mid-air or jumping erratically.

3. 2D-to-3D Transition:

   • The system handles the transition between 2D interfaces (e.g., a browser window) and 3D space.
   • As the user moves the cursor from a 2D panel into the 3D environment, the cursor smoothly transitions from planar movement to depth-adaptive movement, allowing actions like dragging a 2D image onto a 3D physical wall.

B. Technical Implementation Details

• Geometric Reconstruction: The system constructs a mesh from the convex hulls of detected objects relative to the camera view. This ensures stability for both static and dynamic objects without requiring expensive recalculation for every viewpoint shift.
• Semantic Awareness: The system integrates semantic segmentation to identify interactable regions (e.g., screens, text, physical objects), enabling the cursor to treat physical surfaces as interactive elements.
• Input Modalities: While designed for a physical mouse, the system supports alternative controllers, such as smartphone touchscreens or smartwatches (via XDTK), acting as high-precision trackpads.

3. Key Contributions

• Conceptual Framework: Introduces a unified interaction model that treats physical and virtual worlds as a single, continuous interaction space, challenging the notion that spatial computing must abandon the mouse.
• Technical Innovation: Develops a depth-adaptive cursor algorithm that combines Voronoi-based interpolation (for empty space) with surface normal alignment (for object interaction) to solve the "depth inference" problem in mixed reality.
• Prototype Demonstrations: The authors validate the system through several use cases:
  • Spatial Clipboard: Selecting, copying, and pasting between real-world scans and virtual environments.
  • 3D Authoring: Using 3D gizmos for precise object anchoring, scaling, and spline editing on real-world meshes.
  • IoT Control: Using the cursor to interact with semantic proxies of physical devices (e.g., toggling lights).
  • AI Grounding: Providing a precise mechanism to "ground" AI prompts to specific real-world objects, reducing ambiguity in human-AI interaction.

4. Results and Evaluation

While the paper primarily presents a conceptual and prototype-driven study (rather than a large-scale user study with statistical metrics), the results are demonstrated through functional prototypes:

• Seamless Transitions: The system successfully demonstrated fluid movement between 2D windows and 3D objects without user disorientation.
• Precision: The use of surface normals allowed for fine-grained manipulation (e.g., vertex snapping to real-world meshes) that is difficult to achieve with mid-air gestures.
• Fatigue Reduction: By leveraging a seated desk-based input (mouse) rather than sustained mid-air gesturing, the system addresses the physical fatigue associated with current XR interaction techniques.
• Cross-Reality Workflow: The prototypes showed that users could perform complex tasks like "dragging" a digital file onto a physical wall or controlling IoT devices via a blended interface, validating the feasibility of the approach.

5. Significance and Implications

• Reimagining the Mouse: The paper argues that the mouse is not obsolete in XR but is a "high-performance spatial input" tool that needs recontextualization. It bridges the gap between the familiarity of 2D workflows and the complexity of 3D spaces.
• AI and Spatial Computing: As AI agents become integrated into XR, the World Mouse offers a critical tool for deictic reference. It allows users to precisely point to specific real-world objects for AI to act upon, solving the ambiguity often found in voice or gaze-based commands.
• Design Continuity: The work suggests that continuity (flowing between 2D/3D and real/virtual) is a powerful design strategy. It enables a "low-fatigue" interaction model that preserves dexterity while avoiding the inaccuracies of raycasting.
• Future Work: The authors position this not as a replacement for embodied interaction (like freehand sketching) but as a complementary trajectory for tasks requiring precision, long-duration focus, and complex cross-reality workflows.

In summary, World Mouse provides a robust technical foundation for extending traditional desktop metaphors into the physical world, offering a viable solution for precise, low-fatigue interaction in the era of blended reality and AI-driven spatial computing.