LEGS-POMDP: Language and Gesture-Guided Object Search in Partially Observable Environments

The paper introduces LEGS-POMDP, a modular framework that integrates language, gesture, and visual observations within a Partially Observable Markov Decision Process to effectively handle uncertainty in object identity and location for robust open-world robot search.

The Gist

Imagine you are in a giant, foggy warehouse filled with hundreds of identical-looking boxes. Your friend, standing far away, shouts, "Get me the red one!" and points vaguely in your general direction.

Here's the problem:

1. The Fog: You can't see everything clearly (Partial Observability).
2. The Vague Voice: "The red one" could be any of the 50 red boxes.
3. The Wobbly Point: Your friend's hand is shaking, or they might be pointing at a red box that isn't the red box they want.

If you just guess, you might grab the wrong one. If you just ask for clarification, it takes too long. You need a way to combine the voice, the hand gesture, and your own eyes to figure out exactly which box to grab, even when you aren't 100% sure.

This is exactly what the LEGS-POMDP paper is about. It's a new "brain" for robots that helps them find objects in messy, uncertain worlds by listening to human language and watching human gestures simultaneously.

Here is a breakdown of how it works, using simple analogies:

1. The Robot's "Gut Feeling" (The POMDP)

Most robots today are like students who memorize answers. If they see a specific pattern, they do a specific action. But in a messy world, patterns change.

The LEGS-POMDP robot is more like a detective. Instead of knowing the answer immediately, it maintains a "belief map."

• Imagine a map of the warehouse where every box has a little percentage written on it (e.g., "Box A: 10% chance this is the one," "Box B: 80% chance").
• This is called a POMDP (Partially Observable Markov Decision Process). It's a mathematical way of saying, "I don't know for sure, but here is my best guess based on what I've seen so far."

2. The Three Clues (Multimodal Fusion)

The robot gets clues from three sources, and it treats them like pieces of a puzzle:

• The Voice (Language): The human says, "The cup with the blue stripe."
• The Hand (Gesture): The human points toward a corner.
• The Eyes (Vision): The robot sees a bunch of objects.

The Magic Trick:
Older robots might try to use just the voice or just the hand. If the voice is vague ("the cup") and the hand is shaky, the robot gets confused.
LEGS-POMDP combines them. It's like having a team of three experts in a room:

• The Voice Expert says, "I think it's the blue-striped cup, but I'm only 60% sure."
• The Hand Expert says, "I'm pointing at the left side, but my hand is shaking, so it could be anywhere in this cone shape. I'm 70% sure it's in that cone."
• The Eye Expert says, "I see a blue-striped cup in that cone."

The robot's brain multiplies these probabilities together. Suddenly, the "blue-striped cup in the left cone" goes from a 10% guess to a 90% certainty. This is called Multimodal Fusion.

3. The "Fan" and the "Cone"

The paper introduces two clever ways to model human uncertainty:

• The Vision Fan: The robot knows its camera is like a flashlight. It sees things clearly in the center but gets blurry at the edges and far away. It models this as a "fan" shape where the middle is bright (high confidence) and the edges are dim (low confidence).
• The Gesture Cone: When a human points, they don't point with laser precision. They use their whole arm, their eyes, and their shoulder. The robot models this as a cone (like an ice cream cone) coming out of the human's wrist. The tip of the cone is the most likely spot, but the wide opening acknowledges that the human might be a bit off.

4. The Decision Maker (The Solver)

Once the robot updates its "belief map" with these new clues, it has to decide what to do next. Should it walk forward? Should it look closer? Should it grab the object?

The paper uses a smart algorithm called PO-UCT. Think of this as a super-fast simulator.

• Before the robot actually moves, it runs thousands of "what-if" scenarios in its head in a split second.
• Scenario A: "If I walk left, I might see the target."
• Scenario B: "If I look closer, I might confirm the blue stripe."
• It picks the path that gives it the highest chance of success with the least amount of wasted energy.

5. The Results: Why It Matters

The researchers tested this on a real robot (a four-legged dog-like robot called a "Spot") and in simulations.

• The Result: When the robot used both voice and gestures, it found the object 89% of the time.
• The Comparison: If it only used voice, or only used gestures, it failed much more often.
• The "Wrong" Clues: Even when the human gave wrong information (e.g., pointing at a red cup but saying "blue cup"), the robot's math was smart enough to realize the clues conflicted and didn't just blindly follow the wrong one. It stayed calm and kept searching.

The Big Picture

In the past, robots were like parrots: they repeated what they were told or followed strict rules. If the rules didn't fit the messy real world, they crashed.

LEGS-POMDP makes the robot more like a human partner. It understands that humans are messy, that language is vague, and that gestures are imprecise. By mathematically combining these imperfect clues, it can navigate a chaotic world, figure out what you actually want, and get it for you without needing a perfect instruction manual.

It's the difference between a robot that says, "I don't understand, please repeat," and a robot that says, "You pointed left and said 'red cup,' so I'm pretty sure you mean that red cup over there. Let me go get it."

Technical Summary

Here is a detailed technical summary of the paper "LEGS-POMDP: Language and Gesture-Guided Object Search in Partially Observable Environments."

1. Problem Statement

The paper addresses the challenge of human-instructed object search in unstructured, open-world environments. Robots must locate a target object based on ambiguous human instructions that often involve:

• Underspecified Language: Natural language can be vague (e.g., "the cup" when multiple cups exist).
• Imprecise Gestures: Pointing gestures may indicate a region containing multiple candidates or suffer from sensor noise.
• Partial Observability: The robot cannot see the entire environment at once, and the target's location is hidden.

Existing approaches face limitations:

• Foundation Model-based methods excel at multimodal grounding but lack principled mechanisms for modeling uncertainty in long-horizon tasks and offer limited interpretability.
• Traditional POMDPs provide a framework for planning under uncertainty but are often restricted to tabletop settings, rely solely on language, or make rigid environmental assumptions.

The core problem is how to jointly reason over uncertainty in human intent (which object is the target?) and uncertainty in the environment (where is the object located?) using a unified framework that integrates language, gesture, and vision.

2. Methodology: LEGS-POMDP

The authors propose LEGS-POMDP (Language and Gesture-Guided Object Search in Partially Observable Environments), a modular framework that formulates the task as a Partially Observable Markov Decision Process (POMDP).

A. POMDP Formulation

The system is defined by the tuple $(S, A, T, O, Z, R, \gamma)$:

• State Space ($S$): Defined as $s = (s_r, s_o)$, where $s_r$ is the robot pose and $s_o$ is the latent target location. The framework uses an object-independent representation (Target vs. Distractor) rather than specific categories to focus on uncertainty reasoning.
• Action Space ($A$): Includes movement primitives ($a_{move}$), observation gathering ($a_{look}$), and termination ($a_{find}$).
• Observation Space ($O$): Multimodal signals $(o_v, o_g, o_l)$ from Vision, Gesture, and Language.
• Observation Model ($Z$): The core innovation. It models the likelihood of observations given a state by fusing three modalities in log-space:
  $$\log Z(o|s) = w_v \log P_v(o_v|s) + w_g \log P_g(o_g|s) + w_l \log P_l(o_l|s)$$
  This allows for weighted integration of probabilistic likelihoods from different sources.
• Reward ($R$): Positive reward for correct identification, negative costs for movement and observation steps to encourage efficiency.

B. Multimodal Observation Models

1. Visual ($P_v$): Modeled as a decaying fan-shaped sensor. Detection likelihood decreases based on angular deviation from the camera axis and distance from the robot.
2. Language ($P_l$): Uses a similarity function $\kappa$ between the instruction and the hypothesized target state. This is converted to a likelihood using false-positive ($\epsilon^-$) and true-positive ($\epsilon^+$) rates to handle graded confidence rather than binary matching.
3. Gesture ($P_g$): Instead of a single vector, the system models pointing as a probabilistic cone. The cone's axis is the mean of multiple anatomical vectors (eye-to-wrist, shoulder-to-wrist, etc.), and the opening angle captures the spread of these vectors. This accounts for human variability and noise.

C. Planning Solver

The system employs Partially Observable UCT (PO-UCT), a Monte Carlo Tree Search algorithm. It balances exploration and exploitation by simulating trajectories from the current belief state, allowing the robot to plan sequences of actions that maximize the probability of finding the target while minimizing steps.

3. Key Contributions

1. Dual-Uncertainty POMDP Formulation: The first framework to explicitly model two distinct sources of partial observability simultaneously: uncertainty over target identity (human intent) and spatial location (environment state).
2. Modular Multimodal Observation Model: A principled Bayesian approach that integrates language, gesture, and vision as probabilistic likelihoods. This design allows for the flexible replacement of perception modules (e.g., swapping detectors) while maintaining interpretability and rigorous belief updates.
3. Comprehensive Evaluation: Extensive validation across modular benchmarks, simulated grid worlds of varying complexity, and real-world deployment on a Boston Dynamics Spot quadruped mobile manipulator.

4. Results

The evaluation demonstrates the superiority of the multimodal approach over unimodal baselines.

• Modular Grounding:
  • Gesture: A probabilistic cone representation (averaging multiple skeletal vectors) outperformed single-vector baselines, achieving 89.0% coverage and 14.4° angular error compared to 86.5% and 17.0° for the best single vector.
  • Visual Grounding: A Set-of-Marks (SoM) pipeline (SAM2 segmentation + GPT-4o reasoning) achieved higher grounding accuracy (91.4%) than detector-based methods (62.4%), particularly in resolving ambiguous spatial or attribute queries, though it incurred higher inference time.
• System Performance (Simulation):
  • Solver Comparison: PO-UCT achieved a 96% success rate with histogram belief representation, significantly outperforming Greedy (63%) and Heuristic (68%) baselines.
  • Modality Fusion: Multimodal input (Gesture + Language) achieved an 88.8% success rate with the fewest steps (76.8) and fastest time (16.7s). In contrast, "conflicted" multimodal inputs (contradictory cues) caused success rates to drop to near zero (2.4%), highlighting the system's sensitivity to noise.
  • Complexity: As environment size and ambiguity increased, single-modality performance degraded sharply, while multimodal fusion maintained high success rates.
• Real-World Deployment:
  • Tested on a quadruped robot in a 10x10 grid.
  • Entropy Reduction: Multimodal input reduced belief entropy by 60.8%, significantly outperforming gesture-only (40.6%) and language-only (34.2%) inputs, proving faster disambiguation in physical settings.

5. Significance

• Robustness in Open Worlds: LEGS-POMDP demonstrates that robots can effectively operate in large, unstructured environments where instructions are inherently ambiguous, a scenario where end-to-end learning often fails due to data scarcity and lack of explicit uncertainty modeling.
• Interpretability: By using a modular, probabilistic framework, the system provides explainable decision-making. Users can understand why the robot chose a path (e.g., "I am moving here because the gesture cone and language description both increase the probability of the target being there").
• Complementarity of Modalities: The work empirically validates that language and gesture are complementary; gestures disambiguate vague language, and language clarifies imprecise gestures.
• Scalability: The modular design allows the system to adapt to new sensors or perception models without retraining the entire planning pipeline, facilitating easier transfer from simulation to real-world hardware.

In conclusion, LEGS-POMDP offers a rigorous, interpretable, and highly effective solution for human-robot collaboration in search tasks, bridging the gap between the flexibility of foundation models and the reliability of probabilistic planning.