Uncertainty Mitigation and Intent Inference: A Dual-Mode Human-Machine Joint Planning System

This paper proposes a dual-mode human-robot joint planning system that combines an LLM-assisted active elicitation mechanism with real-time intent inference to effectively mitigate task-relevant knowledge gaps and latent human intent, significantly reducing interaction costs and execution time in open-world environments.

The Gist

Imagine you are a rescue worker trying to navigate a chaotic, smoke-filled building with a robot drone. In the old days, the robot was like a very obedient but slightly confused dog: you had to give it very specific commands ("Go to the red box, then turn left"), and if you didn't, it would either get stuck or guess wrong. It couldn't really "think" about what you meant or ask for help.

This paper introduces a new kind of robot teammate that acts more like a smart, proactive human partner. It solves two big problems that happen when humans and robots work together in the real world: not knowing what to do and not knowing what the human is thinking.

Here is how their system works, broken down into two main "modes" using simple analogies:

Mode 1: The "Clarifying Detective" (Uncertainty Mitigation)

The Problem: You tell the robot, "Go get the medicine and bring it to the injured person." But there are three boxes in the room, and you didn't say which one has the medicine. Also, there's a net and some smoke blocking the path, and you aren't sure if the robot can fly through them.

The Old Way:

• The "Guessing" Robot: Just picks a box (maybe the wrong one) and flies into the smoke (crashing).
• The "Over-Questioning" Robot: Asks you about every single thing ("Is the blue box safe? Is the red box safe? Is the net safe?"). This wastes time and annoys you.

The New Way (This Paper's Solution):
The robot acts like a smart detective.

1. It thinks first: It uses a "brain" (an AI) to guess which box is most likely the medicine box.
2. It asks the right question: Instead of asking about everything, it calculates the most efficient path. It realizes, "If I ask about the fire, I might not need to ask about the net later." So, it only asks you: "Is the fire blocking the path?"
3. The Result: It solves the puzzle with half the questions compared to other methods. It saves time and energy by only asking for the specific information it needs to make a safe plan.

Mode 2: The "Mind-Reading Partner" (Intent Inference)

The Problem: You and the robot are working together to save a person. You start walking toward the injured person, but you haven't said a word. The robot needs to decide: Should I follow you? Should I go get the bandages? Should I clear the path?

The Old Way:

• The "Clueless" Robot: It just waits for you to speak. If you don't speak, it stands still or does something random. It doesn't realize you are already moving toward the patient.

The New Way (This Paper's Solution):
The robot acts like a mind-reading partner who watches your body language.

1. It watches your moves: It tracks where you are walking and which way you are facing.
2. It guesses your goal: It calculates, "You are walking toward the injured person. You probably want me to help there."
3. It acts without being told:
   • If the task is cooperative (both need to be there), the robot rushes to help you immediately.
   • If the task is independent (you are doing one thing, it can do another), it doesn't get in your way. Instead, it grabs the bandages from the other side of the room so you don't have to.
4. The Result: You don't have to stop and give orders. The robot just knows what to do, making the whole team move faster and smoother.

The Real-World Test

The researchers didn't just write this on paper; they built a real drone and tested it in a simulated building and a real room with obstacles like nets and smoke.

• The "Detective" Mode: The robot asked 52% fewer questions than the "over-questioning" robot but still got the job done 100% of the time.
• The "Mind-Reader" Mode: The team finished their mission 25% faster because the robot stopped waiting for orders and started helping immediately.

The Big Picture

Think of this system as upgrading a robot from a remote-controlled car (which needs constant, perfect instructions) to a co-pilot (which can read the map, ask smart questions when confused, and anticipate your next move).

By combining smart questioning (to fix confusion) and body-language reading (to guess intent), this system creates a robot that feels less like a machine and more like a true teammate you can trust in a crisis.

Technical Summary

Here is a detailed technical summary of the paper "Uncertainty Mitigation and Intent Inference: A Dual-Mode Human-Machine Joint Planning System."

1. Problem Statement

Effective human-robot collaboration (HRC) in open-world environments is hindered by two primary sources of uncertainty:

1. Task-Relevant Knowledge Gaps: Ambiguities in natural language instructions (e.g., "the blue box") and partial observability of environmental states (e.g., whether an obstacle like smoke or a net is passable).
2. Latent Human Intent: The difficulty in predicting a human's unobserved goals and evolving priorities during real-time cooperation without explicit communication.

Existing approaches often treat humans as passive supervisors (providing corrections) or rely solely on implicit intent inference (observation-only). These methods fail to actively model knowledge gaps, leading to redundant communication, inefficient planning, or suboptimal coordination. There is a critical need for autonomous agents that can proactively reason about uncertainty, query humans strategically, and adapt to latent intent in real-time.

2. Methodology

The authors propose a unified Human-Robot Joint Planning System featuring a core planning engine that operates in two complementary modes, integrated with a Vision-Language Model (VLM) for perception and a voice interface for interaction.

A. System Architecture

• Perception: Utilizes a VLM-based pipeline (Grounded-SAM + 3D Gaussian Splatting) to create a 3D semantic map from RGB-D data, allowing for natural language object retrieval and robust spatial reasoning.
• Core Planning Engine: Dynamically switches between two modes based on task type:
  1. Uncertainty-Mitigation Joint Planning (Human-instructed tasks).
  2. Real-Time Intent-Aware Collaboration (Human-participated tasks).

B. Mode 1: Uncertainty-Mitigation Joint Planning

Designed for scenarios where the robot receives a natural language instruction with ambiguous targets or unknown obstacle traversability.

• Target Ambiguity Resolution: The system uses an LLM to ground language descriptions to detected objects. If multiple candidates exist, it employs an LLM-assisted active elicitation mechanism to refine the target belief through iterative queries.
• Obstacle Traversability & Query Optimization:
  • The system formulates a hypothesis-augmented A* search where nodes include sets of assumed passable obstacles.
  • It constructs a decision tree mapping environment configurations to candidate paths.
  • Dynamic Programming (DP) is used to compute an optimal querying policy. The agent calculates the expected cost of interaction ($\lambda_1$) and verification ($\lambda_2$) to determine the minimal set of questions required to identify a definitive safe path, avoiding redundant queries.

C. Mode 2: Real-Time Intent-Aware Collaboration

Designed for scenarios where a human and robot work concurrently without explicit communication.

• Probabilistic Intent Belief: The robot maintains a real-time belief distribution over the human's latent task target ($g_t$).
• Evidence Accumulation: The belief is updated using two geometric cues:
  1. Distance-to-task: Proximity of the human to potential targets.
  2. Directional Alignment: Cosine similarity between the human's velocity vector and the direction to the task.
  • These cues are fused via exponential smoothing to handle noise.
• Coordination-Aware Task Selection:
  • The robot distinguishes between Independent Tasks (completable by one agent) and Cooperative Tasks (requiring both).
  • Strategy: If the human targets a cooperative task, the robot prioritizes convergence to minimize synchronization delay. If the human targets an independent task, the robot selects a different available task to avoid redundancy.
  • Stability Gating: The robot only switches targets when confidence exceeds a threshold, preventing oscillatory behavior.

3. Key Contributions

1. Dual-Mode Planning Framework: A unified system that seamlessly transitions between explicit uncertainty resolution (via strategic querying) and implicit intent inference (via geometric cues).
2. Optimal Querying Policy: A novel approach combining hypothesis-augmented A* search with dynamic programming to minimize interaction costs while guaranteeing the discovery of a safe path.
3. Lightweight Intent Inference: A computationally efficient, online probabilistic belief update mechanism that adapts to human motion and task status without requiring retraining or complex deep learning models for intent prediction.
4. End-to-End Real-World Deployment: Integration of the planning engine with a VLM-based 3D perception pipeline, voice interface, and low-level drone controllers, validated in both Gazebo simulations and physical UAV deployments.

4. Experimental Results

The system was evaluated in Gazebo simulations and real-world UAV deployments (12m x 6m indoor space) against baselines (No-query/Passive and Exhaustive-query/Non-cooperative).

A. Uncertainty-Mitigation Performance:

• Success Rate: Achieved 100% success in both simple and complex scenarios, compared to 71% (simple) and 40% (complex) for the passive baseline.
• Efficiency: Reduced interaction queries by 51.9% and total token usage by 30.3% compared to exhaustive querying, while maintaining 100% success.
• Path Quality: Generated significantly shorter paths than the conservative passive baseline.

B. Intent-Aware Collaboration Performance:

• Simulation: Reduced total task execution time by 23.0% and total travel distance by 10.7% compared to non-cooperative baselines.
• Real-World Deployment:
  • Reduced execution time by 25.4%.
  • Reduced total travel distance by 17.9%.
  • Reduced human travel distance by 18.3%, indicating the robot effectively assumed more workload.
• Intent Recognition: Achieved an average true-target probability of 74.3% and a top-1 accuracy of 95.0% in real-world tests, significantly outperforming uniform baselines.

5. Significance

This work bridges the gap between passive automation and proactive teamwork. By treating uncertainty as a first-class citizen and employing a dual-mode strategy, the system enables robots to:

• Act as Human-Like Teammates: They can ask the right questions at the right time rather than waiting for commands or guessing blindly.
• Operate Efficiently in Open Worlds: The system handles semantic ambiguity and dynamic environmental constraints without requiring perfect prior knowledge.
• Scale to Real-World Applications: The successful deployment on UAVs demonstrates the feasibility of using LLMs and probabilistic planning for complex, safety-critical tasks like search-and-rescue, where communication bandwidth may be limited and environmental conditions are uncertain.

The proposed framework sets a new standard for HRC by balancing the trade-off between communication cost and planning optimality, paving the way for more autonomous and adaptive robotic assistants.