A Conversational Framework for Human-Robot Collaborative Manipulation with Distributed Generative AI models

Technical Summary: A Conversational Framework for Human-Robot Collaborative Manipulation with Distributed Generative AI Models

Problem Statement
While robotic manipulation has advanced significantly in planning and control, natural-language interaction in real-world collaborative settings remains challenging. User commands are often incomplete, scene descriptions ambiguous, and objects visually similar from the robot's perspective. Furthermore, recent Large Language Models (LLMs) and Vision-Language Models (VLMs) introduce reliability risks; specifically, the tendency of LLMs to "hallucinate" (generate plausible but incorrect content) poses a direct safety threat in robotics, where a misinterpretation can trigger unintended physical actions. Existing approaches often rely on monolithic end-to-end policies or lack explicit safety barriers, making it difficult to debug failures or ensure operator oversight before motion execution.

Methodology
The authors propose a distributed conversational framework implemented as a ROS 2 multi-agent pipeline. The system decouples language understanding, visual grounding, orchestration, and motion execution into separate nodes, allowing for flexible deployment across heterogeneous hardware (e.g., edge devices and workstation GPUs).

• Architecture: The pipeline consists of:
  • Web Dashboard: Accepts free-form natural language commands and displays intermediate intent and grounding overlays (pixel, depth, and robot-frame coordinates).
  • Coordinator Node: Routes requests, combines visual data with depth/calibration, and manages a safety-oriented confirmation loop.
  • LLM Node (Language Understanding): Parses user commands into structured action requests (e.g., pick, place, handover) and extracts parameters. In the default setup, this runs on an NVIDIA Jetson AGX Orin using the ministral-3:8b model.
  • VLM Node (Visual Grounding): Processes RGB-D images to locate objects based on natural language descriptions. In the default setup, this runs on an RTX 6000 workstation using Qwen2.5-VL 32B.
  • Motion Executor: Generates and executes robot trajectories via MoveIt2.
• Safety Mechanism: The system enforces a strict "human-in-the-loop" safety barrier. Language and vision outputs are translated into structured action requests, but no motion is executed without explicit operator confirmation via the dashboard. The dashboard visualizes the interpreted intent and the specific target location (pixel, depth, and robot-frame) before the user approves the action.
• Grounding Pipeline: The VLM returns image-space targets (pixel coordinates). These are fused with depth data using a robust local estimator (median of a 5×5 neighborhood) to recover 3D camera-frame points. These are then transformed into the robot's base frame using offline ArUco-marker calibration and fixed offsets.
• Deployment: Both AI models run locally via Ollama to ensure privacy and predictable latency, avoiding cloud dependencies. The system supports zero-shot operation without fine-tuning on specific object categories, relying on the general pre-trained knowledge of the models.

Key Contributions

1. Distributed ROS 2 Multi-Agent Pipeline: A modular architecture that separates language, vision, orchestration, and motion execution, enabling flexible single-machine or multi-device deployments and easy adaptation to available hardware.
2. Web Dashboard with Explicit Confirmation: A user interface that exposes intermediate intent and grounding overlays, enforcing an explicit operator confirmation gate before any robot motion is triggered.
3. Open-Vocabulary Grounding-to-Robot-Frame Pipeline: A method for generating metric robot-frame targets for pick, place, and handover primitives from pixel/depth data without requiring task-specific model fine-tuning.
4. Evaluation Under Ambiguity: An experimental benchmark evaluating end-to-end task reliability and latency across increasing scene ambiguity (single, multiple, and overlapped objects) while comparing alternative LLM/VLM configurations within the same modular architecture.

Results
The framework was evaluated on a Franka Emika FR3 robot with a Realsense D435i camera. The benchmark involved 180 runs across pick, place, and handover tasks under single, multiple, and overlapped object scenarios.

• Best Configuration: The default distributed deployment (ministral-3:8b on Jetson + Qwen2.5-VL 32B on RTX 6000) achieved 100% success (5/5) across all tasks and ambiguity levels. It also demonstrated the lowest latencies (5.47s for LLM, 9.57s for VLM).
• Hardware Impact: Running the VLM on a shared workstation with the LLM (e.g., using LLaMA 3.1 8B on the same RTX 6000) increased VLM latency significantly (to 42.83s) and reduced success rates in complex scenarios (e.g., handover dropped to 4/5).
• Model Sensitivity: Configurations using smaller or less capable models (e.g., DeepSeek-R1 7B + LLaVA 7B) resulted in 0% success, failing to complete trials. Larger VLMs (e.g., Llama 3.2-Vision 90B) did not necessarily improve success rates and introduced higher memory and latency costs.
• Memory Constraints: Runtime RAM was dominated by the VLM (e.g., ~127 GB for the 32B model), justifying the distributed approach where the vision model runs on a high-memory workstation while the language planner runs on an edge device (43 GB observed on Jetson).
• Failure Modes: Qualitative analysis showed that even with high-confidence VLM outputs, grounding errors can occur (e.g., selecting the wrong object in an overlapped scene), validating the necessity of the explicit confirmation step.

Significance and Claims
The paper claims that its primary contribution is not a novel end-to-end policy, but rather a system integration framework that prioritizes safety, interpretability, and flexibility. By separating "what" (language), "where" (vision), and "how" (control) into inspectable, typed intermediate representations, the system allows for:

• Proactive Safety: Preventing unintended physical actions through explicit human confirmation of grounded targets.
• Debuggability: The ability to isolate failures to specific modules (language vs. vision) and swap models without redesigning the control stack.
• Practical Deployment: A viable route for deploying generative AI in real-world HRI by balancing computational requirements (edge vs. workstation) and maintaining privacy through local inference.

The authors acknowledge limitations, noting that the system struggles with highly complex compositional reasoning (e.g., conditional logic based on object values) and lacks conversational memory for context-dependent follow-up commands. However, they position the framework as a practical step toward safer, interpretable language-based manipulation in human-robot collaboration.