Technical Summary: Multi-Agent Robotic Control with Onboard Vision-Language Models

Problem Statement

Vision Language Models (VLMs) and Vision Language Action (VLA) models have demonstrated potential in robotic control but face significant barriers to practical deployment in industrial settings. These limitations include:

• Compute and Connectivity Dependencies: Many solutions require substantial external computing power and network connectivity, increasing operational costs and latency.
• Explainability and Safety: A lack of interpretability in model decisions raises safety and ethical concerns.
• Generalization Issues: Existing methods, including those using interpretability techniques or constraint functions, often struggle to generalize to out-of-distribution data.
• Context Retention: Deploying a single VLM to handle diverse, long-horizon tasks (e.g., safety inspection, maintenance, search, and human interaction) risks context window overflow, where crucial information is lost as the task variety grows.

Methodology

The authors propose a Multi-Agent System (MAS) architecture designed to run entirely on onboard hardware, eliminating dependence on cloud or edge servers. The system controls a multipurpose autonomous mobile manipulator (RB-KAIROS+) within a simulated industrial warehouse environment.

System Architecture

The MAS is built using the RAI framework and integrates with ROS 2, MoveIt 2, and Nav2. The architecture comprises several specialized agents:

1. Megamind (Orchestration Agent): A supervisory agent addressing context retention issues in compact models. It operates via a two-state self-feedback loop:
   • Planning State: Selects the next task based on the robot's state and task queue, delegating to sub-agents.
   • Analysis State: Evaluates sub-agent outputs. If recovery is needed, it generates corrective steps and appends them to the queue before returning to planning. This externalizes task progress and recovery logic.
2. Inspection Agent: Monitors the front-facing camera for anomalies (misplaced boxes, obstructions, spills). It uses a two-stage workflow: generating a natural language description and flagging issues, then categorizing them (e.g., "box" for graspable items, "trash," "other," or "nothing"). It outputs structured JSON with issue types and 3D poses.
3. Safety Agent: Performs continuous regulatory compliance monitoring. It employs a three-stage pipeline:
   • Vision: The VLM identifies potential hazards.
   • Retrieval: The agent queries a vector database (FAISS) containing OSHA 29 CFR 1910 regulations, filtered for warehouse relevance. It uses Qwen3-Embedding-0.6B for indexing and Qwen3-Reranker-0.6B for re-ranking.
   • Reasoning: The VLM cross-references the visual scene with retrieved regulatory excerpts to determine violations, outputting regulation IDs, severity levels, and rationales.
4. MoveIt and Nav2 Agents: Provide low-level control for manipulation and navigation, executing commands from Megamind.
5. Human-Robot Interface (HRI): A Qt-based graphical interface allowing operators to issue natural language commands and view live video/status updates.

Model Strategy

• Base Model: The system utilizes a compact 3B-parameter VLM (LFM2-VL-3B) for all visual tasks (Inspection, Safety, and Megamind).
• Fine-Tuning via Knowledge Distillation: To address the base model's struggle with package quality verification (Task T4), the authors fine-tuned the model. They used Qwen3-256B-A22B as a teacher model to generate synthetic training data (natural language descriptions) from annotated simulation images. The 3B model was fine-tuned for one epoch on this dataset.
• Hardware: The entire stack runs on an AMD Ryzen™ AI mini PC (128 GB unified RAM, 96 GB allocated to GPU) in a Hardware-in-the-Loop (HIL) simulation.

Key Contributions

1. Onboard MAS Architecture: A fully functional multi-agent system running on local hardware, demonstrating that compact models (3-20B parameters) can control complex mobile manipulators without external compute.
2. Context Management via Orchestration: The "Megamind" agent successfully mitigates context retention limitations of small models through explicit task delegation and a self-feedback recovery loop.
3. Specialized Safety Pipeline: A novel integration of VLMs with regulatory databases (OSHA) using retrieval-augmented generation (RAG) techniques (embedding, reranking, reasoning) to provide interpretable safety compliance.
4. Open Source Release: The simulation environment, including the configurable warehouse, robot platform, and software stack, is released under the Apache 2.0 license to facilitate research in agentic systems for mobile manipulators.
5. Performance Optimization: Demonstrated that fine-tuning a compact model via knowledge distillation significantly improves specific task accuracy (e.g., package inspection) while maintaining low inference costs.

Results

The system was validated in a Hardware-in-the-Loop simulation covering five task categories:

• T1: Safety inspection.
• T2: Warehouse maintenance.
• T3: Warehouse search.
• T4: Package quality verification.
• T5: Responding to human requests.

Quantitative Improvements:
Following fine-tuning for package inspection (Task T4), the model achieved:

• Box condition annotation accuracy: Increased from 76.7% to 91.5%.
• F1-score: Improved from 0.755 to 0.915.
• True Negatives: Increased from 54% to 95.5%.
• False Positives: Decreased from 46% to 4.5%.

The system successfully demonstrated real-time response and operation in dynamic industrial environments, including preempting user tasks to handle high-priority anomalies and executing multi-step workflows (e.g., inspecting, relocating, or disposing of objects).

Significance and Claims

The paper claims that a fully onboard MAS architecture is a viable and cost-efficient alternative to cloud-dependent deployments. By utilizing compact, locally hosted VLMs, the system addresses the high operational costs (electricity, hardware) and latency issues associated with external compute platforms.

The authors position this work as a step toward flexible robotics for small- and medium-sized enterprises (SMEs), where cost-effective, interpretable, and safe autonomous systems are critical. The successful validation across all five task categories justifies the transition to real-robot trials and opens research avenues for agentic systems in industrial settings. The work emphasizes that mature stacks (ROS 2, MoveIt 2) combined with modular agent design can overcome the challenges of real-time reactivity and certifiable control that generalist policies often struggle to meet.