Vision Language Model-based Testing of Industrial Autonomous Mobile Robots

This paper presents RVSG, a Vision Language Model-based testing framework developed with PAL Robotics that automatically generates diverse, requirement-violating human interaction scenarios in simulation to safely and effectively evaluate the robustness of industrial Autonomous Mobile Robots against unpredictable behaviors.

The Gist

Imagine you are training a very smart, self-driving delivery robot to work in a busy warehouse. You want to make sure it never bumps into people, doesn't get dizzy from turning too sharply, and gets its packages delivered on time.

The problem is, people are unpredictable. Sometimes they walk fast, sometimes they stop to chat, and sometimes they drop a box right in the robot's path. If you only test the robot in a perfect, empty warehouse, it will pass every test but might fail miserably in the real world.

This paper introduces a new way to test these robots using a "Digital Imagination Machine" (called a Vision Language Model, or VLM) to create tricky, realistic test scenarios without needing real humans to get in the way.

Here is the breakdown of their method, RVSG, using simple analogies:

1. The Problem: The "Real World" is Too Dangerous and Expensive

Testing a robot in a real warehouse with real humans is like trying to teach a baby to ride a bike by throwing it into a busy highway.

• It's dangerous: If the robot fails, it could hurt a person or break the robot.
• It's expensive: You have to pay people to stand around and act weird on command.
• It's boring: It's hard to get a human to act "unpredictably" on purpose.

2. The Solution: The "Digital Director" (RVSG)

Instead of using real people, the researchers built a system called RVSG that acts like a movie director for a computer simulation.

• The Script (The Requirements): The robot has rules: "Don't hit people," "Don't shake too much," and "Be fast."
• The Set (The Simulation): They use a digital twin of a warehouse (like a video game world) where the robot lives.
• The Director (The VLM): This is the star of the show. It's an AI that can "see" pictures of the warehouse and "read" the rules. It acts like a creative writer who knows how humans behave in real life.

3. How the "Director" Works (The Three-Step Process)

Step 1: Scouting the Location (Environment Preprocessing)
The system takes a picture of the digital warehouse. The AI looks at it and says, "Okay, this is a shelf area, this is a walking path, and this is a narrow aisle." It creates a map of where things are.

Step 2: Writing the Scene (Test Scenario Generation)
This is where the magic happens. The AI is given a rule to break (e.g., "Make the robot crash").

• Old Way: Randomly throw a person in front of the robot. (Like throwing darts blindfolded).
• RVSG Way: The AI thinks, "If I want the robot to crash, I need a human to be carrying a heavy box, walking slowly, and suddenly stop right in front of the robot's nose while the robot is trying to turn a corner."
• The AI writes a detailed script for a digital human actor to follow. It decides exactly where the human walks, what they carry, and when they stop.

Step 3: The Rehearsal and Feedback Loop
The simulation runs. The robot tries to navigate while the "digital human" acts out the script.

• If the robot succeeds: The AI says, "That wasn't hard enough. Let's try again, but this time, have the human run faster."
• If the robot fails (crashes or stops): The AI says, "Perfect! We found a weakness."
• The system remembers these "bad" scenarios so it can create even more challenging ones later. It's like a coach reviewing game tape to find new ways to trick the opponent.

4. Why This is a Game-Changer

The researchers tested this on a real robot from PAL Robotics (a company that makes industrial robots). They compared their "Smart Director" (RVSG) against a "Random Director" (RVSGR) that just guessed where to put people.

• The Result: The Smart Director was much better at finding the robot's weak spots. It created scenarios that made the robot wobble, stop unexpectedly, or almost crash, revealing problems that the Random Director missed.
• The Analogy: Imagine testing a car.
  • Random Testing: Driving on a straight road and hoping a squirrel jumps out.
  • RVSG Testing: A smart AI that knows exactly where the potholes are, predicts when a child might chase a ball into the street, and sets up a perfect "storm" of traffic to see if the car's brakes hold up.

5. The Big Takeaway

This paper proves that we can use AI to test AI. By using a Vision Language Model to understand the environment and human behavior, we can generate thousands of realistic, tricky test scenarios in a computer.

This means:

1. Safer Robots: We can find bugs and dangerous behaviors before the robot ever touches a real human.
2. Cheaper Testing: No need to rent a warehouse or hire actors.
3. Smarter Systems: The robot learns to handle the unexpected because it has "practiced" against the most creative digital humans imaginable.

In short, they built a virtual stress-test gym where robots can get punched, tripped, and confused by digital humans, so they are ready for the real world.

Technical Summary

Here is a detailed technical summary of the paper "Vision Language Model-Based Testing of Industrial Autonomous Mobile Robots" by Jiahui Wu et al.

1. Problem Statement

Autonomous Mobile Robots (AMRs), such as those deployed by PAL Robotics in warehouses and offices, must operate safely alongside humans. However, human behavior is unpredictable, and AMRs may not be trained to handle all possible unknown interactions.

• The Challenge: Ensuring AMR safety requires testing under a wide range of human-robot interactions.
• Limitations of Current Methods:
  • Real-world testing: Is costly, impractical, and hazardous (risk of injury or damage).
  • Simulation-based testing: While safer and cheaper, generating realistic and diverse human behaviors that violate safety requirements is difficult. Traditional methods (e.g., search algorithms) often fail to capture the contextual nuance of human behavior in specific environments (e.g., a warehouse vs. an office).

2. Methodology: RVSG

The authors propose RVSG (Requirement-driven Vision Language Model-based Scenario Generation), a novel framework that leverages Vision Language Models (VLMs) to generate test scenarios that intentionally violate functional and safety (F&S) requirements.

Core Workflow

RVSG operates in two main stages:

1. Environment Preprocessing:
   • Captures top-down images of the simulation world.
   • Labels the map (identifying areas like shelves, aisles) to create a grid map with waypoints.
   • Uses the VLM to analyze the labeled map and grid images via multi-turn conversation, generating a natural language Environment Description (JSON format) that captures layout and navigation context.
2. Test Scenario Generation:
   • Prompt Engineering: A Prompt Generator uses pre-designed templates to construct inputs for the VLM based on specific F&S requirements, robot navigation routes, and the environment description.
   • Multi-turn Reasoning: The VLM engages in a multi-turn dialogue to generate:
     • Scenario Description: Realistic textual descriptions of human activities.
     • High-level Tasks: Specific human workflows (e.g., "sorting packages").
     • Low-level Paths: Valid waypoint sequences for human agents.
     • Configuration: A JSON output defining human agent behavior.
   • Feedback & Memory Loop:
     • Execution: The generated human configuration is simulated in Gazebo using HuNavSim (a human navigation simulator based on social force models and behavior trees) alongside the AMR (TIAGo OMNI Base).
     • Feedback: Simulation results (e.g., collision metrics, path deviations) are fed back to the VLM to refine the human configuration (iterative optimization).
     • Memory: Historical scenario data and feedback are stored to guide the VLM in generating more diverse scenarios in subsequent iterations, avoiding repetition.

Key Components

• VLM: GPT-4.1 is used for its strong commonsense reasoning and ability to interpret visual and textual inputs simultaneously.
• Simulator: Gazebo Classic 11 with ROS 2, using the TIAGo OMNI Base robot model.
• Human Agent: HuNavSim, which simulates realistic crowd flow and individual avoidance behaviors.

3. Key Contributions

1. Novel VLM-based Testing Method: A requirement-driven approach (RVSG) that uses VLMs to generate diverse, context-aware human behaviors that challenge AMR navigation.
2. Adaptive Prompting Framework: A set of prompt templates that allow the system to understand the environment, generate scenarios, and dynamically incorporate simulation feedback and historical memory to improve diversity and effectiveness.
3. Comprehensive Evaluation: A rigorous evaluation on the latest PAL Robotics AMR in a warehouse simulation, covering multiple requirements (Collision Avoidance, Stability, Efficiency) and diverse navigation routes.

4. Experimental Results

The study compared RVSG against a baseline variant (RVSGR) which lacks the feedback and memory mechanisms (essentially unguided generation).

• Effectiveness in Violating Requirements:
  • Collision Avoidance (R1): RVSG significantly reduced the average distance to obstacles (DTO) compared to RVSGR, indicating more effective collision scenarios.
  • Stability (R2): RVSG generated scenarios with higher Jerk values (indicating unstable motion) in 4 out of 5 routes.
  • Efficiency (R3): RVSG effectively increased the Time-to-Reach-Goal (TRG), hindering robot performance.
• Performance Metrics:
  • RVSG consistently degraded robot performance metrics (Robot on Person Collisions, Cumulative Heading Changes, Path Length) more than the baseline.
  • Variability: Scenarios generated by RVSG induced higher standard deviations in robot behavior, revealing more unstable and uncertain robot responses.
• Diversity:
  • RVSG outperformed the baseline in Scenario Description Diversity (SDD) and High-level Task Diversity (HTD).
  • However, Scenario Simulation Diversity (SSD) was mixed; the guided nature of RVSG sometimes limited the exploration of rare edge cases compared to the more random baseline, suggesting a trade-off between guided relevance and broad exploration.
• Impact of Navigation Routes:
  • The complexity of the route significantly influenced results. Routes with environmental complexities (e.g., close to shelves) yielded higher requirement violations and instability compared to straight, open routes.

5. Significance and Implications

• Safety Assurance: RVSG provides a systematic, cost-effective way to stress-test AMRs against unpredictable human behaviors before real-world deployment, directly addressing safety risks.
• Support for Self-Adaptation: The generated scenarios support the MAPLE-K loop (a self-adaptation framework for robots) by providing critical test cases for the Plan, Legitimate, and Knowledge phases, helping robots learn to adapt to unsafe situations.
• Industrial Relevance: Developed in collaboration with PAL Robotics, the tool is designed for industrial adoption. It highlights that while VLMs can automate scenario generation, expert oversight remains crucial to validate feasibility and prevent hallucinations.
• Future Directions: The authors suggest exploring multi-objective optimization (violating multiple requirements simultaneously) and testing the framework on other robot platforms to ensure generalizability.

In conclusion, the paper demonstrates that integrating Vision Language Models into the testing pipeline significantly enhances the ability to generate realistic, diverse, and challenging test scenarios for industrial AMRs, thereby improving the reliability and safety of human-robot collaboration.