AgenticLab: A Real-World Robot Agent Platform that Can See, Think, and Act

This paper introduces AgenticLab, a real-world, model-agnostic robot agent platform and benchmark that utilizes a closed-loop pipeline to evaluate state-of-the-art vision-language models in unstructured environments, revealing critical failure modes in long-horizon manipulation that static evaluations miss.

The Gist

Imagine you want to teach a robot to make you a sandwich. You could write a strict script: "Pick up bread, pick up ham, put ham on bread." But what if the ham is hidden under a napkin? What if the bread is slippery? What if the robot accidentally knocks the ham off the table?

Most current robot brains are like scripted actors who memorize lines but freeze when the stage props move. They can "see" an image and "think" about it, but once they start moving, they often forget to check if their plan is still working.

AgenticLab is a new platform designed to fix this. Think of it as a training gym and a testing arena for robots that can actually think on their feet.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Blindfolded" Robot

Previous robot tests were like asking a student to solve a math problem on a piece of paper (static image) or in a video game (simulation).

• The Flaw: In the real world, things change. A robot might plan to grab a cup, but if the cup slides, the robot needs to stop, look again, and change its plan.
• The Old Way: Many robots just run their plan from start to finish without looking back. If they miss, they keep trying to grab the air, fail, and crash.

2. The Solution: The "Self-Correcting" Loop

AgenticLab introduces a robot that doesn't just "See, Think, Act." It "Sees, Thinks, Acts, Checks, and Fixes."

Imagine a Chef in a busy kitchen:

• See (The Eyes): The robot uses two cameras. One is like a wide-angle security camera (shoulder view) to see the whole kitchen layout. The other is a magnifying glass (wrist view) to look closely at the specific ingredient it's holding.
• Think (The Brain): Instead of just guessing, the robot breaks big tasks (like "make a salad") into tiny, logical steps (find lettuce, find bowl, grab lettuce, put in bowl). It uses a "symbolic planner" (like a strict recipe book) to ensure the steps make sense.
• Act (The Hands): It moves its arm to grab things.
• Check (The Taste Test): This is the magic part. After every single move, the robot stops and asks itself: "Did I actually grab the lettuce? Is my hand empty? Did I knock over the salt?"
• Fix (The Re-plan): If the answer is "No," it doesn't panic. It immediately switches to a new plan. Maybe it moves its arm closer, or looks with the "magnifying glass" camera to see what went wrong.

3. The "Model-Agnostic" Feature: The Universal Adapter

Usually, if you want to test a new robot brain (a specific AI model), you have to rebuild the whole robot's software to fit it. It's like trying to put a Ford engine into a Ferrari chassis; it's a nightmare.

AgenticLab is like a universal power strip. You can plug in any smart brain (like Gemini, GPT, or Qwen) into the same robot body. This allows scientists to fairly compare: "Which brain is actually better at not dropping the toast?" without worrying about the robot's hardware getting in the way.

4. What They Discovered (The "Aha!" Moments)

The researchers tested many different AI brains on real robots in messy, real-world kitchens and labs. They found some surprising things:

• The "Hallucination" Trap: Some very smart AI models are great at chatting but terrible at checking reality. They might confidently say, "I am holding the apple," even when their gripper is empty. In a robot, this "lying" causes the whole task to fail.
• The Bottleneck: The robot isn't limited by how well it understands the language; it's limited by how well it checks its own work. If the "checker" is weak, the whole robot fails, no matter how smart the planner is.
• The "Team" Approach: Instead of using one giant brain for everything, they found that a team of specialists works best. Use one AI to plan the steps, a different (smaller, faster) AI to find the objects, and a third to check if the grab was successful. This "composite" team often outperformed a single, massive brain.

5. Why This Matters

AgenticLab is like the first standardized driving test for self-driving cars, but for robots.

• Before, we only tested robots in perfect, clean video games.
• Now, we have a platform that throws real-world chaos at them: messy tables, bad lighting, and objects that move.

The Bottom Line:
AgenticLab proves that for robots to be useful in our messy homes and offices, they need to be humble and self-correcting. They need to constantly ask, "Did that work?" and be ready to try again if the answer is no. It's not just about being smart; it's about being reliable.

Technical Summary

1. Problem Statement

Recent advances in Large Vision-Language Models (VLMs) have demonstrated impressive open-vocabulary perception and reasoning capabilities. However, their application to real-world robot manipulation remains unproven, particularly for long-horizon, closed-loop execution in unstructured environments.

Current limitations in the field include:

• Evaluation Gaps: Most benchmarks rely on static image understanding (VQA), offline interfaces, or simulation, which fail to capture failure modes arising from physical dynamics, partial observability, and actuation noise.
• Lack of Closed-Loop Reasoning: Many existing VLM-based systems use open-loop planning (generate a plan once and execute) without iterative verification, making them brittle to grasp failures, object movement, or occlusion.
• Benchmarking Inconsistency: Prior systems often use model-specific designs or are evaluated on single VLMs, making it difficult to compare different model families under a consistent protocol.
• VLA Limitations: Vision-Language-Action (VLA) models often require extensive fine-tuning on specific tasks, leading to catastrophic forgetting and poor generalization in open-world scenarios.

2. Methodology: The AgenticLab Platform

The authors introduce AgenticLab, a model-agnostic, real-world robot agent platform designed for open-world manipulation. It integrates hardware and software to support reproducible, closed-loop reasoning.

A. Hardware Stack

• Robot: A UR5e arm mounted on a movable base for indoor/outdoor deployment.
• Perception: Dual RGB-D views:
  • Azure Kinect: Fixed shoulder view for global scene context and layout understanding.
  • RealSense D405: Wrist-mounted camera for detailed, close-range feedback during manipulation.
• End-Effector: A low-cost, custom parallel gripper with Fin-Ray fingers (cost < $200), capable of grasping diverse everyday objects.
• Software: A lightweight robotics stack for trajectory generation, inverse kinematics, and multi-camera calibration (lifting 2D coordinates to 3D metric space).

B. Software Framework: Modular Closed-Loop Pipeline

The core innovation is a model-agnostic pipeline that interleaves execution, verification, and replanning. The system is decomposed into three main phases ("See, Think, Act") with specific modules:

1. See (Perception):

   • Converts RGB-D observations into structured scene representations.
   • Supports open-vocabulary grounding via swappable strategies (e.g., direct VLM prediction or LangSAM + VLM verification).
   • Operates iteratively, using visual feedback to adapt to noise and scene changes.

2. Think (Reasoning & Planning):

   • Task Parser: Translates natural language instructions into a structured PDDL (Planning Domain Definition Language) problem. It grounds objects and predicates based on visual observations.
   • Symbolic Planner: Uses a classical planner (Fast Downward) to generate a sequence of high-level action primitives, ensuring logical consistency.
   • Action Checker: Validates preconditions before execution and effects after execution. It detects failures (e.g., missed grasps, hallucinations) and triggers replanning.
   • Grasp Planner: Generates candidate 6-DoF grasp poses (using AnyGrasp) and uses a VLM to verify semantic correctness and collision safety. If a grasp is rejected, the system switches to the wrist camera for localized replanning.

3. Act (Execution):

   • Executes high-level primitives (pick, place, open/close) via position-based control.
   • Monitors outcomes via discrete verification after each action.
   • Closed-Loop Mechanism: If verification fails, the agent retries the action or replans the entire sequence based on updated observations.

3. Key Contributions

1. Model-Agnostic Pipeline: A unified interface allowing different VLMs (e.g., Gemini, GPT, Qwen) to be swapped in and out without engineering changes, enabling fair comparison.
2. Real-World Benchmark: A comprehensive suite of 5 tasks (Sorting, Stacking, Crossword, Reorientation, Kitchen) evaluated across three environments (Lab, Kitchen, Outdoors). It tests grounded perception, spatial reasoning, and sequential decision-making under closed-loop execution.
3. Open-Source Platform: Release of the full hardware and software stack to lower the barrier to entry for embodied AI research and support reproducible evaluation.

4. Experimental Results & Analysis

The authors evaluated multiple state-of-the-art VLMs and compared them against fine-tuned VLAs.

• Single VLM Performance:

  • Gemini Flash achieved the highest success rate (75%) on the sorting task, balancing speed and grounding.
  • Failure Modes: Models like Qwen-VL-Max and GPT-5.2 failed significantly (near 0% success) primarily due to unreliable action verification (hallucinating that an object was not grasped).
  • Compounding Errors: The paper highlights that even moderate verification errors (e.g., 50% accuracy) lead to catastrophic failure in long-horizon tasks because errors compound exponentially over multiple steps.

• Module Benchmarking:

  • Task Parsing: Cloud-hosted models (Gemini, GPT) generally performed well; local models struggled with structured PDDL generation.
  • Object Detection: Specialized models (e.g., Qwen3-VL-Plus) outperformed larger general models in pointing accuracy, suggesting grounding can be offloaded to smaller, specialized architectures.
  • Verification: This was identified as the primary bottleneck. Models often fail to interpret complex interaction semantics (e.g., occlusion by the gripper).

• Compositional vs. Single VLM:

  • A compositional pipeline (using different models for different modules, e.g., Gemini for planning, Qwen for grounding, Claude for verification) did not consistently outperform the best single model (Gemini Flash) but demonstrated the flexibility to compensate for specific weaknesses.

• Ablation Studies:

  • Action Checker: Dense verification (checking after every action) is crucial for tasks with strong dependencies (like stacking). Disabling it led to 0% success in stacking tasks due to error propagation.
  • Grasp Planner: Enabling grasp evaluation significantly improved success in cluttered scenes by preventing collision-prone grasps.

• AgenticLab vs. Fine-Tuned VLA ($\pi0.5$):

  • AgenticLab outperformed a fine-tuned VLA ($\pi0.5$) on sorting and stacking tasks.
  • The VLA struggled with fine-grained semantic constraints (e.g., picking the wrong object type) and lacked the high-level reasoning and closed-loop recovery capabilities of the AgenticLab pipeline.

5. Significance and Future Work

• Paradigm Shift: The paper argues that for open-world manipulation, closed-loop consistency is more critical than open-loop reasoning depth. Reliable verification and recovery mechanisms are the limiting factors for robustness.
• Practical Guidance: It provides a blueprint for deploying VLMs on physical robots, emphasizing the need for modular design and the trade-off between verification frequency and execution time.
• Future Directions:
  • Moving from manually authored PDDL domains to learning-based domain modeling.
  • Augmenting hand-designed primitives with learning-based atomic skills.
  • Distilling large VLMs into smaller, specialized models to reduce verification latency.

In conclusion, AgenticLab demonstrates that while current VLMs have limitations in real-world robotics, a structured, closed-loop, model-agnostic framework can effectively leverage their reasoning capabilities to perform complex manipulation tasks in unstructured environments, outperforming both open-loop systems and specialized fine-tuned VLAs.