RACAS: Controlling Diverse Robots With a Single Agentic System

The paper introduces RACAS, a robot-agnostic agentic system that uses natural language communication between LLM/VLM-based modules to control diverse robotic platforms without requiring code modifications or retraining, successfully demonstrating its effectiveness across wheeled, multi-jointed, and underwater robots.

The Gist

Imagine you have a toolbox full of very different robots: a wheeled rover that drives on land, a mechanical arm that looks like a human limb, and a submarine that swims underwater.

In the old days, if you wanted to teach the rover to find a fire extinguisher, you had to write a specific computer program for it. Then, if you wanted the submarine to do the same thing, you had to throw away that code and write a completely new one from scratch. It was like trying to teach a dog to fetch and a fish to fetch; you'd need two totally different training manuals.

RACAS (Robot-Agnostic Control via Agentic Systems) is a new way of doing things that changes the game. Instead of writing code for every robot, the researchers created a "Universal Robot Manager" that speaks only English.

Here is how it works, using a simple analogy:

The "Three-Headed Brain"

Instead of one giant, complicated computer brain, RACAS uses three specialized AI assistants (called "agents") that talk to each other like a human team. They don't speak code; they speak natural language.

1. The Controller (The Captain): This is the boss. It looks at the mission ("Find the red box") and asks the team, "What do we see? What should we do next?" It makes the final decision on which button to press.
2. The Monitors (The Eyes): These are the lookouts. They look at the camera feeds and describe what they see in plain English. Instead of sending raw numbers like "Object at coordinates X,Y," they say, "I see a red box about 5 meters to the left."
3. The Memory Curator (The Librarian): This is the most important new feature. As the robot moves, it generates a lot of information. If you just kept adding notes to a piece of paper, it would get messy and too long to read. The Librarian reads the notes, summarizes the important parts, throws away the junk, and keeps a tidy, organized diary of what happened. This helps the robot remember where it's been and what it learned, without getting confused.

The Magic Trick: "No Code Required"

The coolest part of RACAS is how easy it is to switch robots.

• Old Way: To switch from a land rover to a submarine, you had to rewrite the software, retrain the AI, and understand the physics of water vs. wheels.
• RACAS Way: You just write a short note (a "prompt") describing the new robot.
  • Note to the AI: "You are now a submarine. You have 6 thrusters. You can move up, down, left, right, forward, and backward. Your goal is to find a green box."
  • That's it. No coding, no math, no retraining. The AI reads the note, understands the new "body," and starts working immediately.

The Real-World Test

The researchers tested this on three very different robots:

1. A Wheeled Rover (Dingo): Driving on the floor in a warehouse.
2. A Robotic Arm (Alhakami Limb): A complex, multi-jointed arm that the AI had never seen before in its training data.
3. A Submarine (BlueROV2): Swimming underwater in a pool.

The Result: The same "Universal Manager" successfully guided all three robots to find their targets. It didn't matter that one had wheels, one had joints, and one had propellers. The AI figured out how to use their specific "body" just by reading the description.

Why This Matters

Think of RACAS as a universal translator for robots.

Before, building a robot solution was like building a custom house for every single family. It was slow, expensive, and required a different architect for every job.

With RACAS, you can now build a "prefab" solution. You just tell the system what the robot looks like and what the job is, and the AI figures out the rest. This means engineers and scientists can prototype new robots and test them on different tasks much faster, without needing to be experts in coding or complex math for every new machine.

In short: RACAS lets us talk to robots in plain English, and they listen, no matter what shape or size they are.

Technical Summary

Here is a detailed technical summary of the paper "RACAS: Controlling Diverse Robots With a Single Agentic System."

1. Problem Statement

Current robotic control pipelines face a significant barrier to entry: transitioning from low-level hardware APIs (actuator commands, sensor streams) to high-level autonomous behavior requires a complex, multi-stage pipeline involving distinct areas of expertise (e.g., firmware engineering vs. planning algorithms).

• Limitations of Existing Approaches:
  • End-to-End Learning: Requires massive amounts of demonstration data and retraining for every new robot embodiment (morphology). These methods fail when the target platform differs structurally from the training fleet.
  • LLM-Based Control: While Large Language Models (LLMs) and Vision-Language Models (VLMs) have been used to bridge high-level intent and low-level commands, previous systems have only been validated on structurally similar platforms (e.g., different manipulator arms) in controlled lab settings.
• The Gap: There is a lack of zero-training generalization across radically heterogeneous platforms where morphology, dynamics, and operating environments differ fundamentally.

2. Methodology: RACAS Architecture

The authors propose RACAS (Robot-Agnostic Control via Agentic Systems), a cooperative agentic architecture that achieves closed-loop control using only natural language for communication between modules. The system requires no source code changes, model weight modifications, or reward function tuning when switching robots.

Core Components

The system consists of three LLM/VLM-based modules orchestrated in a closed loop:

1. Controller (Decision Maker):
   • Acts as the central planner.
   • Receives a dynamically composed system prompt containing: Robot description ($D$), Action definitions ($A$), Environment memory ($M_{t-1}$), Proprioceptive state ($s_t$), Action history ($H_t$), and Task specification ($\tau$).
   • Generates a targeted visual query ($q_t$) based on the current task state.
   • Receives scene descriptions from Monitors and outputs a reasoning trace followed by a single action ($a_t$).
2. Monitors (Perception):
   • A set of VLM-based agents, each processing a specific camera feed.
   • They receive the visual query ($q_t$) from the Controller and return a natural language scene description ($o^{(c)}_t$).
   • Key Feature: Perception is task-adaptive. Instead of fixed-schema outputs (bounding boxes), the VLM answers specific questions relevant to the current execution stage.
3. Memory Curator (Adaptation):
   • A separate LLM instance that maintains a structured, bounded representation of accumulated knowledge.
   • Incremental Rewriting: Instead of appending logs, the Curator rewrites the memory state ($M_t$) after every step, compressing redundancy, resolving contradictions, and discarding irrelevant details.
   • Structured Schema: Organizes knowledge into four categories:
     1. Physical environment (layout, object inventory).
     2. Robot state (position, orientation, joints).
     3. Curated history of significant commands/outcomes.
     4. Task state (objectives, completion status).
   • Cross-Modal Inference: Infers object positions by intersecting camera observations with movement history (e.g., if an object appears in the right camera after a left turn, it is inferred to be on the left).

Adaptation Mechanism

To adapt RACAS to a new robot, the user only provides three text-based configuration files:

1. Robot Description ($D$): Natural language summary of morphology and sensors.
2. Action Definitions ($A$): Structured JSON schema mapping natural language commands to API calls.
3. Task Specification: Description of the goal and constraints.

3. Experimental Setup

The system was evaluated on three radically different platforms without modifying the control logic or model weights:

1. Alhakami et al. Limb: A novel 4-DOF multi-jointed robotic arm with 4 cameras (100x100 px).
   • Task: Object localization (finding a fire extinguisher) in a cluttered workspace.
2. Clearpath Dingo: A 3-DOF wheeled ground robot.
   • Settings: Simulated (NVIDIA Isaac Sim) and Real-world.
   • Task: Target approach (navigating to a fire extinguisher).
3. BlueROV2: A 6-DOF underwater remotely operated vehicle (ROV).
   • Settings: Small tank and Large pool.
   • Task: Underwater navigation to a designated target (green/blue boxes).

4. Key Results

• Zero-Shot Generalization: RACAS successfully completed all assigned tasks across all three platforms using the exact same Controller, Monitor, and Curator code.
• Performance:
  • RACAS significantly outperformed random baselines (where applicable) with statistical significance ($p < 0.01$).
  • Steps to Completion:
    • Limb: 9.56 steps (vs. 22.07 random).
    • Dingo (Sim): 16.40 steps (vs. 25.00 random).
    • Dingo (Real): 25.80 steps.
    • BlueROV2 (Small Tank): 14.00 steps.
    • BlueROV2 (Large Pool): 19.20 steps.
• Behavioral Observation: Once the target was located, RACAS consistently directed the robot in a near-straight line toward the objective. Performance was primarily limited by sensor fidelity and the time required to locate the object, rather than the reasoning capabilities of the models.
• Memory Validation: A proof-of-concept experiment using a modified Blackjack environment demonstrated that the Memory Curator is essential for inferring hidden targets from episodic feedback, outperforming non-structured or memory-less baselines.

5. Key Contributions

1. RACAS Architecture: A novel cooperative multi-module agentic system where all inter-module communication occurs exclusively in natural language.
2. Declarative Adaptation: Demonstrated that all embodiment-specific knowledge can be confined to prompt configurations, eliminating the need for code changes, weight fine-tuning, or reward engineering.
3. First Zero-Training Generalization: To the authors' knowledge, this is the first demonstration of a single control framework achieving zero-training generalization across three fundamentally different platforms (arm, wheeled ground, underwater), including a recently published robot unknown to the LLM's training data.

6. Significance and Future Work

• Significance: RACAS drastically reduces the barrier to prototyping robotic solutions. It allows practitioners to deploy autonomous behaviors on diverse hardware using only existing APIs and natural language descriptions, bypassing the need for specialized data collection and retraining pipelines.
• Limitations:
  • Current evaluation is limited to navigation and localization; contact-rich manipulation tasks are not yet fully addressed.
  • High inference latency (5–10 seconds per step) makes long-horizon tasks slow.
  • Lack of metric depth information leads to sub-optimal depth inference, causing robots to be under- or over-confident in collision avoidance.
• Future Work: Integrating manipulation primitives as callable tools to reduce decision frequency and incorporating LIDAR or advanced depth estimation into the Monitor subsystems.