⚡ electrical engineering

Test-Driven Agentic Framework for Reliable Robot Controller

This paper presents a test-driven, agentic framework that iteratively refines robot navigation controllers through a dual-tier repair strategy and structured test feedback, significantly improving reliability and robustness compared to one-shot generation methods in both 2D and 3D environments.

Original authors: Shivanshu Tripathi, Reza Akbarian Bafghi, Maziar Raissi

Published 2026-03-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shivanshu Tripathi, Reza Akbarian Bafghi, Maziar Raissi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brand-new, very smart, but slightly clumsy robot how to navigate a maze. You give it a simple instruction: "Go from here to there without hitting anything."

In the old days, a human engineer would have to spend weeks manually writing the code, tweaking the wheels, and debugging why the robot keeps bumping into walls. It's like trying to teach a dog to do a backflip by manually moving its legs for hours.

This paper introduces a new, automated way to do this using Artificial Intelligence (AI) and a "Test-Driven" approach. Think of it as a perfectionist coach who doesn't just give the robot a single try, but keeps coaching it until it gets it right.

Here is how their system works, broken down into simple concepts:

1. The Problem: The "One-Shot" Mistake

Usually, when we ask an AI to write code for a robot, we ask it once and hope for the best. This is like asking a chef to cook a complex dish and telling them, "Just make it, I'll taste it later."

The Risk: The AI might hallucinate (make things up). It might forget that the robot's wheels can only spin so fast, or it might confuse a black wall with a white floor.
The Result: The robot crashes, gets stuck, or the code doesn't even run.

2. The Solution: The "Agentic" Loop

The authors created a closed-loop system (a cycle that keeps going until it works). They call it an "Agentic Framework." Imagine a team of two AI specialists working together:

The Learner (The Architect): This AI tries to write the robot's "brain" (the code) based on your instructions.
The Optimizer (The Quality Control Inspector): This AI acts like a strict test manager. It doesn't just look at the code; it runs the robot in a simulation to see if it crashes.

3. The Process: How the "Coach" Works

Here is the step-by-step cycle, using a Gym Analogy:

The First Attempt: The "Architect" writes a workout plan (the code) for the robot.
The Workout (The Test): The "Inspector" puts the robot in a virtual gym (a simulation) and makes it try to walk the path.
The Feedback:
- Did it trip? The Inspector says, "You tried to walk through a wall!"
- Did you stop too early? The Inspector says, "You didn't go far enough."
- Did the code crash? The Inspector says, "You tried to use a tool that doesn't exist."
The Repair (The Coaching):
- Level 1 (Fixing the Move): If the robot just made a small mistake, the AI tries to tweak the specific move in the code (like telling the robot, "Turn left 10 degrees more").
- Level 2 (Fixing the Instructions): If the robot keeps failing no matter what, the AI realizes the original instructions were confusing. It goes back to the "Architect" and says, "Hey, you didn't tell the robot that the walls are dark, not light. Let's rewrite the instructions."
Repeat: They try again. And again. Until the robot runs the course perfectly without hitting a single wall.

4. Two Different Playgrounds

The team tested this in two different environments:

The 2D Map (The Paper Maze): They took a picture of a map, turned it into a digital grid (like a chessboard), and taught the robot to navigate it. The system had to figure out if black pixels were walls or white pixels were walls.
The 3D Simulator (The Virtual World): They used a realistic video-game-like world (Webots) with a physical robot (an e-puck). Here, the robot has to deal with physics, like gravity and sensor limits, just like a real robot would.

5. The Results: Why It Matters

The paper shows that this "keep trying until you get it right" method is much better than just asking the AI once.

One-Shot: If you ask the AI once, it might fail 80-90% of the time.
Test-Driven Loop: By letting the AI fix its own mistakes, the success rate jumps significantly. Even if the AI starts with a bad idea, the loop corrects it.

The Big Picture

This research is a giant step toward automating robotics engineering. Instead of a human spending days debugging code, the computer does the heavy lifting. It treats robot programming like a game of "Guess and Check," but done at super-speed by AI.

In short: They built a system where the AI writes the code, the AI tests it, the AI finds the mistakes, and the AI fixes them, repeating this cycle until the robot is ready to go out into the real world and do its job safely. It's like having a tireless, super-smart tutor that refuses to let the student fail.

1. Problem Statement

The paper addresses the critical gap between theoretical motion planning and reliable physical execution in robotics. While Large Language Models (LLMs) show promise in synthesizing code from high-level instructions, one-shot code generation is often unreliable for safety-critical robotic tasks.

The Challenge: LLM-generated controllers frequently suffer from syntax errors, runtime crashes, safety violations (collisions), or failure to reach goals. These failures often stem from underspecified prompts that omit crucial details like robot dynamics, unit conversions, timing constraints, and specific API contracts.
The Consequence: Without grounded details, LLMs rely on hallucinated assumptions, leading to brittle performance that requires extensive manual debugging and parameter tuning to fix.
The Goal: To develop an automated, closed-loop framework that synthesizes a deployable, low-level robot controller from high-level natural language instructions, ensuring the code is syntactically correct, safe, and functionally successful before deployment.

2. Methodology

The authors propose a Test-Driven Agentic Framework that replaces manual debugging with an automated, iterative refinement loop. The system operates as a closed-loop agent that generates, tests, diagnoses, and repairs code.

A. Core Workflow

The framework follows a Generate $\rightarrow$ Verify $\rightarrow$ Repair cycle:

Generation: An LLM (the "Learner") generates a candidate Python controller (controller.py) based on a prompt containing task specifications and environment context.
Verification: The generated code is executed against a comprehensive, structured PyTest suite. These tests check for:
- Functional Correctness: Syntax errors, import errors, runtime exceptions.
- Performance & Safety: Collision avoidance, reaching the goal, adherence to kinematic constraints, and task completion within time steps.
Diagnosis & Repair: If tests fail, a diagnostic report is generated. A second LLM (the "Optimizer") analyzes this feedback and initiates a Dual-Tier Repair Strategy:
- Code-Level Repair: Directly edits the source code (controller.py) to fix specific errors. This is attempted for a limited number of iterations (Edit Patience, $J$ ).
- Prompt-Level Repair: If code edits fail to resolve the issue, the system updates the Prompt Template (specifically the AUTO_REPAIR_RULES section) to guide the Learner LLM toward a better solution in the next generation cycle.

B. Two Experimental Settings

The framework is validated in two distinct environments:

2D Map-Based Navigation:
- Input: Raw RGB map images and start/goal coordinates.
- Preprocessing: A robust pipeline converts raw images into grayscale occupancy grids. It uses Otsu's method for binarization, evaluates both dark/light pixel polarities, applies morphological operations, and inflates obstacles for safety. It selects the best grid candidate based on path efficiency and safety scores.
- Controller: Synthesizes a differential-drive controller using ultrasonic sensor simulation (ray-casting) on the occupancy grid.
3D Webots Simulation:
- Input: Webots world files (.wbt) defining the arena, robot (e-puck), and obstacles.
- Process: No map preprocessing is needed. The LLM is grounded directly on the simulator's physics and sensor API (infrared proximity sensors).
- Controller: Generates a controller that interfaces directly with the Webots API to drive the robot in a 3D physics environment.

3. Key Contributions

Novel Agentic Workflow: A closed-loop system that synthesizes controllers from underspecified instructions using automated test-driven refinement, shifting the engineering burden from manual implementation to automated verification.
Dual-Tier Repair Strategy: A unique mechanism that alternates between direct code editing and prompt refinement, allowing the system to recover from both implementation bugs and conceptual misunderstandings.
Unified Methodology: A pipeline that handles both 2D image-based navigation (with robust map preprocessing) and 3D physics-based simulation, demonstrating versatility across different sensing modalities.
Self-Contained Deployment: Unlike other LLM-robotics works that require the LLM to be present during runtime, this framework produces a standalone executable controller that operates without further LLM intervention once verified.

4. Experimental Results

The framework was evaluated across 2D and 3D environments using multiple LLMs (GPT-4o, GPT-4.1, GPT-5.2, Claude-Sonnet-4.5, Claude-Opus-4.5).

Success Rate (SR) Improvement:
- 2D Navigation: Iterative refinement significantly outperformed one-shot generation. For example, GPT-5.2 and Sonnet-4.5 achieved a 45% success rate with the iterative framework, compared to near-zero success for one-shot attempts by weaker models.
- 3D Webots: Performance was generally higher due to reduced unknown parameters. Claude-Opus-5.2 achieved a 71% success rate, significantly outperforming GPT-5.2 (39%) and GPT-4o (21%).
Cumulative Success (CS):
- The iterative approach allowed even lower-performing models to converge on a working solution. For instance, in 3D tasks, Opus-4.5 reached 100% cumulative success by iteration $k=2$ , whereas one-shot attempts failed for many models.
- GPT-4o, which failed completely in single-shot attempts, achieved success by iteration $k=15$ in 2D and $k=7$ in 3D through iterative repair.
Efficiency: The framework consistently produced deployable controllers within a few iterations (typically $k < 10$ ), proving that the cost of iteration is outweighed by the reliability gain.

5. Significance

Reliability over Speed: The paper demonstrates that while LLMs are powerful, their raw output is insufficient for robotics. A test-driven, iterative approach is necessary to bridge the "reality gap" between code generation and safe execution.
Automation of Debugging: The framework automates the tedious process of parameter tuning and heuristic debugging, making it scalable for diverse robot configurations and environments.
Safety-Critical Application: By enforcing strict PyTest suites (checking for collisions, deadlocks, and goal reachability), the framework ensures that only verified, safe controllers are deployed, addressing a major barrier to autonomous robotics adoption.
Future Direction: The authors suggest this paradigm can be extended to dynamic environments with moving obstacles, further advancing the field of autonomous system synthesis.

In summary, this work establishes that agentic, test-driven refinement is a superior paradigm for robotic controller synthesis compared to one-shot prompting, effectively transforming high-level, underspecified instructions into robust, deployable low-level control software.