Technical Summary: ModuLoop – Low-Level Code Generation using Modular Synthesizer and Closed-Loop Debugger for Robotic Control

Problem Statement

While Large Language Models (LLMs) have demonstrated impressive capabilities in code generation and symbolic reasoning, their application to low-level robotic control remains limited. Existing approaches generally fall into two categories: (1) translating natural language into high-level plans that rely on external modules for execution, or (2) directly generating executable low-level code via user-defined APIs. The former limits LLM involvement in the execution loop, while the latter often suffers from prompt dependency, poor generalization, and a lack of feedback-based refinement. These issues are critical in low-level control tasks, which require precise motion, real-time error handling, and adaptation to physical environments without extensive manual tuning or pre-defined datasets.

Methodology: The ModuLoop Framework

The authors propose ModuLoop, a framework designed to enable LLMs to participate actively in the full control loop of low-level robotic tasks without task-specific fine-tuning. The system operates through a closed-loop structure comprising three core components:

1. Simulation-Based Workspace Validation

To ensure physical feasibility before real-world execution, the framework employs a simulation-based validation environment (Isaac Sim).

• Process: The LLM generates candidate end-effector positions based on task constraints and prior feedback.
• Filtering: These candidates undergo Inverse Kinematics (IK) filtering and simulation checks to classify them as reachable, collision-prone, or unreachable.
• Feedback: The results are fed back to the LLM to refine subsequent generations, ensuring only collision-free, kinematically valid coordinates are used for calibration.

2. Modular Code Synthesizer (MCS)

Instead of generating a monolithic script, the LLM decomposes high-level natural language instructions into fine-grained subtasks.

• Decomposition: The task is broken into independent subtasks (e.g., moving to coordinates, acquiring sensor data, computing matrices).
• Generation: Each subtask is implemented as a distinct Python function or code block.
• Integration: These blocks are assembled into a coherent, executable script. This modular approach ensures logical consistency and structural completeness.

3. Closed-Loop Debugger (CLD)

The framework iteratively executes the generated code and uses execution feedback to autonomously refine it.

• Execution & Analysis: The code runs in the robotic environment. If a runtime error occurs, an Analyzer formulates hypotheses about the cause.
• Probing: To validate hypotheses, the system inserts debugging probes (e.g., additional logging or diagnostic checks) into the code and re-executes it.
• Refinement: A Refiner applies targeted modifications based on the analysis.
• Accuracy Evaluation: If no runtime errors occur but the task accuracy is insufficient (e.g., calibration error > threshold), the system diagnoses the cause (e.g., lack of depth filtering) and refines the code to improve precision.
• Termination: The loop continues until the task meets accuracy criteria or a maximum iteration limit is reached.

Key Contributions

The paper outlines three primary contributions:

1. Simulation-Based Feedback Loop: A mechanism that secures collision-free, executable coordinates for hand-eye calibration, significantly enhancing data efficiency and accuracy compared to purely analytical methods.
2. Modular Code Synthesizer: A framework that decomposes natural language commands into executable low-level Python modules, moving beyond single-pass code generation.
3. Closed-Loop Debugging Mechanism: An autonomous system that uses runtime errors and accuracy metrics to iteratively revise and improve generated code, transforming the LLM from a passive planner into an active, self-correcting agent.

Experimental Results

The framework was validated on a UR3 robotic manipulator with an Intel RealSense D435i camera across two tasks:

1. Hand-Eye Calibration

• Performance: The combination of MCS and CLD with probing achieved a 96.67% success rate in generating code and an 86.67% success rate in meeting the accuracy threshold (< 2 cm error).
• Comparison: This significantly outperformed single-prompt generators (SPG) and baseline methods like Code-as-Policies (CaP) and ProgPrompt. For instance, SPG + Debugger achieved only a 10% success rate, while the proposed ModuLoop achieved 96.67%.
• Efficiency: The probing-based debugging reduced the number of iterations required for convergence compared to non-probing methods.
• Model Variants: While GPT-4o yielded the best results, GPT-4.1-mini and Gemini also demonstrated viability, with Gemini achieving comparable code generation success but slightly lower calibration stability.

2. Pick-and-Place Tasks

• Setup: The framework was applied to five tasks of increasing complexity, ranging from simple object targeting to tasks requiring semantic reasoning, spatial ordering, and orientation control.
• Performance: ModuLoop achieved the highest success rates across all tasks, particularly in complex scenarios (Tasks 3–5). For "Hard 1" (spatial reasoning and precise positioning), ModuLoop achieved a 60% success rate, compared to 24% for Single-Prompt and 48% for Modular Code Synthesizer alone.
• Generalizability: The system successfully handled tasks requiring context-aware object identification (e.g., "get me some snacks") and geometric reasoning without explicit programming.

Significance and Claims

The paper asserts that ModuLoop demonstrates the practicality of using pre-trained LLMs to autonomously generate, execute, and adapt low-level robot control code without further training or manual programming.

• Autonomy: The framework elevates LLMs from passive planners to autonomous agents capable of real-time diagnosis and execution-driven refinement.
• Scalability: By validating the system in both calibration (requiring high precision) and manipulation (requiring semantic reasoning), the authors claim the framework is scalable and extensible to diverse robotic tasks.
• Limitations: The authors modestly acknowledge that the current implementation relies on minimal APIs (primarily object positions) and is best suited for tasks that do not require complex contact-rich manipulation (e.g., assembly or drawer opening) without richer environmental APIs. They also note that the inherent latency of LLMs may limit responsiveness in high-speed industrial applications.

In conclusion, ModuLoop provides a robust pathway for integrating LLMs into the physical control loop, bridging the gap between natural language instructions and precise, executable robotic behavior.