Agent-Driven Autonomous Reinforcement Learning Research: Iterative Policy Improvement for Quadruped Locomotion

1. Problem Statement

The paper addresses the challenge of automating the Reinforcement Learning (RL) research loop in the complex domain of robotics. Unlike standard ML tasks (e.g., image classification) that run in stable, single-GPU environments, robotics RL involves:

• Physics Simulation: Dependence on simulators (Isaac Sim/Lab) prone to runtime errors (e.g., PhysX deadlocks).
• Reward Engineering: The need to design and tune dense reward functions for locomotion.
• Terrain Curricula: Managing complex terrain generation that can cause simulation crashes.
• Multi-GPU Management: Orchestrating parallel experiments across multiple GPUs.

The core question is: Can an autonomous agent, guided only by high-level human directives, execute the full iterative research cycle (hypothesis, code editing, execution, diagnosis, and pivoting) in a failure-prone robotics setting?

2. Methodology

The study employs a Wave-Based Autonomous Research Loop rather than a single global hyperparameter sweep. The system architecture consists of:

• Human Supervisor: Provides high-level goals (e.g., "tune Basic PPO and HIM," "consult reference implementations").
• Agent Execution Layer: An AI agent (Claude via OpenCode) operating in an agentic coding environment. It has access to shell commands, file editing, logs, and GPU schedulers.
• Hardware/Software Stack:
  • Robot: DHAV1 12-DoF quadruped.
  • Simulator: Isaac Lab / Isaac Sim.
  • Compute: 4 NVIDIA RTX PRO 4500 Blackwell GPUs.
  • Algorithms: Basic PPO and Hierarchical Inverse Model (HIM).

The Research Loop (5 Stages):

1. Observation: Inspecting logs, reward decompositions, and failure traces.
2. Hypothesis: Identifying bottlenecks (e.g., "Is the crash due to terrain primitives or reward penalties?").
3. Intervention: Editing code, modifying reward weights, or changing terrain configurations.
4. Execution: Launching parallel experiments and monitoring for hangs.
5. Triage: Terminating unproductive or hung runs and scheduling replacements.

The agent operated over 14 waves and 70+ experiments, making autonomous decisions on code modifications, environment counts, and reward porting.

3. Key Contributions

The paper's primary contribution is empirical and documentary, demonstrating that an agent can materially advance a robotics RL project with limited human intervention. Key autonomous achievements include:

• Root-Cause Diagnosis: The agent independently identified that PhysX deadlocks were caused specifically by terrain sets containing "boxes" and "stair-like" primitives. It successfully pivoted to a smoother, reference-inspired terrain mixture, enabling stable long-duration training.
• Reward Engineering & Porting: Instead of generating rewards from scratch, the agent identified, ported, and tuned four specific reward terms from open-source reference implementations (diagonal gait, air-time variance, joint-mirror symmetry, and feet-contact shaping).
• Strategic Pivoting: The agent observed that the HIM policy consistently failed to achieve terrain progression (stuck at 0.0) despite fixes, while Basic PPO succeeded. It autonomously decided to de-emphasize HIM and focus computational resources on Basic PPO.
• Infrastructure Debugging: The agent resolved Isaac Sim bootstrapping issues (Omni/pxr import errors) and managed GPU resource allocation, including killing hung processes to free up capacity.

4. Results

The study tracked performance from early baselines to a converged optimal policy.

• Performance Trajectory:
  • Initial State (Wave 1): Mean reward ~7–10; frequent simulation hangs; HIM and PPO both struggled.
  • Convergence (Wave 12, exp063):
    • Mean Reward: ~153.4 (a ~16x increase from baseline).
    • Velocity Error: 0.263 (Planar velocity tracking error).
    • Survival Rate: 97% timeout over 2,000 iterations.
• Reproducibility: The best result (exp063) was independently reproduced five times across different GPUs, confirming convergence.
• Policy Comparison: Basic PPO significantly outperformed HIM in this specific rough-terrain setting. HIM could survive episodes but failed to progress across terrain, whereas PPO learned effective locomotion.
• Parameter Sensitivity: The agent systematically explored the configuration space:
  • Gait Weight: Optimal at 0.5 (U-shaped curve; too high or too low hurt performance).
  • Stability Penalties: Optimal at -10.0 (Height/Orientation); -12.0 caused instability.
  • Airtime Variance: Optimal at -6.0.
  • Tracking Weights: Higher weights inflated total reward but degraded actual tracking error, highlighting the need for scale-independent metrics.

Key Finding: The agent discovered that reducing gait enforcement while strengthening stability penalties and airtime regularization yielded the best tracking performance—a non-obvious insight derived from systematic single-variable experimentation.

5. Significance

• Beyond Code Completion: This work moves beyond simple code generation to closed-loop scientific discovery. The agent did not just write code; it diagnosed simulation failures, interpreted physical constraints, and altered the research agenda based on empirical evidence.
• Realistic Robotics Constraints: Unlike previous autonomous research papers (e.g., AutoResearch or The AI Scientist) that focus on clean, single-GPU ML tasks, this study operates in a "messy" robotics environment involving physics engine crashes, multi-GPU scheduling, and complex terrain generation.
• Human-Agent Collaboration: The paper clarifies the division of labor: Humans define the broad agenda (e.g., "improve rough-terrain locomotion"), while the agent executes the detailed research loop (debugging, tuning, triaging).
• Methodological Insight: It demonstrates that root-cause diagnosis (e.g., identifying the specific terrain primitive causing a deadlock) is often more critical to progress than hyperparameter tuning alone.

Conclusion: The study proves that agent-driven systems can execute the iterative RL research loop in robotics, handling engineering triage and reward engineering to achieve state-of-the-art results in a specific domain, though human guidance remains essential for setting the high-level research direction.