When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?

Technical Summary: When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?

Problem Statement
Reinforcement Learning (RL) has established a central goal of producing policies that maximize expected return through interaction with an environment. However, standard RL algorithms (e.g., PPO, SAC, DQN) are often computationally intensive, requiring substantial environment interactions, and are brittle due to their dependence on careful hyperparameter tuning, model architecture selection, and algorithm choice. This paper investigates whether Large Language Models (LLMs) can serve as effective "black-box" policy optimizers that act as drop-in replacements for standard RL algorithms. Specifically, the authors ask: Can an LLM, given only a Python description of the state space, action space, and reward function, generate and refine a policy that maximizes expected return without the extensive design decisions and sample inefficiencies typical of traditional RL?

Methodology: Prompted Policy Optimization (PromptPO)
The authors introduce Prompted Policy Optimization (PromptPO), an iterative framework where an LLM generates executable Python code representing a policy. The process operates as follows:

1. Input: The LLM receives a Python-formatted description of the environment's state space, action space, and reward function. Crucially, the authors avoid providing explicit transition dynamics to evaluate the method in model-free settings.
2. Evaluation Function Generation: Before policy generation, the LLM is prompted to implement a Feedback class. This class summarizes trajectory-level statistics (e.g., mean reward, min/max values) to provide supervision beyond the raw reward signal, analogous to how RL algorithms use temporal difference errors.
3. Policy Generation: The LLM is prompted to generate a policy class (implemented in Python) that takes an observation and outputs an action to maximize expected return.
4. Iterative Refinement:
   • $N$ candidate policies are sampled from the LLM.
   • Each policy is rolled out in the environment.
   • The best-performing policy (based on return) is selected.
   • The LLM receives a natural language reflection comparing the new policy's performance against the history of previous attempts.
   • Using this feedback and the history of best policies, the LLM generates a new policy for the next round.
   • If code execution errors occur, the LLM is prompted to re-implement the relevant code artifact.

The authors use Gemini 3 Pro for all experiments. The method relies on the LLM's ability to synthesize code that implements various optimization strategies, ranging from rule-based plans to planning algorithms like value iteration, without explicit prompting to use them.

Key Contributions

• Framework Introduction: The proposal of PromptPO as a method to treat LLMs as policy optimizers that output executable code rather than direct actions or parameter vectors.
• Empirical Benchmarking: A comprehensive evaluation across four distinct domains: hard-exploration environments (NoiseWorld, Point Maze), robotic manipulation (Meta-World), continuous control (MuJoCo), and real-world control tasks (Pandemic Mitigation, Glucose Monitoring, Traffic Control).
• Analysis of Sufficient Conditions: The identification of scenarios where LLM-based optimization is sufficient versus where it fails, specifically linking success to the ability to leverage prior knowledge about the environment or optimization strategy.

Experimental Results
The study compares PromptPO against standard model-free RL baselines (PPO, SAC, DQN/QR-DQN, TD3) across 19 environments:

• Exploration Environments: PromptPO matches or exceeds the performance of the best RL baselines in 15/19 environments overall. In NoiseWorld (randomly generated grid worlds) and Point Maze, PromptPO demonstrates strong exploration capabilities. Notably, in NoiseWorld, PromptPO often achieves optimal performance without needing environment feedback updates, suggesting it leverages priors to implement planning algorithms (e.g., Value Iteration, Dijkstra) directly in the generated code.
• Robotics (Meta-World vs. MuJoCo):
  • Meta-World: PromptPO is significantly more sample-efficient than RL across all tasks and outperforms final RL performance in the "Pick and Place" task. The action space here involves end-effector displacements.
  • MuJoCo: PromptPO underperforms standard RL algorithms. The authors attribute this to the requirement for fine-grained continuous control (joint torques), which current LLMs struggle to reason about effectively.
• Real-World Control: In tasks involving pandemic lockdown regulations, insulin administration, and autonomous vehicle traffic control, PromptPO is substantially more sample-efficient (e.g., >60x in Pandemic Mitigation) and often achieves higher final returns than PPO within the same training budget.
• Policy Types: Without explicit instruction, PromptPO autonomously generates diverse policy types, including tuned proportional controllers, rule-based heuristics, and planning algorithms, adapting the strategy to the environment's nature.

Significance and Claims
The paper claims that LLM-based policy optimization is a sufficient and often superior alternative to standard RL for many sequential decision-making tasks, particularly those where the LLM can leverage prior knowledge about the environment or effective optimization strategies.

• Sample Efficiency: The primary advantage of PromptPO is its sample efficiency, derived from the LLM's pretraining on vast datasets that encode structural knowledge about optimization and control, reducing the need for extensive environment interaction.
• Accessibility: PromptPO lowers the barrier to entry for RL, as it requires fewer hyperparameters and algorithmic design decisions than standard methods, making it accessible to practitioners without deep RL expertise.
• Limitations: The authors modestly acknowledge that the method currently struggles in environments requiring fine-grained continuous control (like MuJoCo joint torques) or where the reward function is a non-interpretable neural network (e.g., learned via RLHF) rather than a Python function.
• Generalization: The results suggest that the success of PromptPO may stem less from specific knowledge of the environment (as evidenced by success in randomly generated NoiseWorlds) and more from a prior over effective optimization strategies.

The paper concludes that while LLMs may not yet surpass human capabilities in all domains, they represent a practical and powerful tool for generating high-performing policies in a wide range of settings, potentially facilitating broader adoption of RL in real-world applications.