Technical Summary: Environment-Grounded Automated Prompt Optimization for LLM Game Agents

Problem Statement

Large Language Model (LLM) agents in interactive environments are highly sensitive to their prompts, yet prompt engineering remains a manual, task-specific process. While LLMs serve as powerful reasoning engines, they often struggle with spatial interpretation, long-term planning, and adaptation in partial-information settings like interactive games. Existing solutions either rely on hand-crafted, domain-specific architectures that require manual adaptation for new domains or involve costly model fine-tuning. Furthermore, current prompt optimization methods often rely on static datasets, which can lead to overfitting and fail to capture the non-stationary nature of interactive environments. The core challenge is to automatically derive task-specific prompts that maximize agent performance without updating model weights or requiring extensive human supervision.

Methodology: RAPOA

The authors propose RAPOA (Reward-driven Automatic Prompt Optimisation for Agentic systems), a framework that combines a decomposed agent architecture with an evolutionary prompt optimization loop guided by environment returns.

1. Agentic Architecture: The SPA Agent

Instead of a monolithic agent handling state parsing, strategy, and action selection simultaneously, RAPOA distributes these tasks across two sub-agents:

• Descriptor Agent ($A_{des}$): A perception module that conditions the raw environment observation into a goal-relevant summary. It extracts task-relevant information without the burden of planning.
• Action Selection Agent ($A_{act}$): A planning module that takes the mission, the descriptor's summary, and the agent's current state to select the next action.
  This "Split Perception Action" (SPA) architecture mimics the actor-critic separation found in Reinforcement Learning (RL), allowing for more targeted prompt optimization.

2. The Optimization Loop

RAPOA operates via an iterative evolutionary process:

• Rollout: The agent interacts with the environment (BabyAI tasks) to generate trajectories (episodes) of observations, actions, and rewards.
• Behavior Analyzer (BA): This module analyzes failed and successful trajectories to attribute outcomes to specific prompt components. It identifies failure modes (e.g., "Actions do not consider collision") and successful behaviors, ranking them by severity.
• Mutator: Based on the BA's analysis, the Mutator proposes targeted revisions to the sub-agent prompts. It ensures changes are minimal, coherent, and respect the architectural boundary (e.g., the Descriptor does not reason about task completion).
• Two-Stage Evaluation: Proposed mutations undergo a strict acceptance test:
  1. In-loop Optimization: The candidate must improve performance over the current prompt on a set of optimization seeds ($S_{opt}$) by a threshold $\delta$.
  2. Held-out Selection: The candidate must also improve performance on a separate, fixed set of selection seeds ($S_{select}$) to ensure generalization and prevent overfitting to specific trajectories.

3. Key Components

• Environment Grounding: Unlike methods using LLM-as-a-judge, RAPOA uses actual environment rewards as the optimization signal, making the signal non-stationary and harder to overfit.
• Severity-Ordered Adaptation: The system prioritizes fixing the most severe failure modes first, adapting sub-agents sequentially based on the BA's ranked list.

Key Contributions

1. Automated Prompt Optimization Framework: A system that optimizes both monolithic and multi-agent (decomposed) LLM agents automatically, removing the need for manual prompt engineering.
2. Joint Prompt Optimization Procedure: A method that leverages episode returns and behavior analysis of state transitions to refine prompts for multiple sub-agents simultaneously.
3. Decomposition and Optimization Synergy: The demonstration that splitting the agent into perception and action modules, combined with automated prompt tuning, significantly outperforms monolithic baselines without requiring gradient updates or weight fine-tuning.

Experimental Results

The framework was evaluated on the BALROG benchmark, specifically on five BabyAI task families: GOTO, PICKUP, OPEN, PUTNEXT, and PICKUPSEQGOTO. The experiments compared the RAPOA-optimized SPA agent against the BALROG baseline (RobustCoTAgent) under both "Plain" (minimal) and "Guided" (task-specific) prompt initializations.

• Performance Gains: Optimization consistently improved performance across all tasks and conditions.
  • On the PUTNEXT task (a complex multi-step coordination task), the baseline RobustCoTAgent achieved 0% success. The RAPOA framework, using the same underlying LLM (GPT-OSS-20B), achieved up to 72.5% success rate with optimized prompts.
  • For the SPA agent with guided initialization, the mean success rate across all five tasks increased from 65.5% to 79.2%.
  • Even with plain (minimal) prompts, the SPA agent matched or exceeded the performance of the guided BALROG baseline, demonstrating the robustness of the decomposition.
• Generalization: Optimized prompts for complex tasks (like PUTNEXT) showed strong transferability to simpler tasks (like PICKUP), whereas prompts optimized for isolated tasks (like OPEN) did not generalize well, suggesting the optimizer learns universal sub-skills for complex coordination.
• Ablation Studies:
  • Acceptance Criteria: Strict acceptance gates (requiring performance improvement) were found to be crucial. Without them (always-accept), the optimizer tended to drift away from optimal states, particularly on sparse-reward tasks.
  • Selection Pressure: Low selection pressure allowed for more exploration in sparse reward landscapes, often yielding better results on difficult tasks like PUTNEXT.
  • Module Constraints: When the optimizer was restricted to mutating only the Action Selection agent, it still outperformed the monolithic baseline, though the full decomposed system performed best.

Significance and Claims

The paper claims that RAPOA offers a novel and automated way to adapt LLM agents to new tasks without the computational cost of fine-tuning or the labor of manual redesign. The results suggest that:

• Decomposition is effective: Separating state interpretation from planning makes the optimization process more tractable and effective.
• Environment-grounded optimization is superior: Using environment rewards rather than static datasets or LLM judges allows for the discovery of robust, generalizable strategies.
• Prompt optimization is a viable alternative to fine-tuning: Significant performance gains (e.g., from 0% to 72.5% on PUTNEXT) can be achieved solely by refining the prompt, keeping the model weights fixed.

The authors maintain modesty regarding the scope, noting that experiments are currently restricted to a single domain (BabyAI) and one LLM. They identify future work in testing generalization across diverse environments, task distributions, and larger meta-LLMs, as well as exploring population-based optimization variants to reduce premature convergence. They also highlight procedurally generated game environments as a promising testbed for avoiding test-set contamination in prompt optimization research.