Technical Summary: A Complementary Study on PlanGPT

Problem Statement

Automated Planning, a subfield of Artificial Intelligence, involves generating sequences of actions (plans) to transition from an initial state to a goal state, typically formalized using the Planning Domain Definition Language (PDDL). While traditional planners have long been the standard, recent research has explored the use of Large Language Models (LLMs) for this task. PlanGPT, a state-of-the-art LLM built from scratch and inspired by GPT-2, was proposed as a planner capable of learning general policies to solve planning problems across specific domains.

However, the original PlanGPT paper presented limitations in its evaluation. It primarily focused on "plan coverage" (the percentage of valid plans generated) without providing comprehensive performance metrics such as plan cost (solution quality) or plan generation time. Furthermore, the specific International Planning Competition (IPC) benchmark problems used were not explicitly detailed, and the comparison was limited to other LLMs and Graph Neural Networks rather than traditional, optimal planners. This study addresses the question of whether planning with LLMs is pertinent and worthwhile, given the significant computational resources required, by conducting a rigorous, complementary evaluation of PlanGPT against traditional planning strategies.

Methodology

The authors conducted a comprehensive re-evaluation of PlanGPT using the IPC benchmark problems across seven domains: BlocksWorld, Depot, Driverlog, Logistics, Satellite, Zenotravel, and Floortile (Visitall was excluded). The experimental setup involved three distinct strategies:

1. PlanGPT: The LLM-based planner, which predicts tokens corresponding to actions and objects to generate a plan.
2. FastDownward (A):* A traditional optimal planner using the A* search heuristic, which guarantees optimal plan cost but may be computationally expensive.
3. FastDownward (Greedy): A traditional planner using a Greedy search heuristic, which prioritizes speed over optimality.

Experimental Procedure:

• Input Processing: PDDL problem files were normalized using domain-dependent shell scripts. PlanGPT tokenized "fluents" (predicates and objects) from the :init and :goal sections, predicting action tokens sequentially until an <end> token was generated.
• Validation: Generated plans were validated using the official VAL validator to determine plan coverage.
• Metrics: The study evaluated three key metrics:
  • Plan Coverage: The percentage of problems for which a valid plan was generated.
  • Plan Cost: The total number of actions in the generated plan.
  • Planning Time: The time required to generate the plan.
• Scoring: IPC normalized scores were computed for both cost and time. For cost, the strategy with the lowest cost for a problem received a score of 1, with others scaled proportionally. For time, a logarithmic formula was used relative to a 300-second timeout.

The experiments were conducted on the LIG Aker GPU cluster, though the authors note that a standard GPU (e.g., NVIDIA GTX 1080 TI) or even a CPU is sufficient for running PlanGPT.

Key Contributions

1. Verification of Plan Coverage: The study confirmed that PlanGPT's plan coverage results are largely consistent with those reported in the original paper, with a minor average difference of approximately 10% attributed to different parameter settings and the use of the IPC benchmark set.
2. Expanded Performance Evaluation: Unlike the original work, this paper introduces Plan Cost and Plan Generation Time as critical evaluation metrics, providing a more holistic view of PlanGPT's utility.
3. Direct Comparison with Traditional Planners: The study compares PlanGPT directly against FastDownward using both A* (optimal) and Greedy (fast) heuristics, offering a baseline for whether LLMs can compete with established planning algorithms.
4. Resource Analysis: The paper highlights the resource intensity of PlanGPT, noting that it requires a separate model for each of its 8 supported domains, raising questions about the efficiency of such an approach compared to general-purpose planners.

Results

The results indicate that PlanGPT's performance is mixed and generally does not surpass traditional planning strategies when considering the full scope of metrics:

• Plan Coverage: PlanGPT achieved an average coverage of 79% across the tested domains. This is comparable to the Greedy strategy (88%) but significantly lower than the optimal A* strategy in some domains (e.g., Logistics, where A* solved 71% vs. PlanGPT's 43%).
• Plan Cost:
  • In the BlocksWorld domain, PlanGPT performed well, often matching the optimal cost found by A* and outperforming the Greedy strategy.
  • In other domains like Driverlog, PlanGPT frequently failed to find valid plans. When it did succeed, the cost was often higher than that of the Greedy strategy.
  • Overall: The Greedy strategy generally outperformed PlanGPT in terms of total IPC cost score (5.36 vs. 5.02). While PlanGPT showed superiority in specific domains (BlocksWorld, Depot, Floortile), it was less efficient globally.
• Planning Time:
  • The Greedy strategy was consistently the fastest, achieving near-perfect IPC time scores (5.82) across most domains.
  • PlanGPT's generation time was variable; it was sometimes faster than A* but generally slower than Greedy.
  • Overall: Greedy obtained the highest total IPC time score (5.82), followed by PlanGPT (4.95), with A* scoring the lowest (2.46) due to timeout issues on complex problems.

Summary of IPC Scores (Total):

• Cost: Greedy (5.36) > PlanGPT (5.02) > A* (3.26)
• Time: Greedy (5.82) > PlanGPT (4.95) > A* (2.46)

Significance and Claims

The paper concludes with a modest assessment of the current state of LLMs in automated planning. The authors state that their findings do not definitively confirm or deny the hypothesis that LLMs are currently competent planners.

• Limitations: The study highlights that PlanGPT is currently no better than a Greedy search strategy in terms of overall efficiency and solution quality. The requirement for a separate model per domain and the fixed limits on object counts restrict its scalability and applicability to complex, real-world problems.
• Resource Efficiency: The authors question the "worth" of using LLMs for planning given the resources consumed. Even if an LLM were to outperform Greedy in specific metrics, the computational cost (including training time, which was not factored into the generation time metric) may render it less "competent" than traditional approaches.
• Future Directions: The paper suggests that while LLMs may not be the optimal solution for general planning at this time, the field requires further research. It posits that alternative architectures, such as Large Reasoning Models (LRMs), might be better suited for planning tasks than current LLMs.

Ultimately, the paper serves as a critical reality check for the Automated Planning community, demonstrating that while LLMs show promise in specific niches, they currently do not offer a general, resource-efficient alternative to traditional planners like FastDownward.