🤖 AI

Training the Orchestrator: A Supervised Approach to End-to-End PDDL Planning with LLM Agents

This paper introduces HALO, a hybrid agent-learned orchestrator that leverages verifier-certified refinement trajectories to train a small, cost-efficient policy for end-to-end PDDL planning, achieving success rates comparable to frontier LLMs while reducing orchestration costs by over an order of magnitude.

Original authors: Rajesh Mangannavar, Zachary Coalson, Pranay Dugar, Prasad Tadepalli

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rajesh Mangannavar, Zachary Coalson, Pranay Dugar, Prasad Tadepalli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Translator" Bottleneck

Imagine you want to build a complex piece of furniture. You have a rough sketch and a list of ideas written in plain English (Natural Language). However, the machine that actually builds the furniture (the Classical Planner) only understands a very strict, robotic code called PDDL.

If you ask a regular AI (a Large Language Model or LLM) to write that code, it often makes mistakes. It might invent parts that don't exist, forget to list the screws, or write instructions that the machine can't read.

To fix this, researchers created a "repair team." They have a Manager (the Orchestrator) who looks at the broken code and hires a team of Specialists (Agents) to fix specific errors. One specialist fixes grammar, another fixes logic, and another checks if the parts fit together.

The Catch: In the old system, the Manager was a super-expensive, high-end AI (like GPT-5 or Gemini). Every time the team needed to fix a mistake, you had to pay the Manager a huge fee to think about the problem and pick the next specialist. If a project took 10 steps to fix, you paid the Manager 10 times. This made the whole process too expensive for small labs or local computers.

The Solution: HALO (The Intern Manager)

The authors of this paper, Rajesh Mangannavar and his team, asked: "Do we really need a super-expensive Manager for every single step? Can we train a cheap, local intern to do the job?"

They created HALO (Hybrid Agent-Learned Orchestrator). Here is how it works, using three key tricks:

1. Learning from "Winning" Playbooks

Usually, training an AI is hard because you don't know the "right" answer for every step. But in this system, there is a Referee (a Verifier) that checks the final furniture.

If the final plan works, the Referee says, "Good job!"
The team realized: Every time the Referee said "Good job," the sequence of Managers and Specialists they used was a winning strategy.

They took thousands of these "winning playbooks," stripped away the expensive Manager's voice, and just kept the record of: "When the code looked like X, the winning team picked Specialist Y." They used this data to train a small, cheap AI (HALO) to mimic those winning decisions.

2. The "Rulebook" Shortcut

HALO isn't just a brain; it's a brain with a cheat sheet. The authors realized that some decisions are so obvious that even a human wouldn't need to think about them.

Example: If the code is empty, the rule is "Call the Emergency Agent."
Example: If the code has a spelling error, the rule is "Call the Syntax Agent."
Example: If the plan is perfect, the rule is "Stop."

HALO checks these simple rules first. If a rule applies, it acts immediately without using its "brain" (the AI model). This saves time and money. Only when the problem is truly confusing does HALO use its trained AI to make a decision.

3. A Bigger Toolbox

The team also expanded the team of Specialists. They added 8 new experts to the original 13, bringing the total to 21. Some of these new experts are "deterministic," meaning they are simple computer scripts that run instantly and cost nothing, rather than expensive AI models. This gave HALO more tools to fix problems quickly.

The Results: Faster, Cheaper, and Just as Good

The team tested HALO against the expensive, high-end AI Managers on 11 different types of planning problems (like logistics, robot movement, and scheduling).

Success Rate: HALO was just as good as, or sometimes even better than, the expensive AI. It solved the problems at the same rate.
Speed: Because HALO uses simple rules for easy problems and a small local model for hard ones, it makes decisions much faster.
Cost: This is the biggest win.
- The old way (using GPT-5) cost about $0.18 per task.
- HALO costs about $0.004 per task.
- That is roughly 45 times cheaper. It's like going from hiring a celebrity chef for every meal to having a highly trained local cook who uses a recipe book for simple dishes.

The Bottom Line

The paper proves that you don't need a "super-intelligent" AI to manage a team of repair agents. If you have a strict referee (the Verifier) that tells you when a plan is successful, you can train a small, cheap AI to learn from those successes.

This allows complex planning systems to run on a single computer in a lab (or even a laptop) without needing to pay for expensive cloud APIs every time they make a decision. It turns a luxury service into something anyone can run locally.

Technical Summary: HALO – A Supervised Approach to End-to-End PDDL Planning with LLM Agents

1. Problem Statement

Automated planning requires translating natural-language goals into formal Planning Domain Definition Language (PDDL) specifications, which classical solvers can process. While Large Language Models (LLMs) offer a bridge between natural language and formal representations, they are unreliable when used in isolation, often hallucinating predicates, mis-encoding constraints, or producing invalid plans.

Recent agentic frameworks (e.g., La Malfa et al. [2026]) address this by orchestrating a pool of specialized repair agents within a verifier-checked refinement loop. In these systems, a "frontier" LLM (e.g., GPT-5, Gemini-3) acts as the orchestrator, inspecting the current state (PDDL, solver logs, validator errors) and selecting the appropriate repair agent at every step. While effective, this approach creates a significant bottleneck: the orchestrator itself requires a costly API call to a frontier LLM at every refinement step. This dependency limits deployment to scenarios with continuous API access, making the framework prohibitively expensive for lab-scale, edge, or air-gapped environments.

The core challenge addressed by this paper is: Can the frontier-LLM orchestrator be replaced by a small, locally-served model without sacrificing planning success rates?

2. Methodology: HALO

The authors propose HALO (Hybrid Agent-Learned Orchestrator), a system that replaces the prompted frontier orchestrator with a small, QLoRA-tuned policy paired with a hardcoded rule layer.

2.1 The Training Signal: Verifier-Guided Trajectories

A key insight of the paper is that the external verifier (compiler and validator) already provides a strong supervision signal. In the agentic framework, a refinement trajectory is a sequence of (state, agent) decisions. If the trajectory ends in a valid plan (certified by the verifier), every decision within that trajectory is implicitly "correct" in the sense that it contributed to a successful outcome.

Data Collection: The authors run a strong prompted teacher (GPT-5-mini) on training problems across 11 PDDL domains.
Filtering: Trajectories are filtered strictly by the external verifier. Only those ending in a valid plan are retained.
Supervision: The training set consists of (state, agent) pairs extracted from these verified trajectories. The policy is trained via supervised fine-tuning (SFT) to predict the agent ID that the teacher selected, conditioned on the state.

2.2 Expanded Action Space

The framework expands the agent pool from the original 13 agents to 21 agents, categorized as:

Baseline Agents (13): LLM-prompted repair routines (e.g., syntax fixer, hallucination stripper) from prior work.
Deterministic Plan-Repair Agents (5): Non-LLM routines (e.g., variable swapping, action reordering, redundancy stripping) that run in milliseconds with zero LLM cost.
PDDL-Aware LLM Agents (3): Specialized agents for modeling errors (e.g., type hierarchy fixes, predicate generalization).

2.3 Hybrid Inference Policy

At inference time, HALO operates as a two-layer policy ( $\pi_{hybrid}$ ):

Layer 1 (Hardcoded Rules): Three trivially decidable cases are resolved by rules without invoking a model:
- Cold start (empty PDDL) $\rightarrow$ Select AgentEmergency.
- Syntax errors detected $\rightarrow$ Select AgentSyntaxPDDL.
- Valid plan reached $\rightarrow$ Select NoOpAgent (terminate).
Layer 2 (Learned Policy): If no rule applies, a small, QLoRA-tuned LLM (e.g., Llama-3-8B) predicts the next agent ID. The model is fine-tuned to output a single token corresponding to the agent ID (0–20).

2.4 Training Details

Architecture: QLoRA fine-tuning (4-bit quantization, LoRA rank 16) on base models like Llama-3-8B, Qwen-2.5-7B, and Gemma-2-9B.
Loss Function: Single-token cross-entropy on the agent ID, with prompt tokens masked.
Data Volume: Approximately 12,000–15,000 (state, agent) pairs derived from ~2,000 high-quality, verifier-accepted trajectories.

3. Key Contributions

HALO (Hybrid Agent-Learned Orchestrator): A trained orchestrator that replaces the frontier-LLM API call with a local, small-model inference. It matches or exceeds the performance of GPT-5-mini and Gemini-3-Flash baselines while reducing orchestration costs by more than an order of magnitude.
Verifier-Guided Trajectory Supervision: The authors formulate orchestrator training as imitation learning on trajectories certified by an external compiler/validator. The verifier acts as a data-acceptance filter rather than a loss function, providing a strong, noise-free supervision signal that generalizes across domains.
Expanded Action Space with Hybrid Policy: The introduction of 21 agents (including deterministic repair routines) and a hybrid policy that offloads trivial decisions to hardcoded rules. This allows the learned policy to focus its capacity on genuinely ambiguous decisions, improving efficiency and reducing the training burden.

4. Experimental Results

The system was evaluated across 11 PDDL domains spanning PlanBench, Google Natural Plan, and Classical Planning benchmarks.

Success Rate:
- HALO matches or exceeds the GPT-5-mini prompted baseline across all families (e.g., +3.3% on PlanBench, +37.3% on Natural Plan).
- HALO sits within 3 percentage points of the stronger Gemini-3-Flash baseline on PlanBench and Natural Plan, and exceeds it on Classical planning (95.8% vs 93.0%).
Efficiency & Cost:
- LLM Calls: HALO reduces total LLM calls per episode by 40–50% compared to baselines, primarily due to the hybrid rule layer resolving ~35% of decisions without a model call.
- Cost: Orchestration cost drops from ~ $0.18–$ 0.22 per task (GPT-5-mini) and ~ $0.06–$ 0.08 (Gemini-3-Flash) to $0.004 per task for HALO. This represents a 45× reduction against GPT-5-mini and a 15× reduction against Gemini-3-Flash.
Generalization: The model generalizes to held-out domains with a performance drop of only ~5–8 percentage points, suggesting the learned signal captures domain-general orchestration patterns rather than memorizing specific shortcuts.

5. Significance and Claims

The paper claims that agentic frameworks built around formal verifiers admit a structural simplification: the orchestrator does not need to be a generalist frontier LLM. Instead, it can be a small, locally-served, trainable model, provided the verifier provides per-trajectory acceptance signals.

This finding has significant implications for deployment:

Decoupling from APIs: It removes the dependency on continuous access to expensive frontier-LLM APIs.
Accessibility: It makes end-to-end natural-language planning feasible for lab-scale, edge, and air-gapped settings where API access is impractical or prohibited.
Cost Efficiency: It demonstrates that high-quality planning can be achieved at a fraction of the cost (roughly 45× cheaper) by leveraging the "free" supervision inherent in the verification loop.

The authors note limitations, including the reliance on the teacher's behavior (bounded by the verifier-filtered trajectories) and the inability to handle PDDL extensions outside the 21-agent scope (e.g., numeric fluents). However, they posit that this supervised approach serves as a strong baseline, with future work potentially exploring Reinforcement Learning from Verifiable Rewards (RLVR) to break the "teacher ceiling."