💻 computer science

Any House Any Task: Scalable Long-Horizon Planning for Abstract Human Tasks

The paper proposes AHAT, a scalable long-horizon planning framework that leverages an LLM trained via a novel reinforcement learning algorithm (TGPO) to translate ambiguous human instructions and scene graphs into PDDL subgoals, enabling robots to generate optimal plans for complex household tasks across large environments.

Original authors: Zhihong Liu, Yang Li, Rengming Huang, Cewu Lu, Panpan Cai

Published 2026-08-04

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhihong Liu, Yang Li, Rengming Huang, Cewu Lu, Panpan Cai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to clean your messy room. You don't want to give it a step-by-step manual like "pick up the sock, then walk two steps, then bend down." That's too boring and too fragile; if the sock moves, the robot gets confused. Instead, you want to just say, "Make this room tidy," and have the robot figure out the rest. This is the holy grail of robotics: taking a vague, human wish and turning it into a perfect, physical action plan.

To do this, scientists often use two different tools. The first is a Large Language Model (LLM), which is like a super-smart, well-read librarian who knows a million facts about the world but sometimes makes up stories or forgets the rules of physics. The second is a Symbolic Planner, which is like a strict, unyielding math teacher who follows rules perfectly but has no idea what "tidy" means or how to talk to humans. The big challenge is getting the chatty librarian and the strict math teacher to work together without the librarian getting the robot stuck in a loop or the math teacher refusing to start because the instructions were too vague.

This is exactly the problem tackled in a new paper called TGPO (Trace-Guided Policy Optimization). The researchers are trying to teach a robot how to break down big, fuzzy human commands into a series of small, checkable steps that a robot can actually follow. They found that simply asking a smart AI to "just do it" often leads to plans that look good on paper but fail in the real world. Instead, they developed a clever training method where the AI learns to generate a "trace" of its thinking, gets corrected by a helper when it makes a mistake, and then tries again. It's like teaching a student not just by giving them a grade at the end, but by walking them through their homework, fixing their logic errors in real-time, and letting them practice until they get it right. The result is a system that is much better at planning long, complex tasks—like preparing a whole house for a party—than previous methods, especially when the instructions are vague.

The Problem: The "Magic Wand" That Breaks

Imagine you have a robot butler. You tell it, "I'm hosting a festival gathering, so clean up the house." A human understands this immediately: they know to pick up trash, wipe tables, and maybe set out decorations. But for a robot, this is a nightmare. If you ask a standard AI to just "write a plan," it might hallucinate. It might say, "Move the sofa to the kitchen," even if the sofa is too heavy, or "Turn on the stove to clean the floor," which is a terrible idea.

The paper explains that current AI models are great at guessing what comes next in a sentence, but they are terrible at ensuring their plans are feasible. They make small mistakes early on—like forgetting to pick up a cup before moving a table—and those mistakes pile up like a house of cards until the whole plan collapses. On the other hand, traditional robot planners are very safe; they won't do anything impossible. But they are also very dumb; they need you to tell them exactly what to do in a very specific code (called PDDL), and they can't understand the nuance of "clean up for a party."

The Solution: TGPO (The "Trace-Guided" Coach)

The authors propose a new way to train robots called Trace-Guided Policy Optimization (TGPO). Think of TGPO not as a robot that just guesses, but as a student learning with a very strict, helpful coach.

Here is how the training works, using a simple analogy:

The Student (The AI Policy): The robot's brain tries to break down your command ("Clean for the party") into a list of sub-goals. Maybe it says: "1. Pick up trash. 2. Wash dishes. 3. Vacuum."
The Coach (The Verifier): A strict checker looks at this list. It asks, "Wait, can you actually wash the dishes if the sink is full of food? Did you remember to move the chairs before vacuuming?"
The "Trace" Correction: If the student's plan is wrong, the Coach doesn't just say "Fail." Instead, it looks at the thinking process (the trace) and fixes the specific mistake. It might say, "You forgot to empty the sink first. Here is the corrected list of steps."
The Practice Loop: The robot then takes this corrected list and tries to generate the rest of the plan based on it. It learns from the correction, not just from the final "pass/fail" grade.

This is different from how most AI is trained today. Usually, you ask an AI to do something, and if it fails, you just tell it "bad job" and try again. That's like failing a math test and getting an "F" without seeing the red pen marks showing where you went wrong. TGPO is like getting the red pen marks while you are taking the test, so you can fix your logic before you hand it in.

What They Found: The Robot Gets Smarter

The researchers tested this new method on a huge variety of tasks, from simple ones like "bring me a towel" to incredibly complex ones like "prepare the house for a festival," which involves moving furniture, cleaning, and organizing in a specific order.

They compared their robot to two other types of planners:

The "Prompting" Robots: These are AI models that are just asked to write a plan. The paper found that as tasks got harder and more abstract, these robots failed miserably. They would get confused by the complexity and suggest impossible actions.
The "Standard Learning" Robots: These are robots trained with standard reinforcement learning (trial and error). They did better than the prompting robots but still struggled when the instructions were vague or the tasks were very long.

The Results:
The TGPO robot was the clear winner.

On easy tasks, it succeeded about 88% of the time.
On complex tasks (long chains of actions), it succeeded 77% of the time.
On abstract tasks (where the robot had to guess what "clean up" meant), it succeeded 70% of the time.

In comparison, the best "prompting" robot only succeeded about 22% of the time on those same abstract tasks. The paper suggests that TGPO's ability to fix its own thinking process in real-time is the key. It doesn't just guess; it reasons, checks, corrects, and then acts.

Why This Matters

The paper shows that by teaching robots to generate "verifiable subgoals" (small, checkable steps) and using a system that corrects their thinking traces, we can make them much more reliable in the real world. The authors note that this works even when the environment is messy or the instructions are vague.

However, they are careful to point out that this isn't a magic solution for every robot problem yet. The system still relies on a pre-defined map of the world (a "scene graph") and a set of rules the robot knows. It doesn't learn to see the world from scratch just by looking at a camera; it needs that structured map to do its planning. But for the specific job of turning human words into safe, executable robot actions, TGPO suggests a major step forward, proving that with the right kind of "coaching," AI can learn to plan much better than it does on its own.

Technical Summary: Trace-Guided Policy Optimization (TGPO) for Robot Task Planning

1. Problem Formulation

The paper addresses the challenge of long-horizon robot task planning in real-world environments. The core difficulty lies in mapping abstract, underspecified natural language instructions into executable action sequences that satisfy complex constraints over extended time horizons.

Current approaches face a dichotomy:

Large Language Models (LLMs): Possess strong commonsense reasoning but often fail to generate reliable, feasible plans due to a lack of explicit constraint modeling, compounding errors in autoregressive generation, and weak grounding to environment states.
Symbolic Planners: Guarantee feasibility and optimality but require well-specified symbolic goals (e.g., in Planning Domain Definition Language, PDDL) and cannot directly interpret high-level human intent.

The authors formulate the problem as learning to generate verifiable subgoals. Instead of generating a single holistic goal or a direct action sequence, the system learns a policy $\pi_\theta$ that decomposes an instruction $q$ and environment state $S$ into a sequence of intermediate PDDL subgoals $G = \{g_k\}$ . These subgoals are then solved by a classical symbolic planner to produce the final action plan. The learning objective is to maximize a reward function based on feasibility (solvable subgoals) and task completion (satisfying the instruction).

2. Methodology: Trace-Guided Policy Optimization (TGPO)

To address the challenges of sparse, noisy, and delayed rewards inherent in this learning problem, the authors propose Trace-Guided Policy Optimization (TGPO). This is a reinforcement learning (RL) framework built upon Group Relative Policy Optimization (GRPO) but enhanced with structured learning signals derived from intermediate reasoning traces.

TGPO operates through a three-component mechanism:

A. Verifier-Grounded Rewards

The system utilizes a verifier to provide outcome-level supervision. The reward $R(T, G)$ is a product of:

Feasibility ( $R_{feas}$ ): A binary check on whether all generated subgoals are solvable by the PDDL planner.
Task Completion ( $R_{comp}$ ): A score (0 to 1) evaluating whether the composed plan satisfies the user's intent, often assessed via an LLM-as-judge protocol or simulator goals.

B. Trace-Guided Refinement

Recognizing that scalar rewards are insufficient for credit assignment in long-horizon tasks, TGPO introduces an auxiliary model to correct intermediate reasoning.

Free Rollouts: The policy generates $N$ candidate trajectories (decomposition traces $T$ and subgoals $G$ ).
Trace Correction: Trajectories with low rewards (below a threshold $\tau$ ) are identified. An auxiliary model (the "Trace Improver") revises the decomposition trace $T$ to fix errors such as missing subtasks, incorrect temporal ordering, or incomplete grounding, producing a corrected trace $\tilde{T}$ .
Constrained Rollouts: The policy performs a second rollout conditioned on the corrected trace $\tilde{T}$ to generate the corresponding subgoals $\tilde{G}$ . This is implemented via token-level constrained sampling, forcing the trace tokens while autoregressively generating the subgoals.

C. Constrained Policy Optimization

The optimization process combines the original free rollouts and the trace-corrected rollouts into an augmented group. The policy is updated using a clipped objective similar to GRPO, but the gradient computation includes both the original and the corrected trajectories. This allows the policy to learn from the external corrections, effectively transforming sparse rewards into dense, structured supervision signals.

3. Experimental Setup and Benchmarks

The framework was instantiated for mobile manipulation in large-scale household environments.

Data Generation: A large-scale dataset was constructed using the Habitat Synthetic Scene Dataset (HSSD). Tasks were synthesized using LLMs conditioned on diverse user personas and scene graphs, ranging from explicit commands to abstract intent descriptions.
Benchmarks: Evaluation was conducted on four benchmarks:
1. In-Distribution Synthetic Tasks: Categorized as Easy, Complex (long-horizon, constraint-heavy), and Abstract (implicit intent).
2. Human Tasks: 50 real-world instructions collected via a web interface.
3. PARTNR: A public benchmark for embodied multi-agent tasks.
4. BEHAVIOR-1K: A benchmark for household task planning and skill learning.
Baselines: Comparisons included prompting-based planners (SayPlan, DELTA, LLM+P, NL2Plan, vanilla LLMs) and learning-based methods (SFT, GRPO, Reinforce++, and an End-to-End TGPO variant).

4. Key Results

Superior Performance: TGPO significantly outperformed all baselines across all benchmarks. On in-distribution tasks, improvements over the strongest prompting-based baseline increased with task complexity: 67% on Easy tasks, 131% on Complex tasks, and 213% on Abstract tasks.
Generalization: TGPO demonstrated robust generalization to out-of-distribution environments (PARTNR, BEHAVIOR-1K) and human-authored tasks, achieving success rates of 82.6% and 70.3% respectively, with substantial gains over baselines.
Ablation Studies:
- TGPO vs. SFT: RL optimization provided a higher performance ceiling than supervised fine-tuning alone, particularly for complex and abstract tasks.
- Trace Correction: Removing the trace correction mechanism (standard GRPO) led to rapid performance degradation as task complexity increased, highlighting the necessity of structured guidance for credit assignment.
- Subgoal vs. End-to-End: The subgoal-based approach (TGPO) significantly outperformed the End-to-End variant (TGPO E2E), confirming that separating problem formulation (subgoal generation) from execution (symbolic solving) reduces the learning burden and improves feasibility.
Robustness: The system maintained high success rates (62.3% on abstract tasks) even under stochastic dynamics (10% action failure probability) when coupled with online Monte Carlo Tree Search (MCTS).
System-Level Integration: In a full robotic pipeline with low-level executors, TGPO enabled compositional generalization, solving novel recombined tasks with a 26% end-to-end completion rate, matching the performance of holistic neural policies on their original training tasks but without relying on specific demonstrations.

5. Significance and Contributions

The paper claims three primary contributions:

Learning-Based Subgoal Formulation: It reframes long-horizon planning as learning a policy to generate verifiable subgoals, moving beyond static prompting-based decomposition.
Trace-Guided Policy Optimization: It introduces TGPO, an RL algorithm that leverages external trace correction to optimize structured generation under sparse and noisy rewards, solving the credit assignment problem in intermediate reasoning steps.
Scalable Long-Horizon Planning: It demonstrates that combining learned subgoal generation with symbolic planners enables robust planning in large-scale, complex environments with abstract instructions, significantly outperforming existing methods.

The authors note that while the method is effective, it currently relies on structured scene representations (scene graphs) rather than raw visual inputs and assumes a predefined symbolic domain. Future work is suggested to address learning/adapting domains and integrating perception uncertainty.