Technical Summary: HIPIF (Hierarchical Planning and Information Folding)

Problem Statement

While Large Language Models (LLMs) have demonstrated strong capabilities as autonomous agents in single-step or short-horizon tasks, their performance significantly degrades in multi-turn long-horizon agentic tasks. Existing approaches face two primary limitations:

1. Long-Context Interference: As interactions proceed, the observation-action history grows continuously. This accumulation of redundant information weakens the agent's ability to track the global task state, leading to reasoning degradation and decision-making failures.
2. Inadequacy of Existing Solutions:
   • Prompt-based and Behavior Cloning methods rely on hand-crafted prompts or expert trajectories without environmental feedback, limiting adaptability.
   • Reinforcement Learning (RL) methods (e.g., credit assignment, hierarchical RL) often rely on auxiliary models for task decomposition or process-reward annotation, increasing pipeline complexity. Crucially, they rarely train the model to actively organize and fold growing contexts, failing to fundamentally resolve state-tracking failures caused by long-context interference.

Methodology: HIPIF

The authors propose Hierarchical Planning and Information Folding (HIPIF), an end-to-end training framework designed to organize long-horizon execution around explicit subgoals while folding completed subgoal histories. The methodology consists of three core components:

1. End-to-End Training for Hierarchical Planning and Information Folding

Instead of conditioning the policy on the full accumulated trajectory ($\tau_t$), HIPIF restructures the context. The agent decomposes the task into explicit subgoals ($g_k$).

• Context Folding: Once a subgoal is judged complete, its execution history is "folded" into a compact record (e.g., $[g_k, o_{end}^k]$) and appended to a global history of completed subgoals ($H_{<k}$).
• Compact Working Context: At any decision step $j$ within current subgoal $g_k$, the policy conditions on a compressed context: $C_{k,j} = [c; H_{<k}; g_k; T_{k,j}]$, where $T_{k,j}$ is the detailed local history of the current subgoal.
• Optimization: The agent is trained using Group Relative Policy Optimization (GRPO), treating subgoal-centric decision-making as a learnable policy behavior optimized via environmental feedback.

2. Hierarchical Reflection

To stabilize the complex process of proposing subgoals, judging transitions, and executing actions, HIPIF introduces a hierarchical reflection mechanism embedded within the rollout:

• Completion Judgment: After each action, the model generates a reflection ($\eta_{k,t}$) to assess if the current subgoal is completed ($z_{k,t}=1$) or not ($z_{k,t}=0$).
• Branching Logic:
  • If completed: The model reflects on the folded history of previous subgoals to propose the next subgoal ($g_{k+1}$).
  • If not completed: The model reflects on the current subgoal and its immediate execution history to identify the next effective action, avoiding invalid attempts.
• This mechanism serves as a control loop for subgoal termination, transition, and execution without requiring external critics.

3. Subgoal-Oriented Process Rewards

To address sparse rewards in long-horizon tasks, HIPIF introduces fine-grained, rule-based process rewards that penalize specific errors at the step level:

• Subgoal Content Reward ($r_{gr}$): Penalizes subgoals referencing objects or locations absent from the current environment context.
• Subgoal Termination Reward ($r_{term}$): Penalizes subgoals where the terminal observation indicates execution failure (e.g., "Nothing happens"), even if the overall trajectory eventually succeeds.
• Subgoal Execution Reward ($r_{exec}$): Penalizes repeated action-observation pairs within the same subgoal (detecting loops) and enforces valid structured output formats.
• Training Signal: These local penalties are combined with the final task reward to compute step-level returns, which are normalized within a group of trajectories to derive advantages for GRPO.

Key Contributions

1. HIPIF Framework: A novel method that trains LLM agents end-to-end to organize long-horizon execution around explicit subgoals and fold completed histories, directly addressing long-context interference.
2. Stabilization Mechanisms: The introduction of hierarchical reflection and subgoal-oriented process rewards to guide subgoal generation, transition judgment, and execution correction without relying on costly auxiliary models or expert trajectories.
3. Empirical Validation: Extensive experiments demonstrating that HIPIF outperforms existing baselines in effectiveness and efficiency.

Experimental Results

The authors evaluated HIPIF on three publicly available agentic benchmarks: ALFWorld, VirtualHome, and ScienceWorld, using Qwen2.5-3B and Qwen2.5-7B base models.

• Performance: HIPIF achieved state-of-the-art results across all benchmarks.
  • On ALFWorld, HIPIF reached a 100.0% success rate on the PICK task and 96.1% on the complex PICK2 task, significantly outperforming strong baselines like GiGPO (93.8%) and STEP-HRL (92.9%).
  • On VirtualHome and ScienceWorld, HIPIF consistently achieved the highest success rates (e.g., 96.6% and 96.3% respectively on ALFWorld sub-tasks).
• Efficiency:
  • Token Usage: HIPIF demonstrated superior token efficiency, requiring fewer interaction steps and significantly fewer input tokens per trajectory compared to baselines. The "folding" mechanism prevents the accumulation of redundant context tokens.
  • Pipeline Efficiency: Unlike methods requiring expert trajectories (e.g., GLIDER, STEP-HRL) or auxiliary critic models (e.g., HiPER), HIPIF operates without these dependencies, making it lightweight and scalable.
• Ablation Studies:
  • Removing the subgoal structure caused the most significant performance drop, confirming that context folding is fundamental.
  • Removing hierarchical reflection degraded performance, highlighting its role in stabilizing subgoal transitions.
  • Removing process rewards led to noticeable drops, indicating the necessity of fine-grained feedback for reliable execution.

Significance and Claims

The paper claims that hierarchical planning and information folding offer a promising direction for improving both the effectiveness and efficiency of agentic decision-making. By explicitly training agents to decompose tasks and compress completed progress, HIPIF fundamentally mitigates the reasoning degradation caused by long-context interference.

The authors emphasize that their approach achieves superior performance without relying on task-specific expert trajectories or additional auxiliary models, addressing a key scalability bottleneck in existing hierarchical RL methods. The results suggest that learning to organize the decision process via subgoals is more critical for long-horizon tasks than simply increasing model scale or relying on advanced prompting alone.