Technical Summary: Multi2 – Hierarchical Multi-Agent Decision-Making with LLM-Based Agents

1. Problem Statement

The central challenge addressed by this work is the fragility of Large Language Model (LLM) agents in long-horizon interactive environments. While recent LLM-based agents demonstrate strong contextual reasoning, they struggle with sustained interaction, primarily suffering from objective drift. As interactions extend over multiple turns, agents often deviate from their original goals due to the accumulation of small execution errors and the misalignment of their internal state with the initial intent.

Furthermore, existing approaches often rely on ever-growing interaction histories to maintain context, leading to token inefficiency. While hierarchical decomposition (splitting tasks into sub-goals) has been proposed to mitigate planning failures, current methods often fail to ensure stable execution. Specifically, the "executor" component in many hierarchical systems is not explicitly trained to correct compounding errors during interaction, leaving the system vulnerable to loops, constraint violations, and performance degradation as task horizons lengthen.

2. Methodology: The Multi2 Framework

The authors propose Multi2, a hierarchical multi-agent decision-making framework that explicitly decouples task decomposition (planning) from action execution through role specialization. The framework operates on two distinct systems:

• System 1 (High-Level Agent): Focuses on context-aware sub-goal generation. It takes a task description and current observation as input and outputs a sub-goal. This system is trained via Supervised Fine-Tuning (SFT) to ensure consistent, structured semantic planning.
• System 2 (Low-Level Agent): Responsible for executing atomic actions in the dynamic environment to achieve the sub-goal provided by System 1. This system is trained via an Offline-to-Online Reinforcement Learning (RL) pipeline.

Training Pipeline

The training process is divided into three stages to ensure stability and continuous improvement:

1. Offline Training (System 1): System 1 is fine-tuned using behavior cloning (SFT) on a high-level dataset ($D_{sys1}$) containing task descriptions and sequences of observation-sub-goal pairs. The objective is to minimize cross-entropy loss to imitate expert sub-goals.
2. Offline Training (System 2): System 2 is initialized using offline RL on a low-level dataset ($D_{sys2}$) containing transitions (observation, action, reward, next observation) conditioned on sub-goals. The authors employ an actor-critic structure with a novel loss design:
   • Critic Training: Minimizes Temporal-Difference (TD) error using in-distribution (ID) value learning to stabilize training.
   • Actor Training: Uses a policy-anchored advantage term. This combines imitation of offline actions with a weighted advantage term that encourages actions estimated to yield higher returns, mitigating over-imitation and templated behaviors.
3. Online Training (System 2 Self-Improvement): To enable continual adaptation, System 2 undergoes online RL refinement. A new policy ($\pi_{on}$) is updated using a replay buffer mixing offline data and new online trajectories. The loss function includes:
   • A weighted log-likelihood term reinforcing high-advantage actions.
   • A Kullback-Leibler (KL) regularization term that constrains the online policy to stay close to the offline-trained policy ($\pi_{off}$). This prevents mode collapse and sharp policy shifts, ensuring stable self-improvement.

Architecture

Multi2 utilizes a shared pretrained LLM backbone (e.g., Qwen-2.5, Mistral, Llama-3.1) with Low-Rank Adaptation (LoRA). Crucially, System 1 and System 2 use separate LoRA adapters, allowing for role-specific parameterization without updating the full backbone.

3. Key Contributions

1. Multi2 Framework: A role-specialized hierarchical framework that decouples sub-goal planning (System 1) and action execution (System 2) as distinct optimization problems, addressing the instability of long-horizon interaction.
2. Principled Offline-to-Online RL Objective: A novel training pipeline for the executor (System 2) that combines policy-anchored offline learning with KL-regularized online refinement. This design explicitly aligns with the executor's role to enable stable, sample-efficient self-improvement.
3. Hierarchical Benchmark Datasets: The release of three well-structured hierarchical datasets for training and evaluating interactive LLM agents across multiple environments (ScienceWorld, ALFWorld, TextCraft), filling a gap in the availability of consistent hierarchical data.
4. Empirical Validation: Demonstration that Multi2 consistently outperforms strong baselines in performance, token efficiency, and robustness to objective drift.

4. Experimental Results

The framework was evaluated on ScienceWorld, ALFWorld, and TextCraft using various backbone models (Qwen-2.5 3B, Mistral 7B, Llama-3.1 8B).

• Performance: Multi2 achieved the highest success rates across most backbones and evaluation splits (In-Distribution and Out-of-Distribution). For instance, on the ScienceWorld ID split with Llama-3.1 8B, Multi2 achieved a 67.61% success rate compared to 60.48% for the hierarchical baseline Glider and 23.23% for the prompt-based ReAct.
• Robustness to Objective Drift: Multi2 demonstrated superior performance on harder tasks requiring longer horizons. While prompt-based agents degraded sharply with task difficulty, and other fine-tuned baselines showed drops from medium to hard tasks, Multi2 maintained consistent performance, indicating effective mitigation of objective drift.
• Token Efficiency: Multi2 was significantly more token-efficient than baselines. By using hierarchical, on-demand invocation and RL-tuned action generation, it achieved higher performance with fewer inference-time tokens compared to prompt-based methods that rely on long context windows and in-context demonstrations.
• Ablation Studies:
  • Role Specialization: Swapping training paradigms (training System 1 with RL and System 2 with SFT) resulted in the worst performance, confirming the necessity of SFT for planning and RL for execution.
  • Architecture: The hierarchical structure with role-specific adapters outperformed single-model and shared-adapter baselines.
  • Loss Design: The proposed policy-anchored offline loss and KL-regularized online loss were critical for reducing variance and preventing low-performance failures compared to vanilla IQL or AWAC variants.

5. Significance and Claims

The paper claims that Multi2 provides a reliable and efficient solution for building interactive LLM-based agents capable of solving long-horizon tasks. Its significance lies in:

• Addressing Fundamental Limitations: It moves beyond simple hierarchical decomposition by explicitly training the execution layer to correct compounding errors, a gap identified in prior works like Glider.
• Stability in Interaction: The combination of SFT for intent preservation and KL-regularized RL for execution refinement offers a principled mechanism for stable coordination over extended interactions.
• Resource Efficiency: The framework achieves state-of-the-art performance while reducing token consumption, making it more practical for real-world deployment.
• Community Resource: By releasing hierarchical datasets and code, the authors aim to facilitate broader adoption and fair comparison in the development of hierarchical decision-making for LLM agents.

The authors position this work as a step toward agentic AI that can maintain intent and adapt effectively in dynamic environments, contributing to more effective learning algorithms for interactive systems.