Technical Summary: QueenBee Planner

Problem Statement

The design paradigm for Large Language Model (LLM) multi-agent systems has shifted from optimizing individual agent reasoning (e.g., prompting, chains of thought) to optimizing the organization of interacting agents. While recent work has established that communication topology is a first-class determinant of system accuracy, cost, and robustness, existing approaches share a critical limitation: they typically design a topology for a specific task and then stop. They fail to accumulate reusable design knowledge about communication structures across tasks. Furthermore, current methods often conflate task-solving behavior with architectural design, or they optimize routing without storing explicit topology lessons.

The core challenge addressed by this paper is preventing a self-improving design loop from turning noise (lucky runs, plausible but false explanations) into policy. A trustworthy self-evolving architect requires explicit discipline to ensure that improvements in communication structure are genuine, reproducible, and distinct from the capabilities of the underlying worker agents.

Methodology

The paper introduces QueenBee Planner, a framework that treats inter-agent communication topology as a retrievable, self-improving design skill. The system operates under a strict separation of concerns:

1. Planner-Worker Separation

• Frozen Workers: A pool of worker agents, a task adapter, and a scoring function remain fixed throughout the learning process. Workers only observe private shards of a task and update their belief states based on received messages.
• Generative Planner: An outer LLM acts as an architect. Its sole learning objective is to generate temporal communication Directed Acyclic Graphs (DAGs). These DAGs specify who sends information to whom, in which round, who merges messages, and who emits the final answer.

2. Temporal Communication DAG

The planner generates a time-unrolled graph $G = (N, T, E, h, I, R)$, where:

• $N$ is the number of workers.
• $T$ is the number of communication rounds.
• $E$ represents directed temporal edges (message flows).
• $h$ is the final answer holder.
• The graph is constrained by feasibility limits on rounds ($T_{max}$), total edges ($M$), and maximum fan-in ($\beta$) to control token costs and merge complexity.

3. Skill Bank and Design Rules

The planner's memory is a Skill Bank containing evidence-backed design rules. Each skill $s = (z, r, a, q, E)$ consists of:

• Condition ($z$): Task features (e.g., agent count, budget).
• Rule ($r$): Structural pattern (e.g., "staged reduction," "audit edge").
• Action ($a$): One of three actions:
  • Preserve: Keep the structure as an executable scaffold.
  • Modify: Adapt the core structure for slightly different task conditions.
  • Avoid: Treat the structure as a negative constraint due to past failures.
• Evidence ($E$): Supporting data and counterexamples.

4. Disciplined Self-Evolution Loop

To prevent drift and overfitting, the system employs several rigorous mechanisms before updating the Skill Bank:

• Motif-Level Credit Assignment: Instead of assigning credit to entire graphs, the system decomposes graphs into structural motifs (e.g., fan-in limits, reduction depth). Credit is assigned to these motifs with a variance-aware penalty to prevent lucky single-run successes from dominating.
• Held-Out Acceptance Gate: Updates ($\Delta S$) are only admitted to the long-term Skill Bank if they demonstrably improve performance on held-out tasks and seeds, not just training data.
• Variance-Aware Credit: Rules require a minimum sample count and conservative loss estimates to ensure stability.
• Insight Falsification: Natural language insights proposed by the planner are treated as hypotheses and must pass falsification checks on held-out evidence before becoming skills.
• Transfer Trust: Skills are not automatically applied to all tasks; they are deployed only if there is direct or indirect success evidence for the specific task slot.
• Structural Deduplication: Isomorphic graph specifications are merged to prevent the Skill Bank from growing linearly with execution rounds.

Key Contributions

1. Reframing Topology as a Skill: The paper recasts multi-agent communication topology as a retrievable, self-improving design skill, separating the "architect" (planner) from the "workers." The learned object is communication design knowledge, not task-solving behavior.
2. Mechanism for Disciplined Self-Improvement: It introduces a comprehensive framework for generating temporal DAGs conditioned on a skill bank, featuring held-out gates, motif-level attribution, variance-aware credit, and insight falsification to ensure trustworthy evolution.
3. Empirical Claim on Architectural Learning: The authors demonstrate that under these disciplines, a planner can improve its generated communication structures on held-out tasks while the worker pool remains frozen, proving the system learns to design organizations rather than merely memorizing answers.

Experimental Results

The method was evaluated on two task families: Count-Frequency (CF) aggregation and Silo-Bench-style distributed coordination.

• Count-Frequency (CF) Task:

  • In the DeepSeek-V4-Flash fulltest setting, the self-evolved best-so-far graph (a hybrid reduce-audit structure) reduced RMSE from 12.53 (strongest fixed topology) to 7.87.
  • This improvement came with significant efficiency gains: a 74.2% reduction in messages, 54.5% reduction in model calls, and 59.6% reduction in token cost compared to the best fixed topology.
  • Improvements were consistent across various worker models (DeepSeek, Qwen, GPT-4o, MiMo), with RMSE reductions ranging from 14% to 56%.

• Silo-Bench-Style Tasks:

  • Self-evolved graph generation achieved an Exact Match (EM) of 0.539 (after 3 rounds), outperforming cold generation (0.125), the best single fixed topology (0.333), and even a per-case oracle fixed topology (0.417).
  • This suggests the system learns fine-grained, task-conditioned structures rather than simply selecting from a fixed set of topologies.

• Evolution Trajectory:

  • Round-by-round analysis showed that while exploration introduces noise (fluctuating RMSE), the retained "best-so-far" archive consistently improves, demonstrating that the system accumulates reusable design knowledge (e.g., staged reduction, sink coverage) over time.

Significance and Claims

The paper claims that the architecture of a multi-agent system is not merely an implementation detail but an explicit object of learning. QueenBee Planner demonstrates that multi-agent systems can learn reusable architectural design knowledge (when to preserve, modify, or avoid communication patterns) rather than just memorizing task answers.

The authors remain modest regarding the scope, noting that the current results are based on specific task families (CF and Silo-style) and a limited number of evolution rounds. They posit that the system does not discover a single universally optimal topology but accumulates transferable priors. The primary contribution is the validation that a frozen-worker system can improve its performance solely by evolving its communication design policy through disciplined, evidence-backed self-reflection.