🤖 AI

Combee: Scaling Prompt Learning for Self-Improving Language Model Agents

The paper proposes Combee, a novel framework that scales parallel prompt learning for self-improving language model agents using parallel scans, an augmented shuffle mechanism, and a dynamic batch size controller to achieve up to 17x speedup while maintaining or improving accuracy across multiple benchmarks.

Original authors: Hanchen Li, Runyuan He, Qizheng Zhang, Changxiu Ji, Qiuyang Mang, Xiaokun Chen, Lakshya A Agrawal, Wei-Liang Liao, Eric Yang, Alvin Cheung, James Zou, Kunle Olukotun, Ion Stoica, Joseph E. Gonzalez

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hanchen Li, Runyuan He, Qizheng Zhang, Changxiu Ji, Qiuyang Mang, Xiaokun Chen, Lakshya A Agrawal, Wei-Liang Liao, Eric Yang, Alvin Cheung, James Zou, Kunle Olukotun, Ion Stoica, Joseph E. Gonzalez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching a Robot by Reading Its Diary

Imagine you have a very smart robot assistant (a Large Language Model) that helps you solve problems. Usually, you have to teach this robot by changing its internal code (like installing a new software update). But that's slow, expensive, and hard to do.

Prompt Learning is a newer, smarter way. Instead of changing the robot's code, you give it a "cheat sheet" or a "playbook" right before it starts working. As the robot tries to solve problems, it makes mistakes, learns from them, and writes down new rules in its playbook. This playbook gets better over time, making the robot smarter without ever touching its brain.

The Problem: The "Too Many Notes" Bottleneck

The researchers noticed a problem with how this learning happens.

The Old Way (Sequential): Imagine the robot solves one problem, writes a note in its playbook, and then stops. Then it solves the next problem, writes another note, and stops. This is slow.
The "Naive" Parallel Way: To speed things up, people tried to have 100 robots working at the same time. They all solve problems, write notes, and then dump all 100 notes into a single "Master Robot" to combine them into one big playbook.

Here is where it breaks: The Master Robot gets overwhelmed. It's like trying to read 100 different diaries at once to write a summary. The Master Robot gets confused, forgets the important details, and only writes down generic, boring advice like "Be careful." It throws away the specific, brilliant tips that actually help solve the hard problems. This is called "Context Overload."

The Solution: Combee (The Bee Colony)

The authors created a new system called Combee. They named it after bees because, like a hive, it uses a massive team to build something efficient without chaos.

Combee fixes the "Too Many Notes" problem using three clever tricks:

1. The Parallel Scan (The Team Huddle)

Instead of dumping all 100 notes into one giant pile, Combee organizes them like a military drill or a sports team huddle.

How it works: The 100 robots are split into 10 small groups. Each group combines their notes first. Then, those 10 group-leaders combine their summaries. Finally, the top leader combines the final 10 summaries.
The Analogy: Imagine a classroom. Instead of asking 30 students to shout their answers at the teacher all at once (chaos!), the teacher asks them to group into tables of 3. Each table agrees on one answer. Then the table leaders agree on a final answer. The teacher only has to listen to a few clear voices, not a roar. This prevents the "Master Robot" from getting overwhelmed.

2. Augmented Shuffling (The "Copy-Paste" Safety Net)

Sometimes, a really brilliant note gets lost in the shuffle because it's unique.

How it works: Before the notes go into the groups, Combee makes a few copies of every single note and mixes them up randomly.
The Analogy: Imagine you are passing a secret message down a long line of people. If you only pass it once, one person might drop it. But if you make three copies and pass them down three different lines, the chance that all copies get lost is almost zero. This ensures the "Master Robot" sees the best ideas, even if the group sizes are huge.

3. The Dynamic Batch Size (The Smart Traffic Light)

How many robots should work at once? Too few is slow; too many causes the overload.

How it works: Combee has a built-in traffic controller. It tests different group sizes in real-time. If it sees that adding more robots is slowing down the learning (because the notes are getting too messy), it automatically reduces the group size. If it sees it's safe to add more, it speeds up.
The Analogy: It's like a smart highway ramp meter. It doesn't just let 100 cars merge at once and cause a crash. It watches the traffic flow and lets cars merge in the perfect amount to keep everything moving fast without stopping.

The Results: Fast, Cheap, and Smart

The researchers tested Combee on difficult tasks like solving math problems, coding, and analyzing financial documents.

Speed: Combee was up to 17 times faster than the old methods.
Quality: Even though it was faster, it didn't make mistakes. In fact, it learned better because it didn't throw away the good notes.
Cost: It cost the same amount of money to run as the slow methods.

Summary

Combee is a system that lets AI agents learn from their mistakes much faster. It stops the "learning boss" from getting overwhelmed by too much information by organizing the learning process like a well-run bee colony: working in small teams, copying important info to be safe, and adjusting the team size on the fly. This means AI can get smarter in minutes instead of hours, without losing any of its brilliance.

1. Problem Statement: Context Overload in Parallel Prompt Learning

The paper addresses a critical bottleneck in scaling inference-time prompt learning (also known as "generate-reflect-update" loops) for Large Language Model (LLM) agents.

The Context: Existing methods (e.g., ACE, GEPA) allow agents to learn from execution traces by reflecting on their performance and updating a shared context artifact (e.g., a system prompt or playbook) without updating model weights.
The Bottleneck: Current methods are designed for sequential or low-parallelism settings. When researchers attempt to scale this by increasing the batch size (running $N$ agents in parallel and aggregating their reflections into a single update), they encounter Context Overload.
The Phenomenon: As the batch size increases, the "aggregator" LLM is forced to process an increasingly long context containing many reflections. Instead of retaining specific, high-value insights, the aggregator performs lossy compression, defaulting to generic patterns and discarding the nuanced, task-specific strategies that drive performance.
- Evidence: On the Formula dataset, increasing the batch size from 1 to 100 caused accuracy to drop from 87.0% to 72.5%, while the number of useful context entries dropped from 264 to 21.
The Gap: There is no principled strategy to scale prompt learning to high parallelism without degrading the quality of the learned context.

2. Methodology: The Combee Framework

The authors propose Combee, a distributed framework designed to scale parallel prompt learning while maintaining context quality. Combee adopts a Map-Shuffle-Reduce paradigm inspired by parallel computing and distributed training.

Key Components:

Parallel Scan Aggregation (The "Reduce" Phase):
- Instead of feeding all $N$ reflections directly into a single aggregator (which causes overload), Combee uses a multi-level hierarchical aggregation tree.
- Mechanism: The $N$ $N$ reflections are split into $k$ $k$ subgroups (where $k \approx \sqrt{N}$ $k \approx N$ ).
  - Level 1: Each subgroup is aggregated into a local context update.
  - Level 2: The $k$ local updates are aggregated into a single global update.
- Benefit: This ensures that no single LLM call processes a context window larger than $\sqrt{N}$ reflections, preventing the "lossy compression" effect while still aggregating global knowledge.
Augmented Shuffling (The "Shuffle" Phase):
- To prevent information loss during the hierarchical reduction, Combee introduces a redundancy mechanism.
- Mechanism: Before dispatching reflections to the aggregation tree, each reflection is duplicated $p$ times (default $p=2$ ) and the set is shuffled.
- Benefit: This increases the probability that high-value, specific insights survive the aggregation process, echoing the principle of self-consistency in reasoning.
Dynamic Batch Size Controller:
- Combee automatically balances the trade-off between training speed (larger batches) and learning quality (smaller batches).
- Mechanism: It profiles the delay at various candidate batch sizes and fits a power-law curve ( $T_{epoch} = A \cdot bs^{-\alpha}$ ). It selects the largest batch size where the marginal reduction in delay falls below a specific threshold ( $\tau$ ), ensuring the system operates at the "sweet spot" of efficiency without risking quality degradation.

3. Key Contributions

Problem Identification: Formalized the "Context Overload" problem, demonstrating that naive parallel scaling destroys the fine-grained knowledge necessary for effective prompt learning.
Framework Design: Introduced Combee, the first framework to enable high-degree parallelism in prompt learning using parallel scan aggregation and augmented shuffling.
Framework Agnosticism: Demonstrated that Combee can be integrated with existing prompt learning methods (specifically ACE and GEPA) with minimal changes, suggesting it generalizes to other "generate-reflect-update" systems.
Dynamic Optimization: Developed a runtime controller to automatically tune batch sizes for optimal speed-quality trade-offs.

4. Experimental Results

Combee was evaluated on four benchmarks: AppWorld (multi-step API tasks), Terminal-Bench 2.0 (software engineering), FiNER (financial entity recognition), and Formula (numerical reasoning).

Speedup: Combee achieved up to 17× speedup in training time compared to sequential baselines.
Accuracy: Unlike naive parallel methods (which degrade significantly at high batch sizes), Combee maintained comparable or better accuracy than the sequential baseline.
- Example (AppWorld): At batch size 40, naive ACE dropped to 55.7% accuracy (near the no-learning baseline), while Combee achieved 65.8% (beating the sequential baseline of 58.1%).
Context Retention: Combee preserved significantly more information. In the AppWorld experiment, the Combee playbook at batch 40 contained 6,887 tokens of context, whereas the naive batch 40 playbook contained only 526 tokens.
Cost Efficiency: Combee achieved these results with equivalent or slightly lower costs compared to baselines, as the massive reduction in training time offset the overhead of the shuffling/aggregation steps.
Robustness: The framework performed consistently across different base models (DeepSeek-V3.1, GPT-OSS 120B) and task domains.

5. Significance and Impact

Enabling Scale: Combee solves the fundamental systems challenge of scaling agent learning. As agentic systems move toward massive parallel deployments (e.g., thousands of agents running concurrently), Combee provides the architectural foundation to learn from these traces efficiently.
Quality Preservation: It proves that high parallelism does not necessitate a trade-off with learning quality, provided the aggregation strategy is principled (hierarchical) rather than naive (flat).
New Paradigm: The paper draws a strong analogy between parallel prompt learning and distributed machine learning training (where gradients are aggregated). It establishes "context" as a first-class medium for scalable learning, suggesting that techniques from distributed systems (like parallel scans and shuffling) are directly applicable to LLM agent evolution.
Practical Utility: By reducing training time from hours to minutes while improving performance, Combee makes self-improving agents more viable for real-world, time-sensitive applications.

In summary, Combee transforms prompt learning from a sequential, slow process into a scalable, distributed system, enabling agents to rapidly consolidate knowledge from massive parallel execution traces without losing the specific, high-value insights required for complex task performance.