🤖 AI

Efficient LLM Collaboration via Planning

The paper proposes COPE, a test-time collaboration framework where small and large language models alternately act as planners and executors in a multi-stage cascade, achieving performance comparable to large proprietary models while drastically reducing inference costs.

Original authors: Byeongchan Lee, Jonghoon Lee, Dongyoung Kim, Jaehyung Kim, Kyungjoon Park, Dongjun Lee, Jinwoo Shin

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Byeongchan Lee, Jonghoon Lee, Dongyoung Kim, Jaehyung Kim, Kyungjoon Park, Dongjun Lee, Jinwoo Shin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a complex task to solve, like fixing a broken engine or solving a tricky math puzzle. You have two helpers available:

The Local Handyman: A small, free, and fast worker who lives right in your garage. They are great at simple jobs but sometimes get stuck on complicated problems.
The Master Architect: A huge, expensive expert who lives in a distant, high-tech tower. They can solve almost anything, but hiring them costs a fortune every time you ask for help.

The problem is: If you hire the Master Architect for every single job, you'll go broke. If you only use the Local Handyman, they might fail on the hard stuff.

This paper introduces a new way to work called COPE (Collaborative Planning and Execution). It's like a smart team-up system that lets these two workers collaborate efficiently without wasting money.

How COPE Works: The "Plan First" Strategy

Instead of just asking the Handyman to "fix the engine" and hoping for the best, COPE uses a Planning step. Think of a "Plan" as a quick, high-level map or a set of instructions written before the actual work begins.

Here is the step-by-step process of how they collaborate:

Stage 1: The Handyman Tries to Lead
The Local Handyman tries to do the whole job alone. First, they write a quick "Plan" (a simple goal or guideline) for themselves, and then they try to solve the problem based on that plan.

The Check: If the Handyman is very confident they got it right (like if they tried it three times and got the same answer), they finish the job. It's free and fast!
The Fail: If they are unsure or the answer looks shaky, they don't give up. They move to Stage 2.

Stage 2: The Architect Draws a Map, The Handyman Builds
Now, the expensive Master Architect steps in, but only for a moment. The Architect doesn't do the heavy lifting yet. Instead, they write a new, better Plan (a clearer map) based on the problem.
They hand this new Plan to the Local Handyman. The Handyman then tries to solve the problem again, but this time they are following the Architect's expert map.

The Check: If the Handyman, guided by the Architect's map, feels confident, they finish the job. You only paid for a tiny bit of the Architect's time to draw the map, saving a lot of money.
The Fail: If the Handyman is still stuck even with the map, they escalate to Stage 3.

Stage 3: The Architect Takes Over
If the first two steps didn't work, the Master Architect finally does the whole job themselves—both planning and executing. This is the most expensive step, but it only happens when absolutely necessary.

Why This is a Big Deal

The paper found some surprising things about how this teamwork works:

Big brains help small brains: When the Master Architect writes the plan for the Handyman, the Handyman gets much smarter. It's like a student suddenly understanding a math problem after a professor explains the strategy (the plan), even if the student does the actual calculations.
Small brains can confuse big brains: Interestingly, if the Handyman tries to write the plan for the Master Architect, it actually makes the Architect perform worse. The Architect is so smart that a simple, low-quality plan from the Handyman can actually distract them. This is why the system is designed so the Architect only plans for the Handyman, not the other way around.
It saves money without losing quality: In tests involving math, coding, and even complex agent tasks (like a robot navigating a house), COPE achieved results that were just as good as, or even better than, using the Master Architect alone. However, it cost 45% to 75% less because the expensive Architect was only called in for the hardest parts or just to draw the map.

The Bottom Line

COPE is like a smart manager who knows when to let the local worker do the job, when to ask the expert for a quick tip (the plan), and when to call the expert to do the whole thing. By using "Planning" as a lightweight bridge between a small model and a large model, it allows them to work together efficiently, solving hard problems without breaking the bank.

Technical Summary: Efficient LLM Collaboration via Planning (COPE)

Problem Statement

Large Language Models (LLMs) face a critical trade-off between capability and computational cost. While large proprietary models achieve state-of-the-art performance on complex tasks, their inference costs are substantial, making frequent use impractical for many applications. Conversely, small, open-source models are cost-effective and easily deployable on edge devices but often lack the capacity to solve complex problems independently. Existing cost-aware inference methods, such as multi-stage cascades (e.g., LLM Cascade, ABC), typically operate via independent delegation: a small model attempts a task, and if it fails a confidence check, the entire task is handed off to a larger model. These approaches lack structured, interactive collaboration within a stage, limiting their ability to jointly solve complex tasks.

Methodology: COPE Framework

The authors propose COPE (Collaborative Planning and Execution), a test-time collaboration framework that structures the interaction between small and large models through planning. Instead of directly executing a task, a model first generates a high-level abstraction (a "plan"), such as a goal or a guideline, which serves as a lightweight intermediate to guide a downstream executor.

COPE operates in a three-stage adaptive cascade:

Stage 1 (Small Model Planning & Execution): A small model acts as both planner and executor. It samples $n$ goal-type plans and generates corresponding solutions. The system aggregates answers via majority voting. If the consensus ratio (the fraction of samples agreeing on the most frequent answer) exceeds a threshold $\tau_1$ , the answer is accepted. Otherwise, the problem escalates.
Stage 2 (Large Model Planning, Small Model Execution): If Stage 1 fails, a large model generates a new guideline-type plan. The small model then re-attempts execution, leveraging both the original plan from Stage 1 and the new plan from the large model. It generates new solutions, and majority voting is applied again. If the consensus ratio exceeds a second threshold $\tau_2$ , the answer is accepted; otherwise, it escalates.
Stage 3 (Large Model Planning & Execution): If Stage 2 fails, the large model assumes full control, reusing the plan generated in Stage 2 to directly execute the task.

Key Technical Features:

Training-Free: COPE requires no fine-tuning; it relies solely on prompt engineering at inference time.
Adaptive Compute: The framework dynamically allocates resources, handling easy tasks with small models and escalating only difficult tasks to larger models.
Task-Specific Confidence: The confidence metric is adapted to the task type:
- Mathematical Reasoning: Majority voting on extracted answers.
- Code Generation: Pass rate on hidden test cases (if any output passes, it is selected).
- Open-Ended Tasks: Perplexity minimization (lowest perplexity output selected; escalation if perplexity is too high).
- Agent Tasks: Majority voting on action steps.

Key Contributions

Framework Proposal: Introduction of COPE, a test-time framework that utilizes planning as a mediator for cross-model collaboration, enabling small and large models to scaffold each other's thinking.
Training-Free Efficiency: The method is training-free and few-shot-free, requiring only prompts, which facilitates easy deployment across diverse tasks.
Performance-Cost Trade-off: Demonstrated ability to match or surpass the performance of large proprietary models while drastically reducing monetary inference costs.
Broad Applicability: Validated across mathematical reasoning, code generation, open-ended generation, and agent tasks, showing that planning serves as an effective prior for cost-efficient inference.

Experimental Results

The authors evaluated COPE on multiple benchmarks, comparing it against baselines including single-model inference, standard cascades (Vote/Verify), ABC, and FrugalGPT.

Mathematical Reasoning (MATH-500): COPE achieved 75.8% accuracy, outperforming GPT-4o (75.2%) while reducing inference cost by nearly 45% (from 5025 to 2766 USD per million problems). When using GPT-4o-mini as the large model, COPE achieved 74.4% accuracy (vs. 73.8% for the large model alone) with a 40% cost reduction.
Mathematical Reasoning (AIME-2024): On this more difficult dataset, COPE achieved 40.0% accuracy, the highest among all baselines, with the lowest cost (478 vs. 502 for the large model alone).
Code Generation (MBPP): COPE improved accuracy to 66.4% compared to GPT-4o's 64.0%, while cutting inference costs by nearly 75%.
Open-Ended Tasks (MT-Bench): COPE achieved a 44.8% win rate against the large model baseline (vs. 42.5% for the large model alone) with significantly lower cost.
Agent Tasks (ALFWorld): COPE improved the success rate to 36.9% (vs. 35.0% for the large model) while reducing costs by approximately 29%.

The authors note that the performance gains of COPE over small-model baselines are most pronounced on difficult problems (Level 5 in MATH-500), where accuracy improved by 15.0 percentage points.

Significance and Claims

The paper claims that planning functions as an effective prior for cost-efficient inference. By structuring collaboration through planning, COPE allows models to exchange high-level foresight without the overhead of full task delegation. The authors highlight that this approach aligns naturally with edge-cloud and agentic settings, where most steps can be handled cheaply by small models, with larger models intervening selectively for harder decisions.

The work suggests that cross-model scaling via planning is a viable alternative to simply pushing a single model further or relying on independent cascades. The results indicate that COPE is a broadly effective and scalable framework that can reduce unnecessary compute, contributing to more sustainable AI usage without sacrificing performance.