🤖 AI

COMPAS: Difficulty-Aware Joint Search for Optimizing Code Generation

COMPAS is a difficulty-aware framework that optimizes code generation by jointly searching for the best model, prompt, and decoding settings for specific task difficulty groups, achieving significant improvements in both pass rates and cost efficiency on benchmarks like LiveCodeBench and SWE-bench.

Original authors: Jingzhi Gong, Jie M. Zhang, Gunel Jahangirova, Dong Huang, Mohammad Reza Mousavi, Mark Harman

Published 2026-08-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jingzhi Gong, Jie M. Zhang, Gunel Jahangirova, Dong Huang, Mohammad Reza Mousavi, Mark Harman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to bake the perfect cake, but you have a magical oven that can talk to you. In the world of computer science, this "oven" is a Large Language Model (LLM), a super-smart AI that can write computer code just like a human programmer. To get the best cake, you need to choose three things: which oven to use (the model), what recipe to give it (the prompt), and how to set the temperature and timer (decoding settings). For a long time, scientists tried to find the one perfect recipe and oven setting that would work for every single cake, from a simple cupcake to a complex wedding tier. But they kept running into a problem: a setting that makes a cupcake fluffy might burn a wedding cake, and a setting that saves money on flour might ruin the taste. The big question was: how do we find the perfect combination for each specific task without spending a fortune or years of time trying every possibility?

Enter COMPAS, a new method that acts like a brilliant, budget-conscious head chef. Instead of guessing one rule for everyone, COMPAS realizes that different tasks have different "difficulty levels." It suggests that we should group our tasks—like sorting problems into "Easy," "Medium," and "Hard"—and then find a unique, perfect recipe for each group. The researchers found that prompts (the instructions) and decoding settings (the oven dials) work best when tweaked together, not separately, and that what works for one AI model might fail for another. By using a smart two-step process—first picking the right oven for the job, then fine-tuning the recipe and dials together—they built a "menu" of options for each difficulty level. When a new task arrives, COMPAS instantly checks its difficulty, picks the perfect pre-made menu for that group, and gets to work.

In their experiments, this approach was a huge success. On a test set of coding problems called LiveCodeBench, COMPAS managed to get the right answer 52.8% of the time, beating the previous best method which only reached 45.9%. Even better, it did this while spending significantly less money: the cost dropped from $36.57 down to just $4.92. They also tested it on a more complex challenge involving fixing bugs in entire software projects (SWE-bench), where it solved 76.0% of the tasks, again outperforming the best existing methods.

The secret sauce of COMPAS is its "difficulty-aware" strategy. The researchers discovered three key things through their experiments. First, the instructions and the oven settings interact; changing both together yields better results than changing them one by one. Second, a tweak that helps one AI model might actually hurt another, so you have to pick your model carefully before you start tweaking. Third, and most importantly, the "best" setting for an easy problem is totally different from the "best" setting for a hard one. A global, one-size-fits-all approach just doesn't work.

To solve this, COMPAS uses a two-phase plan. In the offline phase (the preparation stage), it splits all the training tasks into groups based on how hard they are. It then runs a fast, cheap test to pick the best AI model for each group. Once the model is chosen, it goes into a "joint search" mode, where it uses a smart feedback loop to simultaneously adjust the prompt and the decoding settings. It doesn't just try random combinations; it learns from its mistakes and successes, building a "quality-cost front"—essentially a list of the best possible trade-offs between getting a good result and saving money—for each difficulty group.

In the online phase (the real-time cooking), when a new task comes in, COMPAS doesn't waste time searching. It simply looks at the task's difficulty label, finds the matching group, and picks the best configuration from that group's pre-built menu. It's like having a library of perfectly tuned recipes ready to go, so you never have to start from scratch.

The paper shows that this method is robust. Even when they changed the random seeds (like shuffling the deck of cards differently) or tested it on different types of AI models, COMPAS consistently outperformed other methods. It also proved that breaking tasks down by difficulty is crucial; if you ignore the difficulty levels and try to use one setting for everything, your results drop significantly. While the method currently works best when the AI models are from the same "family," the researchers suggest that this approach could be expanded in the future. For now, COMPAS stands as a powerful demonstration that being smart about how you search for the right settings is just as important as the settings themselves.

Technical Summary: COMPAS

Problem Statement

Code generation systems utilizing Large Language Models (LLMs) require three critical configuration choices for every inference call: the model, the prompt, and the decoding settings (e.g., temperature, top-p, frequency penalty). Existing optimization approaches suffer from significant limitations:

Global Optimizers (e.g., EcoTune, EcoOptiGen) search for a single configuration applicable to all tasks, ignoring that optimal settings vary by task difficulty.
Routers (e.g., RouteLLM) select only the model while keeping prompts and decoding settings fixed.
Prompt Optimizers (e.g., GEPA) tune prompts but fix the model and decoding settings.

This fragmentation leaves the joint, group-specific interactions between models, prompts, and decoding settings unclear. Furthermore, a brute-force search of the configuration space is computationally prohibitive; for instance, searching across 10 prompts, 2 models, and 10 values for three decoding settings across three difficulty groups yields 60,000 candidates, requiring over 20 days of serial evaluation.

Empirical Motivations

The authors conducted a large-scale empirical study on the LiveCodeBench release-v5 set (880 tasks) using DeepSeek-V3 and DeepSeek-V3.2 to identify patterns guiding the design of COMPAS. Three key observations emerged:

Prompts and Decoding Interact: The best joint configuration (72.2% Pass@1) significantly outperforms the best prompt-only (64.3%) or decoding-only (64.9%) configurations. This necessitates a joint search rather than independent tuning.
Joint Tuning is Model-Specific: A prompt-decoding update that improves one model (e.g., DeepSeek-V3 from 76.3% to 77.2%) may degrade another (e.g., DeepSeek-V3.2 from 74.0% to 72.1%). This suggests that model selection must precede joint tuning to reduce the search space.
Best Config Differs by Group: The best configuration for a specific difficulty group (62.4% average Pass@1) outperforms both the default configuration (54.1%) and the best single global configuration (53.4%). This indicates that a single global front is suboptimal; separate quality-cost fronts are required for different difficulty groups.

Methodology: The COMPAS Framework

COMPAS (Code-generation Optimization over Models, Prompts, And Decoding Settings) is a two-phase, difficulty-aware method that separates offline search from online routing.

1. Offline Search Phase

Given a training set and a fixed token budget, COMPAS partitions tasks into difficulty groups (Easy, Medium, Hard) based on benchmark labels (e.g., Elo ratings for LiveCodeBench, estimated fix time for SWE-bench). It then optimizes each group in two stages:

Stage 1: Fast Model Selection:
- COMPAS probes candidate models on a small, random subset of tasks (capped at 8 problems per group) using a fixed default prompt and decoding settings.
- A cost-aware rule selects the winning model ( $m^*_g$ ) for each group, balancing quality and cost using a weight $\alpha_g$ (0.2 for Easy, 0.5 for Medium, 0.8 for Hard).
- This reduces the subsequent search space to a single model per group.
Stage 2: Joint Tuning of Prompt and Decoding Settings:
- Using the selected model, COMPAS performs a reflective search to jointly mutate the prompt and decoding settings (temperature, top-p, frequency penalty).
- Reflection Loop: An LLM analyzes a compact context containing failure/success patterns, search history, and current Pareto fronts to propose a new candidate configuration.
- Mini-batch Evaluation: Candidates are evaluated on a mini-batch from a search pool and a held-out pool to prevent overfitting. A candidate is accepted only if it improves the score or reduces cost without degrading performance on the held-out set.
- Pareto Front Construction: Accepted candidates are added to a group-specific quality-cost front ( $F_g$ ) if they are Pareto-optimal.

2. Online Testing Phase

During deployment, no further search occurs.

Routing: A task $x$ is assigned to its difficulty group $g(x)$ using existing benchmark labels.
Selection: The system selects the configuration from $F_{g(x)}$ that maximizes a normalized quality-cost trade-off using the same weights $\alpha_g$ established in Stage 1.
Fallback: If a group's front is empty, the system falls back to a coarser difficulty front or the default configuration.

Key Contributions

Empirical Characterization: The paper provides evidence that prompt-decoding interactions are significant, tuning effects are model-specific, and optimal configurations vary by task difficulty.
Difficulty-Aware Search: Unlike methods that build a single global front, COMPAS partitions problems into groups and constructs separate quality-cost fronts for each, enabling specialized optimization.
Two-Stage Efficient Search: The framework combines a cheap model probe with an LLM-based reflection loop for joint prompt-decoding optimization, making the search space tractable.

Experimental Results

The method was evaluated on LiveCodeBench (function-level and repository-level code generation) and SWE-bench Verified-mini.

LiveCodeBench (v6-only): Under a matched search budget (1M tokens), COMPAS achieved a Pass@1 of 52.8% with a cost of $4.92. This outperformed the best baseline (EcoTune), which achieved 45.9% Pass@1 at a cost of $36.57. COMPAS reduced costs by 7.4× while improving quality by 6.9 percentage points.
Robustness: Results held across independent random splits and multiple random seeds. COMPAS consistently led in Pass@1 and cost across five different seeds.
Generalization:
- On SWE-bench, COMPAS resolved 76.0% of tasks (38/50), surpassing the best baseline (70.0%), though at a higher cost due to routing more tasks to stronger models.
- Across different model families (DeepSeek, Devstral, Qwen3.5), COMPAS generally improved upon baselines, particularly on Qwen3.5 (40.6% Pass@1).
Ablation Studies: Removing any component (e.g., task grouping, model selection, joint search) degraded performance. Task grouping was identified as the most critical component; removing it dropped Pass@1 to 39.4% (below the default baseline).

Significance and Claims

The paper claims that COMPAS demonstrates that jointly selecting models, prompts, and decoding settings based on task difficulty yields superior quality-cost trade-offs compared to routers, global tuners, or prompt-only optimizers.

The authors emphasize that their approach is a search-based software engineering solution that relies on execution feedback rather than hand-designed configurations. They note that their results are specific to the fixed harness used in the experiments and that the primary limitation is the testing of only two same-family LLMs per pool. They suggest future work could extend this to cross-family searches. The paper does not claim to solve the general problem of code generation but rather optimizes the configuration strategy within existing generation frameworks.